Sign up to save your podcasts
Or
Share MH-MoE: Multi-Head Mixture-of-Experts
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
MH-MoE: Multi-Head Mixture-of-Experts
🤗 Paper Upvotes: 17 | cs.CL
Authors:
Shaohan Huang, Xun Wu, Shuming Ma, Furu Wei
Title:
MH-MoE: Multi-Head Mixture-of-Experts
Arxiv:
http://arxiv.org/abs/2411.16205v2
Abstract:
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.
View all episodes
By Jingwen Liang, Gengyu Wang🤗 Paper Upvotes: 17 | cs.CL
Authors:
Shaohan Huang, Xun Wu, Shuming Ma, Furu Wei
Title:
MH-MoE: Multi-Head Mixture-of-Experts
Arxiv:
http://arxiv.org/abs/2411.16205v2
Abstract:
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.