The provided text introduces DeepSeekMoE, an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in large language models. The authors propose two key strategies: fine-grained expert segmentation, which divides experts into smaller, more numerous units for flexible combinations, and shared expert isolation, which designates specific experts for common knowledge to reduce redundancy. Through comprehensive experimentation, DeepSeekMoE demonstrates superior performance and computational efficiency compared to conventional MoE models like GShard and dense models, even when scaled up to 145B parameters. The research also highlights DeepSeekMoE's adaptability for fine-tuning into chat models and emphasizes its lower redundancy among routed experts, ultimately aiming for more accurate and efficient knowledge acquisition.

Source: 2024 - https://arxiv.org/pdf/2401.06066