The sources discuss Mixture-of-Experts (MoE) models, a type of neural network that selectively activates different parameters for incoming data, offering a high parameter count at a constant computational cost. One paper introduces "MoE-Infinity," an offloading-efficient system designed to serve these memory-intensive models, particularly for users with limited GPU resources. It addresses latency issues in existing offloading approaches by introducing "Expert Activation Matrix" (EAM) for request-level tracing of expert usage, enabling more effective prefetching and caching strategies. The second source, "Switch Transformers," details a simplified MoE architecture that improves routing efficiency, reduces communication costs, and enhances training stability, even allowing lower-precision training. This innovation significantly accelerates pre-training speeds for large language models, demonstrating the benefits of scaling models by increasing sparse parameters while keeping computational costs stable.

Sources:

1) 2018 - https://arxiv.org/html/2401.14361v2 - MoE-Infinity: Offloading-Efficient MoE Model Serving

2) 2022 - https://arxiv.org/pdf/2101.03961 - Switch Transformers: Scaling to Trillion Parameter Models

with Simple and Efficient Sparsity