## Short Segments

Speculative decoding just got a major reliability boost with EAGLE 3.1, fixing attention drift in LLM inference. Today, we're diving into how EAGLE 3.1 enhances speculative decoding, a technique that speeds up large language model inference by using a small draft model to propose tokens, which the larger model then verifies. While previous versions struggled with attention drift, EAGLE 3.1 introduces per-layer normalization and a post-norm feedback loop to stabilize performance. This upgrade means up to twice the acceptance length and throughput, depending on hardware and prompt distribution. For developers, this means more reliable and efficient LLM deployments, maintaining compatibility with existing checkpoints. Coming up, we'll explore MEMO, a modular framework that separates memory from reasoning in LLMs, offering a new way to update knowledge without modifying model parameters.

## Feature Story

Introducing MEMO: a modular framework that revolutionizes how large language models handle new knowledge without altering their core parameters. Traditionally, LLMs become static post-pretraining, unable to update as the world evolves. Retraining these models is costly, and fine-tuning risks losing previously learned information. Enter MEMO, developed by researchers from the National University of Singapore, MIT CSAIL, A*STAR, and SMART. This approach separates memory from reasoning, using a dedicated MEMORY model to internalize new knowledge while keeping the main EXECUTIVE model unchanged. MEMO addresses the limitations of existing methods like retrieval-augmented generation, which struggles with cross-document reasoning, and parametric methods that are computationally expensive and prone to catastrophic forgetting. By decoupling memory updates from the base model, MEMO offers a robust solution for continual learning without degrading existing knowledge. This separation allows for more flexible and transferable knowledge integration across different LLMs. In practical terms, MEMO enables developers to update a model's knowledge base without the need for extensive retraining, making it a cost-effective and efficient solution for keeping AI systems current. As AI continues to advance towards Artificial General Intelligence, frameworks like MEMO are crucial for overcoming the static nature of traditional LLMs, paving the way for more adaptable and intelligent systems. For AI practitioners, MEMO represents a significant step forward in managing and updating AI knowledge bases, offering a new paradigm for integrating and reasoning with new information. As we look to the future, MEMO's modular approach could become a standard in AI development, providing a scalable and efficient method for maintaining up-to-date AI systems. Stay tuned as we continue to explore the latest advancements in AI tools and technologies.