AI Post Transformers

Ten different sources are used in this episode which are excerpts from academic papers and technical reports focusing on mechanistic interpretability and sparse autoencoders in language models (LLMs) and vision-language models (VLMs). This episode explores the state-of-the-art in Mechanistic Interpretability (MI), focusing on how researchers are decomposing large language models (LLMs) and multimodal models (MLLMs) into understandable building blocks. A central theme is the power of Sparse Autoencoders (SAEs), which address the issue of polysemanticity—where a single neuron represents many unrelated concepts—by training overcomplete bases to extract sparse, monosemantic features. The episode would detail the successful scaling of SAEs to production models like Claude 3 Sonnet and Claude 3.5 Haiku, demonstrating that these techniques reveal features that are often abstract, multilingual, and even generalize across modalities (from text to images). Listeners would learn how advanced techniques like Specialized SAEs (SSAEs) are developed using dense retrieval to target and interpret rare or domain-specific "dark matter" concepts, such as specialized physics knowledge or toxicity patterns, that are often missed by general methods. The fundamental goal is establishing a linear representation of concepts that facilitates precise understanding and, crucially, manipulation of model internals. The second half of the episode dives into the application of these features to trace computational pathways, or circuits, using tools like attribution graphs and causal interventions. We explore concrete discoveries regarding LLM reasoning, such as identifying the modular circuit components—like queried-rule locating, fact-processing, and decision heads—that execute propositional logic and multi-step reasoning. We review how these mechanistic insights enable precise control, such as editing a model's diagnostic hypothesis (e.g., in medical scenarios) or circumventing refusal behaviors (jailbreaks) by overriding harmful request features. We cover cutting-edge intervention methods like Attenuation via Posterior Probabilities (APP), which leverages the improved separation of concepts achieved by SAEs to perform highly effective and minimally disruptive concept erasure.Sources:1. 2025, Carnegie Mellon University: https://aclanthology.org/2025.findings-naacl.87.pdf (Source for Specialized Sparse Autoencoders)2. 2025, OpenAI: (Implicit Source: PDF for the paper titled "Weight-sparse transformers have interpretable circuits," attributed to an OpenAI author)3. 2024, Anthropic: (Implied Source URL for the work "Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet," published May 21, 2024)4. 2024, Anthropic: The claude 3 model family: Opus, sonnet, haiku (URL/document cited in circuit analysis work)5. 2024, Gemma Team: https://arxiv.org/abs/2408.00118 (Gemma 2: Improving open language models at a practical size)6. 2024, OpenAI: https://openai.com/index/learning-to-reason-with-llms/ (Learning to reason with LLMs)7. 2023, Transformer Circuits Thread: https://transformer-circuits.pub/2023/monosemantic-features/index.html (Towards Monosemanticity: Decomposing Language Models With Dictionary Learning)8. 2022, AI Alignment Forum: https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing (Causal scrubbing)9. 2022, Transformer Circuits Thread: https://transformer-circuits.pub/2022/solu/index.html (Softmax Linear Units)10. 2021, Transformer Circuits Thread: https://transformer-circuits.pub/2021/framework/index.html (A mathematical framework for transformer circuits)

The July 13, 2025 paper " Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications" introduces a practical framework for evaluating safety risks in real-world Large Language Model (LLM) applications, arguing that current methods focusing only on foundation models are inadequate. This framework consists of two main parts: principles for developing customized safety risk taxonomies and practices for evaluating these risks within the application itself, which often includes components like system prompts and guardrails. It emphasizes the need for organizations to contextualize general risks and create taxonomies that are practical and specific to their operational context, as demonstrated by a case study from a government agency. The document then outlines a safety testing pipeline that involves curating meaningful and diverse adversarial prompts, running automated black-box tests, and evaluating model responses, particularly focusing on the use of refusals as a measure of safety. Source: July 13, 2025 Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications https://arxiv.org/pdf/2507.09820

The November 22, 2024 paper from UT Texas introduces AdaFlow, a novel imitation learning framework designed to improve both the efficiency and diversity of policy generation, addressing computational bottlenecks found in previous diffusion-based methods. AdaFlow utilizes flow-based generative modeling represented by ordinary differential equations (ODEs) and incorporates a variance-adaptive ODE solver that dynamically adjusts the number of inference steps based on the complexity of the state. This adaptive approach allows AdaFlow to function as a highly efficient one-step action generator for states with deterministic actions while retaining the ability to produce diverse actions for multi-modal scenarios. Empirical results across various benchmarks, including maze navigation and complex robot manipulation tasks, demonstrate that AdaFlow achieves high success rates with significantly reduced inference time compared to state-of-the-art models like Diffusion Policy. The research establishes a connection between the conditional variance of the training loss and the discretization error of the ODEs, providing the theoretical basis for AdaFlow’s computational adaptivity. Source: https://arxiv.org/pdf/2402.04292

The November 3, 2025 paper provide an overview of the AlphaEvolve system, an AI-powered evolutionary approach for mathematical exploration and discovery, often involving large language models (LLMs) to generate and mutate programs that act as search heuristics for finding extremal mathematical objects. The first source is a GitHub repository from Google DeepMind that serves as a collection of 67 mathematical problems used to test AlphaEvolve, noting that the code for the system itself is not included. The second, more extensive source is an academic paper detailing AlphaEvolve's methodology, contrasting it with its predecessor FunSearch, and showcasing its application to solving a wide array of complex, open mathematical problems across combinatorics, geometry, analysis, and number theory, frequently achieving or improving upon state-of-the-art bounds. This system demonstrates proficiency in both a standard search mode for fixed problem sizes and a "generalizer mode" for developing programs that solve problems for arbitrary inputs.Sources:https://arxiv.org/pdf/2511.02864https://github.com/google-deepmind/alphaevolve_repository_of_problems

The June 13, 2025 joint collaboration between Stanford University, Caltech and University at Buffalo introduces a novel method called CARTRIDGE for efficiently handling long text corpora in large language models, addressing the high memory cost associated with standard In-Context Learning (ICL) and its required KV cache. A CARTRIDGE is a smaller, trained KV cache representation of a corpus, which is created offline using a technique termed SELF-STUDY. This training process involves generating synthetic conversational data about the corpus and employing a context-distillation objective to ensure the CARTRIDGE maintains the generality and structural awareness of ICL while dramatically reducing memory consumption (up to 38.6x less) and increasing throughput. The research demonstrates that CARTRIDGES can match or exceed ICL performance, enable context length extrapolation beyond the model's native window, and even be composed together at inference time. The paper also includes detailed ablation studies on the SELF-STUDY components and theoretical analysis contrasting this gradient-descent approach with other memory methods like linear attention on synthetic memory tasks. Source: June 13, 2025 https://arxiv.org/pdf/2506.06266

The August 27, 2025 paper introduces Confucius, a novel multi-agent Large Language Model (LLM) framework developed by Meta for intent-driven network management in hyper-scale environments. The framework models complex management tasks as directed acyclic graphs (DAGs) and integrates LLMs with existing tools using domain-specific languages (DSLs) to enhance planning and execution. Confucius leverages Retrieval-Augmented Generation (RAG) for long-term memory and employs specialized primitives like Translator, Selector, and Collector to improve translation accuracy and systematic validation for critical network operations. Successfully deployed for two years with over 60 onboarded applications, the system aims to significantly reduce manual engineering effort for tasks such as capacity planning and fault diagnosis while maintaining high accuracy. Source: https://dl.acm.org/doi/10.1145/3718958.3750537

These five papers from 2022 up to 2025 discuss various knowledge distillation techniques aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces Contextualization Distillation (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to generate descriptive context for triplets significantly enhances the performance of smaller, specialized KGC models, often outperforming direct use of LLMs for the task. Another source proposes Context Distillation as a general method for language models to internalize abstract instructions, step-by-step reasoning (scratch-pads), and concrete examples, effectively eliminating the need for lengthy prompts and improving inference efficiency. The third document details In-context Learning Distillation, a framework that combines in-context learning objectives with traditional language modeling to effectively transfer few-shot learning abilities from large to smaller models under different tuning paradigms. Finally, Generative Prompt Internalization (GenPI) is presented as a method to fully embed long, complex prompts into a smaller model by training it to generate the prompt content and the reasoning for its corresponding behavior, greatly increasing efficiency in agent-based applications. 2022: Learning by Distillation Context https://arxiv.org/pdf/2209.15189 2022: In-context Learning Distillation: Transferring Few-shot https://arxiv.org/pdf/2212.10670 2024: Contextualization Distillation from Large Language Model for Knowledge Graph Completion https://aclanthology.org/2024.findings-eacl.32.pdf May 12, 2025: Efficient LLM Context Distillation https://arxiv.org/pdf/2409.01930 March 25, 2025: Generative Prompt Internalization https://arxiv.org/pdf/2411.15927

The October 31, 2025 paper introduces Continuous Autoregressive Language Models (CALM), a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to continuous next-vector prediction. This approach compresses a chunk of multiple tokens into a single continuous vector using a high-fidelity autoencoder, thereby reducing the number of generative steps and significantly improving the performance-compute trade-off. To manage the challenges of operating in this continuous, likelihood-free domain, the framework includes a comprehensive toolkit: an energy loss function for training, a novel, sample-based evaluation metric called BrierLM, and likelihood-free algorithms for temperature sampling. Ultimately, the CALM framework establishes semantic bandwidth as a powerful new axis for scaling language models, enabling superior efficiency compared to discrete baselines. Source: October 31, 2025 CONTINUOUS AUTOREGRESSIVE LANGUAGE MODELS https://arxiv.org/pdf/2510.27688

The four papers we review dated from 1967 up to two papers in 2025 collectively discuss the mathematical properties and deep learning applications of doubly stochastic matrices, which are nonnegative matrices whose rows and columns sum to one. One paper, "Concerning Nonnegative Matrices and Doubly Stochastic Matrices," provides the foundational mathematical theory regarding the convergence of iterative row and column scaling (known as the Sinkhorn algorithm) to a unique doubly stochastic matrix, contingent on the original matrix having "total support." The other papers focus on Transformer architecture enhancements, proposing "Sinkformers" and "Sparse Sinkhorn Attention" as variants that replace the standard row-wise SoftMax attention with the Sinkhorn algorithm to enforce doubly stochastic attention matrices for improved performance and theoretical properties, such as a connection to the Wasserstein metric. Furthermore, the "Gradient Multi-Normalization" paper introduces a stateless optimizer that uses a multi-normalization procedure, including a "Square-Root Sinkhorn" variant, demonstrating its efficacy and efficiency in training large language models.Sources:1967:CONCERNING NONNEGATIVE MATRICES AND DOUBLY STOCHASTIC MATRICEShttps://projecteuclid.org/journalArticle/Download?urlId=pjm%2F1102992505June 24, 2022:Sinkformers: Transformers with Doubly Stochastic Attentionhttps://arxiv.org/pdf/2110.11773February 10, 2025:Gradient Multi-Normalization for Stateless and Scalable LLM Traininghttps://arxiv.org/pdf/2502.06742July 12, 2025:ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Planshttps://arxiv.org/pdf/2502.07962

These two academic papers introduce novel programming models aimed at systematically optimizing complex AI systems, particularly those built using Large Language Models (LLMs). The first source presents DSPy, a framework that abstracts traditional, hard-coded LLM pipelines into parameterized, declarative modules that can be automatically optimized using a compiler and teleprompters, demonstrating superior performance compared to hand-crafted prompts on tasks like math word problems. The second source introduces TEXTGRAD, a general optimization framework that utilizes LLMs to generate and propagate natural language gradients—textual feedback—through computation graphs, applying this "textual differentiation" approach successfully across diverse domains, including prompt optimization, code refinement, and scientific applications like molecular and medical treatment plan design. Both works highlight the shift from relying on expert prompt engineering to employing systematic, programmatic optimization techniques for compound AI systems.Sources:October 5, 2023DSPY: COMPILING DECLARATIVE LANGUAGE MODEL CALLS INTO SELF-IMPROVING PIPELINEShttps://arxiv.org/pdf/2310.03714June 11, 2024TextGrad: Automatic “Differentiation” via Texthttps://arxiv.org/pdf/2406.07496