AI Post Transformers

This episode examines why simply adding more identical LLM agents to a multi-agent system stops helping after a handful of copies, and what actually determines whether scaling agent count pays off. Drawing on the paper "Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity," the discussion contrasts homogeneous setups — same model, same prompt, run repeatedly — with heterogeneous mixtures of different models, showing that two diverse agents can outperform sixteen identical ones. The hosts unpack the paper's information-theoretic framing: each agent's output as a "channel" of evidence about the correct answer, with a hard ceiling on how much signal can ever be extracted regardless of agent count, and a formal explanation for why redundant agents contribute diminishing returns while genuinely complementary ones keep adding value. They trace this idea back to prior work like Mixture-of-Agents and documented scaling plateaus, positioning this paper's contribution as the first rigorous account of *why* homogeneous scaling saturates. Listeners building or evaluating multi-agent LLM systems will come away with a concrete framework for deciding when to add more agents versus when to diversify them instead.

Sources:

1. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, Shangding Gu, 2026

http://arxiv.org/abs/2602.03794

2. More Agents Is All You Need — Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, Deheng Ye, 2024

https://scholar.google.com/scholar?q=More+Agents+Is+All+You+Need

3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024

https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities

4. Improving Factuality and Reasoning in Language Models through Multi-Agent Debate — Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch, 2023

https://scholar.google.com/scholar?q=Improving+Factuality+and+Reasoning+in+Language+Models+through+Multi-Agent+Debate

5. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework — Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, et al., 2023

https://scholar.google.com/scholar?q=AutoGen%3A+Enabling+Next-Gen+LLM+Applications+via+Multi-Agent+Conversation+Framework

6. Neural Network Ensembles, Cross Validation, and Active Learning — Anders Krogh, Jesper Vedelsby, 1995

https://scholar.google.com/scholar?q=Neural+Network+Ensembles%2C+Cross+Validation%2C+and+Active+Learning

7. Deep Learning and the Information Bottleneck Principle — Naftali Tishby, Noga Zaslavsky, 2015

https://scholar.google.com/scholar?q=Deep+Learning+and+the+Information+Bottleneck+Principle

8. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, et al., 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

9. Diversity Creation Methods: A Survey and Categorisation — Gavin Brown, Jeremy Wyatt, Rachel Harris, Xin Yao, 2005

https://scholar.google.com/scholar?q=Diversity+Creation+Methods%3A+A+Survey+and+Categorisation

10. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems — Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J., 2024

https://scholar.google.com/scholar?q=Are+More+LLM+Calls+All+You+Need%3F+Towards+Scaling+Laws+of+Compound+Inference+Systems

11. Towards a Science of Scaling Agent Systems — Kim, Y., Gu, K., Park, C., et al., 2025

https://scholar.google.com/scholar?q=Towards+a+Science+of+Scaling+Agent+Systems

12. Why Do Multi-Agent LLM Systems Fail? — Cemri, M., Pan, M. Z., Yang, S., et al., 2025 (NeurIPS Datasets and Benchmarks Track)

https://scholar.google.com/scholar?q=Why+Do+Multi-Agent+LLM+Systems+Fail%3F

13. Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? — Choi, H. K., Zhu, X., and Li, S., 2025

https://scholar.google.com/scholar?q=Debate+or+Vote%3A+Which+Yields+Better+Decisions+in+Multi-Agent+Large+Language+Models%3F

14. The Hidden Strength of Disagreement: Unraveling the Consensus-Diversity Tradeoff in Adaptive Multi-Agent Systems — Wu, Z. and Ito, T., 2025

https://scholar.google.com/scholar?q=The+Hidden+Strength+of+Disagreement%3A+Unraveling+the+Consensus-Diversity+Tradeoff+in+Adaptive+Multi-Agent+Systems

15. PersonaGym: Evaluating Persona Agents and LLMs — Samuel, V., Zou, H. P., Zhou, Y., et al., 2024

https://scholar.google.com/scholar?q=PersonaGym%3A+Evaluating+Persona+Agents+and+LLMs

This episode examines whether a system of small, coordinated language models can outperform a single large model on complex tasks — testing the paper "Can Small Agents Collaborate to Beat a Single Large Language Model?" from researchers at the University of Amsterdam, ETH Zürich, and the University of Copenhagen. It unpacks a strict hub-and-spoke architecture where one orchestrator delegates to isolated sub-agents (a web searcher, coder, and file inspector) that never communicate with each other, relying on structured shared memory instead of raw conversation replay to avoid flooding the orchestrator with unprocessed tool output. The hosts debate whether this design genuinely escapes scaling laws or just relocates capacity into the orchestrator role, with the paper's three research questions — collaborative performance, the value of chain-of-thought reasoning in the hierarchy, and whether bottlenecks trace to the orchestrator or sub-agents — framed as a direct test of that skepticism. Using the Qwen3 model family exclusively (1.7B to 32B) to isolate size as the only variable, the experiments compare direct tool-calling against full delegation across different orchestrator/sub-agent size pairings. Listeners interested in agentic AI system design, cost-efficient alternatives to frontier-scale models, and the mechanics of multi-agent coordination will find a rigorous, skeptically-argued breakdown of where model capacity actually needs to live.

Sources:

1. Can Small Agents Collaborate to Beat a Single Large Language Model? — Agata Żywot, Xinyi Chen, Yifei Yuan, Anders Søgaard, Maarten de Rijke, 2026

http://arxiv.org/abs/2601.11327

2. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation — Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Awadallah, Ryen White, Doug Burger, Chi Wang, 2023

https://scholar.google.com/scholar?q=AutoGen%3A+Enabling+Next-Gen+LLM+Applications+via+Multi-Agent+Conversation

3. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework — Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, 2023

https://scholar.google.com/scholar?q=MetaGPT%3A+Meta+Programming+for+a+Multi-Agent+Collaborative+Framework

4. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace — Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang, 2023

https://scholar.google.com/scholar?q=HuggingGPT%3A+Solving+AI+Tasks+with+ChatGPT+and+its+Friends+in+HuggingFace

5. More Agents Is All You Need — Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, Deheng Ye, 2024

https://scholar.google.com/scholar?q=More+Agents+Is+All+You+Need

6. ToolOrchestra: Elevating intelligence via efficient model and tool orchestration — Hongjin Su, Shizhe Diao, Ximing Lu, et al., 2025

https://scholar.google.com/scholar?q=ToolOrchestra%3A+Elevating+intelligence+via+efficient+model+and+tool+orchestration

7. In-the-flow agentic system optimization for effective planning and tool use — Zhuofeng Li, Haoxiang Zhang, Seungju Han, et al., 2025

https://scholar.google.com/scholar?q=In-the-flow+agentic+system+optimization+for+effective+planning+and+tool+use

8. When single-agent with skills replace multi-agent systems and when they fail — Xiaoxiao Li, 2026

https://scholar.google.com/scholar?q=When+single-agent+with+skills+replace+multi-agent+systems+and+when+they+fail

9. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity — Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, et al., 2025

https://scholar.google.com/scholar?q=The+illusion+of+thinking%3A+Understanding+the+strengths+and+limitations+of+reasoning+models+via+the+lens+of+problem+complexity

10. Multi-agent architecture search via agentic supernet (MaAS) — Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, Xiang Wang, 2025

https://scholar.google.com/scholar?q=Multi-agent+architecture+search+via+agentic+supernet+%28MaAS%29

11. The curse of CoT: On the limitations of chain-of-thought in in-context learning — Tianshi Zheng, Yixiang Chen, Chengxi Li, et al., 2025

https://scholar.google.com/scholar?q=The+curse+of+CoT%3A+On+the+limitations+of+chain-of-thought+in+in-context+learning

This episode examines "Scaling Large Language Model-based Multi-Agent Collaboration," which pushes agent-based collaboration to over a thousand coordinated LLM agents, far beyond the handful typically used in prior systems. The discussion traces the paper's intellectual roots in neural scaling laws (Kaplan et al., Chinchilla) and emergent abilities research (Wei et al.), applying that same lens to agent count rather than parameter count. It unpacks MACNET's architecture — a directed acyclic graph where actor agents produce artifacts and critic agents supervise via edges — and how this structure sidesteps the context-window blowup that stalled earlier multi-agent work. The centerpiece is the "collaborative scaling law," a fitted sigmoid-like curve showing performance barely moves for early agent-count doublings, then rises sharply before saturating around sixteen agents. Listeners interested in whether collaboration scales like model capacity will find the empirical curve-fitting and topology comparisons (chain, tree, star, mesh, layer, random) a compelling test of an underexplored hypothesis.

Sources:

1. Scaling Large Language Model-based Multi-Agent Collaboration — Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, Maosong Sun, 2024

http://arxiv.org/abs/2406.07155

2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, et al. (OpenAI), 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

3. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, et al. (DeepMind), 2022

https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models

4. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, et al. (Google Research / Stanford / DeepMind / UNC), 2022

https://scholar.google.com/scholar?q=Emergent+Abilities+of+Large+Language+Models

5. Improving Factuality and Reasoning in Language Models through Multiagent Debate — Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch, 2023

https://scholar.google.com/scholar?q=Improving+Factuality+and+Reasoning+in+Language+Models+through+Multiagent+Debate

6. Collective dynamics of 'small-world' networks — Duncan J. Watts, Steven H. Strogatz, 1998

https://scholar.google.com/scholar?q=Collective+dynamics+of+%27small-world%27+networks

7. Emergence of Scaling in Random Networks — Albert-László Barabási, Réka Albert, 1999

https://scholar.google.com/scholar?q=Emergence+of+Scaling+in+Random+Networks

8. ChatDev: Communicative Agents for Software Development — Chen Qian, Xin Cong, Wei Liu, et al. (Tsinghua University), 2023-2024

https://scholar.google.com/scholar?q=ChatDev%3A+Communicative+Agents+for+Software+Development

9. GPTSwarm: Language Agents as Optimizable Graphs — Mingchen Zhuge, Wenyi Wang, Louis Kirsch, et al., 2024

https://scholar.google.com/scholar?q=GPTSwarm%3A+Language+Agents+as+Optimizable+Graphs

10. Are Emergent Abilities of Large Language Models a Mirage? — Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2024 (NeurIPS)

https://scholar.google.com/scholar?q=Are+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F

11. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems — Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, James Zou, 2024

https://scholar.google.com/scholar?q=Are+More+LLM+Calls+All+You+Need%3F+Towards+Scaling+Laws+of+Compound+Inference+Systems

12. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini, 2024

https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling

13. Experiential Co-Learning of Software-Developing Agents / Iterative Experience Refinement of Software-Developing Agents — Chen Qian, Yufan Dang, Jiahao Li, et al. (same lab/authors as MACNET, incl. ChatDev), 2024

https://scholar.google.com/scholar?q=Experiential+Co-Learning+of+Software-Developing+Agents+%2F+Iterative+Experience+Refinement+of+Software-Developing+Agents

14. Collective Dynamics of Small-World Networks — Duncan J. Watts, Steven H. Strogatz, 1998 (Nature)

https://scholar.google.com/scholar?q=Collective+Dynamics+of+Small-World+Networks

This episode examines "How Far Can Disaggregation Go?", a design-space study of attention–FFN disaggregation (AFD) for serving mixture-of-experts models. It places AFD at the top of a ladder that starts with chunked-prefill aggregation and continues through prefill–decode disaggregation. It explains why attention (memory-bound, KV-cache heavy) and expert FFNs (compute-bound, needing pooled batches) want different hardware, and what that costs in per-layer dispatch and combine traffic. It covers how the attention-to-FFN GPU ratio is chosen, and how the balance shifts by model. Dense GQA models become attention-dominated as context grows, while DeepSeek-V3.2 stays FFN-heavy until long context. The discussion also questions the headline claim that AFD sustains around 4,000 tokens per second under strict latency targets where the alternatives are infeasible. "Feasible" depends on the search space and the latency caps, and AFD is not the best option for throughput. It flags limits in the evidence: the 128-GPU B200 results come from a simulator built on measured kernel data rather than real runs. The 524k-token agentic workload is synthetic, and the heterogeneous-hardware motivation is never tested.

Sources:

1. How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving — Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee, Tuhin Khare, Sudarshan Srinivasan, Suvinay Subramanian, Souvik Kundu, Madhu Kumar, Midhilesh Elavazhagan, William Won, Amir Yazdanbakhsh, Tushar Krishna, 2026

http://arxiv.org/abs/2605.28302

2. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 (OSDI 2024)

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024 (OSDI 2024; earlier arXiv version as SARATHI, 2023)

https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve

4. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, Xin Liu (ByteDance and Peking University), 2025

https://scholar.google.com/scholar?q=MegaScale-Infer%3A+Serving+Mixture-of-Experts+at+Scale+with+Disaggregated+Expert+Parallelism

5. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding — StepFun team (Step-3 authors), 2025

https://scholar.google.com/scholar?q=Step-3+is+Large+yet+Affordable%3A+Model-system+Co-design+for+Cost-effective+Decoding

6. Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures — Chenggang Zhao et al. (DeepSeek-AI), 2025

https://scholar.google.com/scholar?q=Insights+into+DeepSeek-V3%3A+Scaling+Challenges+and+Reflections+on+Hardware+for+AI+Architectures

7. DeepSeek-V3 Technical Report / DeepSeek-V3.2-Exp (DeepSeek Sparse Attention) — DeepSeek-AI, 2024 / 2025

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+%2F+DeepSeek-V3.2-Exp+%28DeepSeek+Sparse+Attention%29

8. Serving Large Language Models on Huawei CloudMatrix384 — Pengfei Zuo et al., 2025

https://scholar.google.com/scholar?q=Serving+Large+Language+Models+on+Huawei+CloudMatrix384

9. NanoFlow: Towards Optimal Large Language Model Serving Throughput — Kan Zhu et al., 2024

https://scholar.google.com/scholar?q=NanoFlow%3A+Towards+Optimal+Large+Language+Model+Serving+Throughput

10. Vidur: A Large-Scale Simulation Framework for LLM Inference — Amey Agrawal et al., 2024

https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+for+LLM+Inference

11. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale — William Won et al., 2023

https://scholar.google.com/scholar?q=ASTRA-sim2.0%3A+Modeling+Hierarchical+Networks+and+Disaggregated+Systems+for+Large-model+Training+at+Scale

12. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

13. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025

https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference

14. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity — Tyler Griggs et al., 2024

https://scholar.google.com/scholar?q=M%C3%A9lange%3A+Cost+Efficient+Large+Language+Model+Serving+by+Exploiting+GPU+Heterogeneity

15. Demystifying Platform Requirements for Diverse LLM Inference Use Cases (GenZ) — Abhimanyu Bambhaniya et al., 2024

https://scholar.google.com/scholar?q=Demystifying+Platform+Requirements+for+Diverse+LLM+Inference+Use+Cases+%28GenZ%29

This episode examines subliminal learning, the phenomenon where a student model absorbs a teacher's hidden traits through outputs that contain no explicit reference to them, and asks whether the effect survives realistic multi-step training rather than just a single gradient update. Using a plain MLP with auxiliary output logits distilled into a same-architecture student, the discussion traces how prior work explained the leak through gradient alignment at a shared initialization, and how the transfer vanishes entirely when teacher and student don't share that starting point. The paper being covered extends this by tracking gradient alignment across a hundred training seeds over full training runs, testing whether the effect persists as parameters drift, and evaluating a proposed mitigation from recent liminal-training research. The stakes are practical: knowledge distillation from shared-initialization checkpoints is standard industry practice, meaning a clean-looking training dataset could still transfer unintended or misaligned behavior into deployed models. Listeners interested in AI safety, model distillation pipelines, or the mechanics of how neural networks encode information beyond their explicit training signal will find the mechanism-level investigation compelling.

Sources:

1. Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment — Chayanon Kitkana, Shivam Arora, 2026

http://arxiv.org/abs/2604.25779

2. Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data — Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans, 2025

https://scholar.google.com/scholar?q=Subliminal+Learning%3A+Language+Models+Transmit+Behavioral+Traits+via+Hidden+Signals+in+Data

3. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

4. Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Jan Betley, Daniel Tan, and collaborators including members of Owain Evans's group, 2025

https://scholar.google.com/scholar?q=Emergent+Misalignment%3A+Narrow+Finetuning+Can+Produce+Broadly+Misaligned+LLMs

5. Gradient Surgery for Multi-Task Learning — Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn, 2020

https://scholar.google.com/scholar?q=Gradient+Surgery+for+Multi-Task+Learning

6. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks — Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, Andrew Rabinovich, 2018

https://scholar.google.com/scholar?q=GradNorm%3A+Gradient+Normalization+for+Adaptive+Loss+Balancing+in+Deep+Multitask+Networks

7. Efficiently Identifying Task Groupings for Multi-Task Learning — Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, Chelsea Finn, 2021

https://scholar.google.com/scholar?q=Efficiently+Identifying+Task+Groupings+for+Multi-Task+Learning

8. Liminal Training: Characterizing and Mitigating Subliminal Learning in Large Language Models — Atsushi Yanagisawa, Akbarzaib Khan, Thanjeetraaj Kaur Balraj Singh, Yunjong Na, Kevin Zhu, Antonio Mari, 2025

https://scholar.google.com/scholar?q=Liminal+Training%3A+Characterizing+and+Mitigating+Subliminal+Learning+in+Large+Language+Models

9. Gradient Surgery for Multi-Task Learning (PCGrad) — Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn, 2020

https://scholar.google.com/scholar?q=Gradient+Surgery+for+Multi-Task+Learning+%28PCGrad%29

This episode examines "Learning Through Noise: Why Subliminal Learning Works and When It Fails," which investigates how a student network can absorb a teacher's knowledge purely by imitating its outputs on random, task-irrelevant noise—no labeled data required. The discussion traces this back to 2025 research showing hidden traits like animal preferences can transfer between models through seemingly innocuous number sequences, raising alarms for the now-ubiquitous practice of knowledge distillation. It unpacks the prevailing explanation—a proven theorem that shared weight initialization between teacher and student drives this transfer—and previews how the featured paper challenges that assumption using a cleverly designed MNIST experiment with surgically split output heads to isolate the mechanism. The stakes are significant: if the "dark knowledge" channel isn't confined to near-identical model pairs, biases, misalignment, or backdoors could silently propagate through distilled models in ways current data-filtering practices can't catch. Listeners interested in AI safety, model distillation pipelines, or the mechanics of how neural networks transmit information beyond their explicit training signal will find the paper's methodology and its safety implications compelling.

Sources:

1. Learning Through Noise: Why Subliminal Learning Works and When It Fails — Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus, Belén Hidalgo-Ogalde, Viola Priesemann, 2026

http://arxiv.org/abs/2605.23645

2. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

3. Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data — Alex Cloud, et al., 2025

https://scholar.google.com/scholar?q=Subliminal+Learning%3A+Language+Models+Transmit+Behavioral+Traits+via+Hidden+Signals+in+Data

4. Knowledge Distillation: A Survey — Jianping Gou, Baosheng Yu, Stephen Maybank, Dacheng Tao, 2021

https://scholar.google.com/scholar?q=Knowledge+Distillation%3A+A+Survey

5. Planting Undetectable Backdoors in Machine Learning Models — Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, Or Zamir, 2022

https://scholar.google.com/scholar?q=Planting+Undetectable+Backdoors+in+Machine+Learning+Models

6. Language models transmit behavioural traits through hidden signals in data — Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Sören Mindermann, Jacob Hilton, Samuel Marks, Owain Evans, 2026 (Nature)

https://scholar.google.com/scholar?q=Language+models+transmit+behavioural+traits+through+hidden+signals+in+data

7. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs — Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans, 2025 (ICML)

https://scholar.google.com/scholar?q=Emergent+misalignment%3A+Narrow+finetuning+can+produce+broadly+misaligned+LLMs

8. Sleeper agents: Training deceptive LLMs that persist through safety training — Evan Hubinger et al., 2024

https://scholar.google.com/scholar?q=Sleeper+agents%3A+Training+deceptive+LLMs+that+persist+through+safety+training

9. Shared global and local geometry of language model embeddings — Andrew Lee, Melanie Weber, Fernanda Viégas, Martin Wattenberg, 2025

https://scholar.google.com/scholar?q=Shared+global+and+local+geometry+of+language+model+embeddings

10. Training-free tokenizer transplantation via orthogonal matching pursuit — Charles Goddard, Fernando Fernandes Neto, 2025

https://scholar.google.com/scholar?q=Training-free+tokenizer+transplantation+via+orthogonal+matching+pursuit

11. Prevalence of neural collapse during the terminal phase of deep learning training — Vardan Papyan, X. Y. Han, David L. Donoho, 2020 (PNAS)

https://scholar.google.com/scholar?q=Prevalence+of+neural+collapse+during+the+terminal+phase+of+deep+learning+training

This episode examines "Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity," a Meta FAIR paper (with Inria, ENS-PSL, and Tübingen collaborators) that tackles the memory-capacity wall in linear-attention architectures. The discussion traces the lineage from Katharopoulos-style linear attention through Gated DeltaNet's delta-rule updates to Product-Key Memory's sparse addressing, showing how the paper combines them: applying a gated delta update over a product-key-addressed memory table so only a handful of slots are read and written per token, letting the state size grow a thousandfold with no added FLOPs. It also covers the paper's second contribution — making the initial memory state a learned parameter so pretrained knowledge can live directly in that expanded memory rather than only being built up in-context. Throughout, the hosts stress a key conceptual distinction between compressed fixed-size RNN state and a growing transformer KV cache, framing the paper's core question as whether sparse addressing can close the long-context recall gap with transformers without touching compute or parameter budgets. Listeners interested in efficient long-context architectures will find the mechanistic walkthrough and the 8-billion-parameter validation particularly relevant to real bottlenecks in agentic coding and long-document workloads.

Sources:

1. Sparse Delta Memory: Scaling Linear RNN State Without Extra Compute

https://arxiv.org/pdf/2607.07386v1

2. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

3. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024

https://scholar.google.com/scholar?q=Memory+Layers+at+Scale

4. Fast-Weight Product Key Memory — Tianyu Zhao, Llion Jones, 2026

https://scholar.google.com/scholar?q=Fast-Weight+Product+Key+Memory

5. Simple linear attention language models balance the recall-throughput tradeoff — Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, Christopher Ré, 2025

https://scholar.google.com/scholar?q=Simple+linear+attention+language+models+balance+the+recall-throughput+tradeoff

6. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes — Jack W. Rae, Jonathan J. Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, Timothy P. Lillicrap, 2016

https://scholar.google.com/scholar?q=Scaling+Memory-Augmented+Neural+Networks+with+Sparse+Reads+and+Writes

7. Short window attention enables long-term memorization — Loïc Cabannes, Maximilian Beck, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Jade Copet, Pierre-Emmanuel Mazaré, Gabriel Synnaeve, Hervé Jégou, 2025

https://scholar.google.com/scholar?q=Short+window+attention+enables+long-term+memorization

This episode examines a paper arguing that frontier LLMs can do useful reasoning that never appears in their chain-of-thought. Inserting an identical filler string, such as "count one to a hundred," between the question and the answer raises accuracy on tasks like 4-digit multiplication, nested arithmetic, and variable counting across 13 models. The discussion separates this "invisible reasoning" from unfaithful explanations and steganography, since here the tokens carry nothing to decode. It contrasts the finding with earlier dummy-token and pause-token work, which needed special training, and notes the authors' hedge that the uplift may not be extra computation alone. It also covers the safety stakes for chain-of-thought monitoring, the efficiency incentives for labs, and which filler types help or hurt across models. Listeners interested in AI oversight and what reading a model's reasoning can actually reveal will find it useful.

Sources:

1. Not All LLM Reasoning is Visible in the Chain-of-Thought — Vatsal Baherwani, Tom Goldstein, Ashwinee Panda, 2026

http://arxiv.org/abs/2607.22925

2. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models — Jacob Pfau, William Merrill, Samuel R. Bowman, 2024

https://scholar.google.com/scholar?q=Let%27s+Think+Dot+by+Dot%3A+Hidden+Computation+in+Transformer+Language+Models

3. Think before you speak: Training Language Models With Pause Tokens — Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan, 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=Think+before+you+speak%3A+Training+Language+Models+With+Pause+Tokens

4. The Expressive Power of Transformers with Chain of Thought — William Merrill, Ashish Sabharwal, 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought

5. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024

https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space+%28Coconut%29

6. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, and many co-authors from multiple labs and institutes, 2025

https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety

7. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, David Farhi, 2025

https://scholar.google.com/scholar?q=Monitoring+Reasoning+Models+for+Misbehavior+and+the+Risks+of+Promoting+Obfuscation

8. Reasoning Models Don't Always Say What They Think — Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez, 2025

https://scholar.google.com/scholar?q=Reasoning+Models+Don%27t+Always+Say+What+They+Think

9. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman, 2023

https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+Prompting

10. Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance — Ryan Greenblatt, 2025

https://scholar.google.com/scholar?q=Recent+LLMs+can+use+filler+tokens+or+problem+repeats+to+improve+%28no-CoT%29+math+performance

11. Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning — Zeru Shi, Yingjia Wan, Zhenting Wang, Qifan Wang, Fan Yang, Elisa Kreiss, Ruixiang Tang, 2025

https://scholar.google.com/scholar?q=Meaningless+Tokens%2C+Meaningful+Gains%3A+How+Activation+Shifts+Enhance+LLM+Reasoning

12. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Zhiyuan Li, Hong Liu, Denny Zhou, Tengyu Ma, 2024

https://scholar.google.com/scholar?q=Chain+of+Thought+Empowers+Transformers+to+Solve+Inherently+Serial+Problems

13. Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability — Artur Zolkowski, Wen Xing, David Lindner, Florian Tramèr, Erik Jenner, 2025

https://scholar.google.com/scholar?q=Can+Reasoning+Models+Obfuscate+Reasoning%3F+Stress-Testing+Chain-of-Thought+Monitorability

14. Lessons from Studying Two-Hop Latent Reasoning — Mikita Balesni, Tomek Korbak, Owain Evans, 2025

https://scholar.google.com/scholar?q=Lessons+from+Studying+Two-Hop+Latent+Reasoning

15. Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought — Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo, 2026

https://scholar.google.com/scholar?q=Reasoning+Theater%3A+Disentangling+Model+Beliefs+from+Chain-of-Thought

16. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025

https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach

17. Auditing Language Models for Hidden Objectives — Samuel Marks et al., 2025

https://scholar.google.com/scholar?q=Auditing+Language+Models+for+Hidden+Objectives

18. Preventing Language Models from Hiding Their Reasoning — Fabien Roger, Ryan Greenblatt, 2023

https://scholar.google.com/scholar?q=Preventing+Language+Models+from+Hiding+Their+Reasoning

This episode examines HijackKV, an integrity attack on position-independent KV cache reuse, where a clean question and a clean retrieved document still yield an attacker's chosen answer because the stored attention state was poisoned earlier. It explains why prefix caching is safe by construction, why position-independent systems such as Prompt Cache, CacheBlend, and EPIC give that up to raise hit rates for retrieval-augmented generation, and how the resulting "attention shift" mismatch, long treated only as an accuracy problem, becomes an unkeyed-input hole much like web cache poisoning. The discussion contrasts the attack with indirect prompt injection and with timing-based cache privacy attacks, and sets out a threat model in which the attacker is an ordinary user who optimizes a prefix with a GCG-style gradient search. It also defines the untargeted and targeted attack-success metrics and tests the abstract's claims of roughly 94% one-shot success and survival under 50% recomputation against the paper's tables. Listeners get a clear look at a security risk in shared LLM serving caches that no text filter can see.

Sources:

1. HijackKV: Poisoning Position-Independent KV Cache Reuse in RAG Serving

https://arxiv.org/pdf/2607.19957

2. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (EuroSys 2025)

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

4. Universal and Transferable Adversarial Attacks on Aligned Language Models — Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson, 2023

https://scholar.google.com/scholar?q=Universal+and+Transferable+Adversarial+Attacks+on+Aligned+Language+Models

5. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz, 2023

https://scholar.google.com/scholar?q=Not+what+you%27ve+signed+up+for%3A+Compromising+Real-World+LLM-Integrated+Applications+with+Indirect+Prompt+Injection

6. Auditing Prompt Caching in Language Model APIs — Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, Tatsunori Hashimoto, 2025

https://scholar.google.com/scholar?q=Auditing+Prompt+Caching+in+Language+Model+APIs

7. The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems — Linke Song, Zixuan Pang, Wenhao Wang, Zihao Wang, XiaoFeng Wang, Hongbo Chen, Wei Song, Yier Jin, Dan Meng, Rui Hou, 2024

https://scholar.google.com/scholar?q=The+Early+Bird+Catches+the+Leak%3A+Unveiling+Timing+Side+Channels+in+LLM+Serving+Systems

8. Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving — Guanlong Wu, Zheng Zhang, Yao Zhang, Weili Wang, Jianyu Niu, Ye Wu, Yinqian Zhang, 2025 (NDSS)

https://scholar.google.com/scholar?q=Prompt+Leakage+via+KV-Cache+Sharing+in+Multi-Tenant+LLM+Serving

9. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu et al., 2025

https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models

10. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2024

https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference

11. Block-Attention for Efficient Prefilling / TurboRAG / APE (Adaptive Parallel Encoding) — Dongyang Ma et al.; Songshuo Lu et al.; Xinyu Yang et al., 2024-2025

https://scholar.google.com/scholar?q=Block-Attention+for+Efficient+Prefilling+%2F+TurboRAG+%2F+APE+%28Adaptive+Parallel+Encoding%29

12. I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving — Guanlong Wu, Zheng Zhang, Yao Zhang, Weili Wang, Jianyu Niu, Ye Wu, Yinqian Zhang, 2025

https://scholar.google.com/scholar?q=I+Know+What+You+Asked%3A+Prompt+Leakage+via+KV-Cache+Sharing+in+Multi-Tenant+LLM+Serving

13. Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG) — Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson, 2023

https://scholar.google.com/scholar?q=Universal+and+Transferable+Adversarial+Attacks+on+Aligned+Language+Models+%28GCG%29

14. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models — Wei Zou, Runpeng Geng, Binghui Wang, Jinyuan Jia, 2025 (USENIX Security)

https://scholar.google.com/scholar?q=PoisonedRAG%3A+Knowledge+Corruption+Attacks+to+Retrieval-Augmented+Generation+of+Large+Language+Models

15. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study

16. Prompt Infection / Not what you've signed up for: indirect prompt injection — Kai Greshake et al.; Donghyun Lee and Mugur Tiwari, 2023; 2024

https://scholar.google.com/scholar?q=Prompt+Infection+%2F+Not+what+you%27ve+signed+up+for%3A+indirect+prompt+injection

17. CacheGen / CacheBlend lineage and RAGCache — Yuhan Liu et al.; Chao Jin et al., 2023-2024

https://scholar.google.com/scholar?q=CacheGen+%2F+CacheBlend+lineage+and+RAGCache

18. vLLM prefix caching with cache_salt and per-organization cache isolation in provider APIs — vLLM project; OpenAI/Anthropic documentation, 2024-2025

https://scholar.google.com/scholar?q=vLLM+prefix+caching+with+cache_salt+and+per-organization+cache+isolation+in+provider+APIs

This episode examines CacheTrap, a gray-box attack that flips a single bit in an LLM's transient KV cache to force a chosen output, without touching the model's weights or inputs and without access to the victim's fine-tuned weights, training data, or gradients. It explains how the KV cache works, why flipping the top exponent bit of a 16-bit float can scale one value by roughly 65,000, and how the attack reuses the Rowhammer-style GPU fault primitive from GPUHammer. It contrasts this with training-time backdoors and gradient-guided weight bit-flip attacks, and notes that the paper doesn't compare against PrisonBreak. It also covers how the attacker picks a target without gradients, using a forward-pass layer sensitivity score and the last prefix token. Listeners get a clear grounding in the attack's mechanics and a skeptical look at its "stealthy Trojan" framing and the assumptions behind it.

Sources:

1. CacheTrap: Unveiling a Stealthier Gray-Box Trojan against LLMs — Mohaiminul Al Nahian, Abeer Matar A. Almalky, Gamana Aragonda, Ranyang Zhou, Sabbir Ahmed, Dmitry Ponomarev, Li Yang, Shaahin Angizi, Adnan Siraj Rakin, 2025

http://arxiv.org/abs/2511.22681

2. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain — Tianyu Gu, Brendan Dolan-Gavitt, Siddharth Garg, 2017

https://scholar.google.com/scholar?q=BadNets%3A+Identifying+Vulnerabilities+in+the+Machine+Learning+Model+Supply+Chain

3. Poisoning Language Models During Instruction Tuning — Alexander Wan, Eric Wallace, Sheng Shen, Dan Klein, 2023

https://scholar.google.com/scholar?q=Poisoning+Language+Models+During+Instruction+Tuning

4. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Evan Hubinger et al. (Anthropic), 2024

https://scholar.google.com/scholar?q=Sleeper+Agents%3A+Training+Deceptive+LLMs+that+Persist+Through+Safety+Training

5. BadEdit: Backdooring Large Language Models by Model Editing — Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, Yang Liu, 2024

https://scholar.google.com/scholar?q=BadEdit%3A+Backdooring+Large+Language+Models+by+Model+Editing

6. Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors — Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, Onur Mutlu, 2014

https://scholar.google.com/scholar?q=Flipping+Bits+in+Memory+Without+Accessing+Them%3A+An+Experimental+Study+of+DRAM+Disturbance+Errors

7. Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search — Adnan Siraj Rakin, Zhezhi He, Deliang Fan, 2019

https://scholar.google.com/scholar?q=Bit-Flip+Attack%3A+Crushing+Neural+Network+with+Progressive+Bit+Search

8. Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks — Sanghyun Hong, Pietro Frigo, Yiğitcan Kaya, Cristiano Giuffrida, Tudor Dumitraș, 2019

https://scholar.google.com/scholar?q=Terminal+Brain+Damage%3A+Exposing+the+Graceless+Degradation+in+Deep+Neural+Networks+Under+Hardware+Fault+Attacks

9. PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips — Zachary Coalson, Jeonghyun Woo, Yu Sun, Shiyang Chen, Lishan Yang, Prashant Nair, Bo Fang, Sanghyun Hong, 2024

https://scholar.google.com/scholar?q=PrisonBreak%3A+Jailbreaking+Large+Language+Models+with+Fewer+Than+Twenty-Five+Targeted+Bit-flips

10. Practical Black-Box Attacks against Machine Learning — Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami, 2017

https://scholar.google.com/scholar?q=Practical+Black-Box+Attacks+against+Machine+Learning

11. Stealing Machine Learning Models via Prediction APIs — Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, Thomas Ristenpart, 2016

https://scholar.google.com/scholar?q=Stealing+Machine+Learning+Models+via+Prediction+APIs

12. Universal and Transferable Adversarial Attacks on Aligned Language Models — Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson, 2023

https://scholar.google.com/scholar?q=Universal+and+Transferable+Adversarial+Attacks+on+Aligned+Language+Models

13. Stealing Part of a Production Language Model — Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, Florian Tramèr, 2024

https://scholar.google.com/scholar?q=Stealing+Part+of+a+Production+Language+Model

14. GPUHammer: Rowhammer Attacks on GPU Memories are Practical — Chris S. Lin, Joyce Qu, Gururaj Saileshwar, 2025

https://scholar.google.com/scholar?q=GPUHammer%3A+Rowhammer+Attacks+on+GPU+Memories+are+Practical

15. GDDRHammer: Greatly Disturbing DRAM Rows—Cross-Component Rowhammer Attacks from Modern GPUs — Y. Hu, N. Brown, Y. Chen, J. Bakita, T. Chen, D. Genkin, A. Kwong, 2026

https://scholar.google.com/scholar?q=GDDRHammer%3A+Greatly+Disturbing+DRAM+Rows%E2%80%94Cross-Component+Rowhammer+Attacks+from+Modern+GPUs

16. SBFA: Single Sneaky Bit Flip Attack to Break Large Language Models — J. Guo, C. Chakrabarti, D. Fan, 2025

https://scholar.google.com/scholar?q=SBFA%3A+Single+Sneaky+Bit+Flip+Attack+to+Break+Large+Language+Models

17. Massive Activations in Large Language Models — Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu, 2024

https://scholar.google.com/scholar?q=Massive+Activations+in+Large+Language+Models

18. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

19. Ranger: Boosting Error Resilience of Deep Neural Networks through Range Restriction — Zitao Chen, Guanpeng Li, Karthik Pattabiraman, 2021

https://scholar.google.com/scholar?q=Ranger%3A+Boosting+Error+Resilience+of+Deep+Neural+Networks+through+Range+Restriction

20. I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving — G. Wu et al., 2025

https://scholar.google.com/scholar?q=I+Know+What+You+Asked%3A+Prompt+Leakage+via+KV-Cache+Sharing+in+Multi-Tenant+LLM+Serving

21. ECC.fail / ECCploit — Kamadan et al. 2025; Cojocar et al. 2019, 2025/2019

https://scholar.google.com/scholar?q=ECC.fail+%2F+ECCploit