This episode explores the UIST 2025 paper "Creating General User Models from Computer Use," which proposes building a persistent user model from raw computer traces such as screenshots, UI text, message context, and app switching. It explains how the system stores confidence-weighted natural-language propositions about a person’s preferences, knowledge, goals, and current situation, aiming to support cross-application assistants that can help proactively rather than waiting for explicit requests. The discussion situates the idea against earlier recommender systems, Bayesian user modeling, and newer LLM memory architectures, arguing that the paper is most interesting as a synthesis of HCI user modeling and retrieval-and-revision style AI memory. Listeners would find it compelling because it gets concrete about both the upside of more context-aware assistants and the hard problems underneath them, including noisy behavioral data, narrow evidence relative to broad claims, and the privacy risks of inferring things users never said aloud.

Sources:

1. Creating General User Models from Computer Use — Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein, 2025

http://arxiv.org/abs/2505.10831

2. User Modeling via Stereotypes — Elaine Rich, 1979

https://scholar.google.com/scholar?q=User+Modeling+via+Stereotypes

3. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users — Eric J. Horvitz, John S. Breese, David Heckerman, David Hovel, Koos Rommelse, 1998

https://scholar.google.com/scholar?q=The+Lumiere+Project%3A+Bayesian+User+Modeling+for+Inferring+the+Goals+and+Needs+of+Software+Users

4. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions — Gediminas Adomavicius, Alexander Tuzhilin, 2005

https://scholar.google.com/scholar?q=Toward+the+Next+Generation+of+Recommender+Systems%3A+A+Survey+of+the+State-of-the-Art+and+Possible+Extensions

5. User Modeling and User Profiling: A Comprehensive Survey — Erasmo Purificato, Ludovico Boratto, Ernesto William De Luca, 2024

https://scholar.google.com/scholar?q=User+Modeling+and+User+Profiling%3A+A+Comprehensive+Survey

6. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2023

https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior

7. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023

https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems

8. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration — Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, Diyi Yang, 2024

https://scholar.google.com/scholar?q=Collaborative+Gym%3A+A+Framework+for+Enabling+and+Evaluating+Human-Agent+Collaboration

9. Supporting Physical Activity Behavior Change with LLM-Based Conversational Agents — Matthew Jörke, Shardul Sapkota, Lyndsea Warkenthien, Niklas Vainio, Paul Schmiedmayer, Emma Brunskill, James Landay, 2024

https://scholar.google.com/scholar?q=Supporting+Physical+Activity+Behavior+Change+with+LLM-Based+Conversational+Agents

10. TaskTracer: a desktop environment to support multi-tasking knowledge workers — Anton N. Dragunov, Thomas G. Dietterich, Kevin Johnsrude, Matthew McLaughlin, Lida Li, Jonathan L. Herlocker, 2005

https://scholar.google.com/scholar?q=TaskTracer%3A+a+desktop+environment+to+support+multi-tasking+knowledge+workers

11. UI-TARS: Pioneering Automated GUI Interaction with Native Agents — Yujia Qin et al., 2025

https://arxiv.org/abs/2501.12326

12. Need Help? Designing Proactive AI Assistants for Programming — Valerie Chen et al., 2024/2025

https://arxiv.org/abs/2410.04596

13. Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants — Deepak Nathani et al., 2026

https://arxiv.org/abs/2604.00842

14. ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation — Jiho Kim et al., 2025/2026

https://arxiv.org/abs/2509.21730

15. AI Post Transformers: Learning Latent Action World Models from Video — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-09-learning-latent-action-world-models-from-1570a4.mp3

16. AI Post Transformers: PaperBench: Can AI Replicate AI Research? — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-17-paperbench-can-ai-replicate-ai-research-862944.mp3

17. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp3

18. AI Post Transformers: Neural Computers as Learned Latent Runtimes — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-11-neural-computers-as-learned-latent-runti-9fa282.mp3

Interactive Visualization: Building General User Models from Computer Use

This episode explores a 2025 study on fine-tuning large language models to predict how people respond in social science experiments, asking whether trained models can simulate new studies more reliably than prompting alone. It explains how the researchers built SOCSCI210, a dataset of 2.9 million responses from more than 400,000 participants across 210 TESS experiments, and why standardizing those studies into respondent-condition-question-answer records is central to the method. The discussion breaks down the paper’s evaluation criteria, including out-of-distribution generalization, distribution matching via Wasserstein distance, normalized individual accuracy, and treatment-effect recovery, to show the difference between sounding plausible and preserving real experimental patterns. Listeners would find it interesting because it treats LLMs not as chatbots but as possible “wind tunnels” for testing study designs in advance, while also confronting the risk that a convincing simulator could still get causal effects wrong.

Sources:

1. Finetuning LLMs for Human Behavior Prediction in Social Science Experiments — Akaash Kolluri, Shengguang Wu, Joon Sung Park, Michael S. Bernstein, 2025

http://arxiv.org/abs/2509.05830

2. Out of One, Many: Using Language Models to Simulate Human Samples — Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, David Wingate, 2022

https://scholar.google.com/scholar?q=Out+of+One%2C+Many%3A+Using+Language+Models+to+Simulate+Human+Samples

3. LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals — Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Michael S. Bernstein, et al., 2024

https://scholar.google.com/scholar?q=LLM+Agents+Grounded+in+Self-Reports+Enable+General-Purpose+Simulation+of+Individuals

4. Large Language Models Show Human-like Social Desirability Biases in Survey Responses — Aadesh Salecha, Molly E. Ireland, Shashanka Subrahmanya, Joao Sedoc, Lyle H. Ungar, Johannes C. Eichstaedt, 2024

https://scholar.google.com/scholar?q=Large+Language+Models+Show+Human-like+Social+Desirability+Biases+in+Survey+Responses

5. Finetuning LLMs for Human Behavior Prediction in Social Science Experiments — Akaash Kolluri, Shengguang Wu, Joon Sung Park, Michael S. Bernstein, 2025

https://scholar.google.com/scholar?q=Finetuning+LLMs+for+Human+Behavior+Prediction+in+Social+Science+Experiments

6. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies — Gati Aher, Rosa I. Arriaga, Adam Tauman Kalai, 2022

https://scholar.google.com/scholar?q=Using+Large+Language+Models+to+Simulate+Multiple+Humans+and+Replicate+Human+Subject+Studies

7. Can Large Language Models Replace Human Subjects? A Large-Scale Replication of Scenario-Based Experiments in Psychology and Management — Ziyan Cui, Ning Li, Huaikang Zhou, 2024

https://scholar.google.com/scholar?q=Can+Large+Language+Models+Replace+Human+Subjects%3F+A+Large-Scale+Replication+of+Scenario-Based+Experiments+in+Psychology+and+Management

8. Using Large Language Models to Create AI Personas for Replication, Generalization and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings — Leo Yeykelis, Kaavya Pichai, James J. Cummings, Byron Reeves, 2024

https://scholar.google.com/scholar?q=Using+Large+Language+Models+to+Create+AI+Personas+for+Replication%2C+Generalization+and+Prediction+of+Media+Effects%3A+An+Empirical+Test+of+133+Published+Experimental+Research+Findings

9. This human study did not involve human subjects: Validating LLM simulations as behavioral evidence — Jessica Hullman, David Broska, Huaman Sun, Aaron Shaw, 2026

https://scholar.google.com/scholar?q=This+human+study+did+not+involve+human+subjects%3A+Validating+LLM+simulations+as+behavioral+evidence

10. Centaur: a Foundation Model of Human Cognition — Marcel Binz et al., 2024

https://scholar.google.com/scholar?q=Centaur%3A+a+Foundation+Model+of+Human+Cognition

11. Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions — Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, Serina Chang, 2025

https://scholar.google.com/scholar?q=Language+Model+Fine-Tuning+on+Scaled+Survey+Data+for+Predicting+Distributions+of+Public+Opinions

12. Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions — Matthias Orlikowski, Jiaxin Pei, Paul Rottger, Philipp Cimiano, David Jurgens, Dirk Hovy, 2025

https://scholar.google.com/scholar?q=Beyond+Demographics%3A+Fine-tuning+Large+Language+Models+to+Predict+Individuals%27+Subjective+Text+Perceptions

13. Large Language Models that Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups — Angelina Wang, Jamie Morgenstern, John P. Dickerson, 2025

https://scholar.google.com/scholar?q=Large+Language+Models+that+Replace+Human+Participants+Can+Harmfully+Misportray+and+Flatten+Identity+Groups

14. Beyond Believability: Accurate Human Behavior Simulation with Fine-Tuned LLMs — Yuxuan Lu et al., 2025

https://scholar.google.com/scholar?q=Beyond+Believability%3A+Accurate+Human+Behavior+Simulation+with+Fine-Tuned+LLMs

15. The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models — Marlene Lutz et al., 2025

https://arxiv.org/abs/2507.16076

16. Prompt Fairness: Sub-group Disparities in LLMs — Meiyu Zhong, Noel Teku, Ravi Tandon, 2025

https://arxiv.org/abs/2511.19956

17. Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment — Bryan Chen Zhengyu Tan et al., 2026

https://arxiv.org/abs/2604.12851

18. Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM — Zizhao Hu, Mohammad Rostami, Jesse Thomason, 2026

https://arxiv.org/abs/2603.18507

19. Causality for Large Language Models — Anpeng Wu et al., 2024

https://arxiv.org/abs/2410.15319

20. AI Post Transformers: PaperBench: Can AI Replicate AI Research? — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-17-paperbench-can-ai-replicate-ai-research-862944.mp3

21. AI Post Transformers: When Many-Shot CoT Becomes Test-Time Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-15-when-many-shot-cot-becomes-test-time-lea-c25bfe.mp3

Interactive Visualization: Fine-Tuning LLMs for Human Behavior Prediction

This episode explores Social Simulacra, a method for using large language models to prototype entire online communities before they exist by generating synthetic members, posts, and reply threads from a community goal, rules, and a small set of seed personas. It explains why that matters for social computing: small pilots can miss emergent failures like norm drift, newcomer enculturation problems, trolling, and moderator overload, while a populated simulation can expose those dynamics much earlier. The discussion breaks down how the paper uses prompt chaining to scale a handful of personas into a larger Reddit-like population and then tests interventions such as comment removal, warnings, and rule restatements. It also argues that human-sounding text is a low bar, and that the real challenge is whether these simulations capture believable long-term behavior, incentives, and feedback loops well enough to inform product design.

Sources:

1. Social Simulacra: Creating Populated Prototypes for Social Computing Systems — Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2022

http://arxiv.org/abs/2208.04024

2. Two Case Studies of Experience Prototyping Machine Learning Systems in the Wild — Qian Yang, 2019

https://scholar.google.com/scholar?q=Two+Case+Studies+of+Experience+Prototyping+Machine+Learning+Systems+in+the+Wild

3. Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools — Stephan Schlogl, Gavin Doherty, Saturnino Luz, 2014

https://scholar.google.com/scholar?q=Wizard+of+Oz+Experimentation+for+Language+Technology+Applications%3A+Challenges+and+Tools

4. Social Simulacra: Creating Populated Prototypes for Social Computing Systems — Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2022

https://scholar.google.com/scholar?q=Social+Simulacra%3A+Creating+Populated+Prototypes+for+Social+Computing+Systems

5. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2023

https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior

6. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies — Gati Aher, Rosa I. Arriaga, Adam Tauman Kalai, 2022

https://scholar.google.com/scholar?q=Using+Large+Language+Models+to+Simulate+Multiple+Humans+and+Replicate+Human+Subject+Studies

7. Out of One, Many: Using Language Models to Simulate Human Samples — Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, David Wingate, 2022

https://scholar.google.com/scholar?q=Out+of+One%2C+Many%3A+Using+Language+Models+to+Simulate+Human+Samples

8. From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents — Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, Zhongyu Wei, 2024

https://scholar.google.com/scholar?q=From+Individual+to+Society%3A+A+Survey+on+Social+Simulation+Driven+by+Large+Language+Model-based+Agents

9. SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice — Mohit Singhal, Chen Ling, Pujan Paudel, Poojitha Thota, Nihal Kumarswamy, Gianluca Stringhini, Shirin Nilizadeh, 2022

https://scholar.google.com/scholar?q=SoK%3A+Content+Moderation+in+Social+Media%2C+from+Guidelines+to+Enforcement%2C+and+Research+to+Practice

10. ModSandbox: Facilitating Online Community Moderation Through Error Prediction and Improvement of Automated Rules — Jean Y. Song, Sangwook Lee, Jisoo Lee, Mina Kim, Juho Kim, 2022

https://scholar.google.com/scholar?q=ModSandbox%3A+Facilitating+Online+Community+Moderation+Through+Error+Prediction+and+Improvement+of+Automated+Rules

11. Shaping Online Dialogue: Examining How Community Rules Affect Discussion Structures on Reddit — Anna Fang, Wenjie Yang, Haiyi Zhu, 2023

https://scholar.google.com/scholar?q=Shaping+Online+Dialogue%3A+Examining+How+Community+Rules+Affect+Discussion+Structures+on+Reddit

12. Post Guidance for Online Communities — Manoel Horta Ribeiro, Robert West, Ryan Lewis, Sanjay Kairam, 2024

https://scholar.google.com/scholar?q=Post+Guidance+for+Online+Communities

13. Piggyback prototyping: Using existing, large-scale social computing systems to prototype new ones — Catherine Grevet and Eric Gilbert, 2015

https://scholar.google.com/scholar?q=Piggyback+prototyping%3A+Using+existing%2C+large-scale+social+computing+systems+to+prototype+new+ones

14. The Internet's Hidden Rules: An Empirical Study of Reddit Norm Violations at Micro, Meso, and Macro Scales — Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert, 2018

https://scholar.google.com/scholar?q=The+Internet%27s+Hidden+Rules%3A+An+Empirical+Study+of+Reddit+Norm+Violations+at+Micro%2C+Meso%2C+and+Macro+Scales

15. Surviving an "Eternal September": How an Online Community Managed a Surge of Newcomers — Charles Kiene, Andres Monroy-Hernandez, and Benjamin Mako Hill, 2016

https://scholar.google.com/scholar?q=Surviving+an+%22Eternal+September%22%3A+How+an+Online+Community+Managed+a+Surge+of+Newcomers

16. Building Successful Online Communities: Evidence-Based Social Design — Robert E. Kraut and Paul Resnick, 2012

https://scholar.google.com/scholar?q=Building+Successful+Online+Communities%3A+Evidence-Based+Social+Design

17. Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market — Matthew J. Salganik, Peter Sheridan Dodds, and Duncan J. Watts, 2006

https://scholar.google.com/scholar?q=Experimental+Study+of+Inequality+and+Unpredictability+in+an+Artificial+Cultural+Market

18. Measuring the Prevalence of Anti-Social Behavior in Online Communities — Joon Sung Park, Joseph Seering, and Michael S. Bernstein, 2022

https://scholar.google.com/scholar?q=Measuring+the+Prevalence+of+Anti-Social+Behavior+in+Online+Communities

19. LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models — Ivar Frisch and Mario Giulianelli, 2024

https://scholar.google.com/scholar?q=LLM+Agents+in+Interaction%3A+Measuring+Personality+Consistency+and+Linguistic+Alignment+in+Interacting+Populations+of+Large+Language+Models

20. Persona Alchemy: Designing, Evaluating, and Implementing Psychologically-Grounded LLM Agents for Diverse Stakeholder Representation — Sola Kim, Dongjune Chang, Jieshu Wang, 2025

https://scholar.google.com/scholar?q=Persona+Alchemy%3A+Designing%2C+Evaluating%2C+and+Implementing+Psychologically-Grounded+LLM+Agents+for+Diverse+Stakeholder+Representation

21. Mind the Sim2Real Gap in User Simulation for Agentic Tasks — Xuhui Zhou et al., 2026

https://scholar.google.com/scholar?q=Mind+the+Sim2Real+Gap+in+User+Simulation+for+Agentic+Tasks

22. Computational Turing Test Reveals Systematic Differences Between Human and AI Language — Nicolo Pagan, Petter Tornberg, Christopher A. Bail, Aniko Hannak, Christopher Barrie, 2025

https://scholar.google.com/scholar?q=Computational+Turing+Test+Reveals+Systematic+Differences+Between+Human+and+AI+Language

23. Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges — Patrick Taillandier et al., 2025

https://scholar.google.com/scholar?q=Integrating+LLM+in+Agent-Based+Social+Simulation%3A+Opportunities+and+Challenges

24. LLM Social Simulations Are a Promising Research Method — Jacy Reese Anthis et al., 2025

https://scholar.google.com/scholar?q=LLM+Social+Simulations+Are+a+Promising+Research+Method

25. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3

Interactive Visualization: Social Simulacra for Prototyping Online Communities

This episode explores a 2026 paper on stabilizing deep reinforcement learning by pushing an agent’s hidden representations toward an isotropic Gaussian shape. It explains how nonstationarity in RL, from shifting data distributions, bootstrapped targets, and primacy bias, can make agents overfit early experience, lose plasticity, and accumulate dormant neurons. The discussion focuses on the paper’s core argument that a round, evenly used feature space makes linear readouts easier to keep tracking as targets drift, reducing collapse and improving adaptation, and it breaks down SIGReg as a lightweight way to enforce that geometry. Listeners would find it interesting because it links an abstract idea from representation geometry to a concrete engineering problem in making RL systems more stable and trainable.

Sources:

1. Stable Deep Reinforcement Learning via Isotropic Gaussian Representations — Ali Saheb Pasand, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro, 2026

http://arxiv.org/abs/2602.19373

2. Whitening for Self-Supervised Representation Learning — Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe, 2020

https://scholar.google.com/scholar?q=Whitening+for+Self-Supervised+Representation+Learning

3. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021

https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning

4. The Dormant Neuron Phenomenon in Deep Reinforcement Learning — Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, Utku Evci, 2023

https://scholar.google.com/scholar?q=The+Dormant+Neuron+Phenomenon+in+Deep+Reinforcement+Learning

5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025

https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics

6. The Primacy Bias in Deep Reinforcement Learning — Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron Courville, 2022

https://scholar.google.com/scholar?q=The+Primacy+Bias+in+Deep+Reinforcement+Learning

7. No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO — Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, Caglar Gulcehre, 2024

https://scholar.google.com/scholar?q=No+Representation%2C+No+Trust%3A+Connecting+Representation%2C+Collapse%2C+and+Trust+Issues+in+PPO

8. Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning — Roger Creus Castanyer, Johan Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, Pablo Samuel Castro, 2025

https://scholar.google.com/scholar?q=Stable+Gradients+for+Stable+Learning+at+Scale+in+Deep+Reinforcement+Learning

9. Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks — Jesse Farebrother et al., 2023

https://scholar.google.com/scholar?q=Proto-Value+Networks%3A+Scaling+Representation+Learning+with+Auxiliary+Tasks

10. Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics — Raphael Bernas et al., 2026

https://scholar.google.com/scholar?q=Revisiting+Anisotropy+in+Language+Transformers%3A+The+Geometry+of+Learning+Dynamics

11. Emergence of Quantised Representations Isolated to Anisotropic Functions — George Bird, 2025

https://scholar.google.com/scholar?q=Emergence+of+Quantised+Representations+Isolated+to+Anisotropic+Functions

12. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3

13. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp3

Hal Turing and Dr. Ada Shannon take a deep dive into EMO: Pretraining Mixture of Experts for Emergent Modularity, a May 7, 2026 paper by Ryan Wang and co-authors from UC Berkeley and the Allen Institute for AI. The episode centers on a practical deployment question: if a workload is mostly code, math, or biomed, why must operators keep an entire giant model in memory instead of loading only the relevant slice? They frame EMO against the broader rise of sparse Mixture-of-Experts systems and explain why industry progress on active-parameter efficiency is not the same as delivering clean, domain-specific modules that can stand on their own at inference time.

The discussion carefully separates standard MoE behavior from the stronger notion of modularity that EMO is targeting. Hal and Ada walk through how sparse-gated MoE and Switch Transformer style routing already allow different tokens to activate different experts, but argue that this still leaves deployment looking monolithic because the router makes local token-level decisions rather than exposing stable task-level components. A biology prompt can still scatter across a messy set of experts, and the next sentence may hit a different set entirely. The hosts use that distinction to unpack the paper’s core concepts: emergent modularity from unlabeled data, semantic expert specialization around meaningful domains like code or math, composable architecture, and the memory-accuracy frontier that determines whether smaller loaded expert pools can preserve real capability.

The episode then gets into EMO’s training design and why the method is more than a single routing tweak. Ada explains the paper’s two-level routing scheme, where a document first selects a shared candidate pool of experts and individual tokens then choose active experts only within that pool, forcing document-consistent structure without removing all local flexibility. They also cover the supporting recipe: random pool-size sampling to expose the model to different memory budgets during training, global load balancing so a few experts do not dominate usage, and document-length-aware training so very long documents do not overwhelm the learning signal. The result is a focused discussion of whether MoE pretraining can produce expert groups that are not just sparsely activated, but genuinely deployable as modular tools.

Sources:

1. EMO: Emergent Modularity for Mixture-of-Experts

https://allenai.org/papers/emo

2. DEMix Layers: Disentangling Domains for Modular Language Modeling — Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer, 2021

https://scholar.google.com/scholar?q=DEMix+Layers%3A+Disentangling+Domains+for+Modular+Language+Modeling

3. ModuleFormer: Modularity Emerges from Mixture-of-Experts — Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, Chuang Gan, 2023

https://scholar.google.com/scholar?q=ModuleFormer%3A+Modularity+Emerges+from+Mixture-of-Experts

4. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models — Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, Yang You, 2024

https://scholar.google.com/scholar?q=OpenMoE%3A+An+Early+Effort+on+Open+Mixture-of-Experts+Language+Models

5. EMO: Pretraining Mixture of Experts for Emergent Modularity — Ryan Wang, Akshita Bhagia, Sewon Min, 2026

https://scholar.google.com/scholar?q=EMO%3A+Pretraining+Mixture+of+Experts+for+Emergent+Modularity

6. FlexOlmo: Open Language Models for Flexible Data Use — Weijia Shi et al., 2025

https://scholar.google.com/scholar?q=FlexOlmo%3A+Open+Language+Models+for+Flexible+Data+Use

7. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM — Sainbayar Sukhbaatar et al., 2024

https://scholar.google.com/scholar?q=Branch-Train-MiX%3A+Mixing+Expert+LLMs+into+a+Mixture-of-Experts+LLM

8. The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise — Xi Wang, Soufiane Hayou, Eric Nalisnick, 2026

https://scholar.google.com/scholar?q=The+Myth+of+Expert+Specialization+in+MoEs%3A+Why+Routing+Reflects+Geometry%2C+Not+Necessarily+Domain+Expertise

9. Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations — Zican Dong et al., 2025

https://scholar.google.com/scholar?q=Domain-Specific+Pruning+of+Large+Mixture-of-Experts+Models+with+Few-shot+Demonstrations

10. Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs — Enshu Liu et al., 2024

https://scholar.google.com/scholar?q=Efficient+Expert+Pruning+for+Sparse+Mixture-of-Experts+Language+Models%3A+Enhancing+Performance+and+Reducing+Inference+Costs

11. MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router — Yanyue Xie et al., 2024

https://scholar.google.com/scholar?q=MoE-Pruner%3A+Pruning+Mixture-of-Experts+Large+Language+Model+using+the+Hints+from+Its+Router

12. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models — Zihao Zeng et al., 2024

https://scholar.google.com/scholar?q=AdaMoE%3A+Token-Adaptive+Routing+with+Null+Experts+for+Mixture-of-Experts+Language+Models

13. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

14. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev and Denis Mazur, 2023

https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading

15. Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding — Zhibin Wang et al., 2025

https://scholar.google.com/scholar?q=Accelerating+Mixture-of-Experts+Inference+by+Hiding+Offloading+Latency+with+Speculative+Decoding

16. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp3

17. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp3

18. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp3

19. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3

Interactive Visualization: EMO: Emergent Modularity for Mixture-of-Experts

This episode explores how a transformer trained on raw bank transaction histories can model customer behavior for financial product recommendation, and why that may outperform pipelines built from hand-engineered tabular features alone. It explains the paper’s core idea of turning each transaction into a tokenized sequence that mixes inflow or outflow, amount buckets, calendar signals, source metadata, and natural-language merchant descriptions, then pretraining the model with self-supervised learning to produce reusable customer embeddings. The discussion argues that transaction text and long-range patterns such as pay cycles, bill timing, and abrupt behavior changes carry signal that conventional tabular systems often flatten away, while a practical deployment can still combine learned embeddings with legacy banking features downstream. A listener would find it interesting because it connects transformer-style representation learning to a concrete banking use case and shows how foundation-model ideas can be adapted to messy, real-world financial behavior.

Sources:

1. Your Spending Needs Attention: Modeling Financial Habits with Transformers — D. T. Braithwaite, Misael Cavalcanti, R. Austin McEver, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, Felipe Meneses, Arissa Yoshida, Evan Wingert, Matheus Ramos, Brian Zanfelice, Aman Gupta, 2025

http://arxiv.org/abs/2507.23267

2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2019

https://scholar.google.com/scholar?q=BERT%3A+Pre-training+of+Deep+Bidirectional+Transformers+for+Language+Understanding

3. CoLES: Contrastive Learning for Event Sequences with Self-Supervision — Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, Alexander Tuzhilin, 2020

https://scholar.google.com/scholar?q=CoLES%3A+Contrastive+Learning+for+Event+Sequences+with+Self-Supervision

4. Dynamic Customer Embeddings for Financial Service Applications — Nima Chitsazan, Samuel Sharpe, Dwipam Katariya, Qianyu Cheng, Karthik Rajasethupathy, 2021

https://scholar.google.com/scholar?q=Dynamic+Customer+Embeddings+for+Financial+Service+Applications

5. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024

https://scholar.google.com/scholar?q=Towards+a+Foundation+Purchasing+Model%3A+Pretrained+Generative+Autoregression+on+Transaction+Sequences

6. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018

https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation

7. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer — Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang, 2019

https://scholar.google.com/scholar?q=BERT4Rec%3A+Sequential+Recommendation+with+Bidirectional+Encoder+Representations+from+Transformer

8. S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization — Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen, 2020

https://scholar.google.com/scholar?q=S%5E3-Rec%3A+Self-Supervised+Learning+for+Sequential+Recommendation+with+Mutual+Information+Maximization

9. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba — Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, Wenwu Ou, 2019

https://scholar.google.com/scholar?q=Behavior+Sequence+Transformer+for+E-commerce+Recommendation+in+Alibaba

10. Text Is All You Need: Learning Language Representations for Sequential Recommendation — Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, Julian McAuley, 2023

https://scholar.google.com/scholar?q=Text+Is+All+You+Need%3A+Learning+Language+Representations+for+Sequential+Recommendation

11. PinnerFormer: Sequence Modeling for User Representation at Pinterest — Nikil Pancha, Andrew Zhai, Jure Leskovec, Charles Rosenberg, 2022

https://scholar.google.com/scholar?q=PinnerFormer%3A+Sequence+Modeling+for+User+Representation+at+Pinterest

12. Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models — Chengkai Liu et al., 2024

https://scholar.google.com/scholar?q=Mamba4Rec%3A+Towards+Efficient+Sequential+Recommendation+with+Selective+State+Space+Models

13. SSD4Rec: A Structured State Space Duality Model for Efficient Sequential Recommendation — Haohao Qu et al., 2024

https://scholar.google.com/scholar?q=SSD4Rec%3A+A+Structured+State+Space+Duality+Model+for+Efficient+Sequential+Recommendation

14. DynLLM: When Large Language Models Meet Dynamic Graph Recommendation — Ziwei Zhao et al., 2024

https://scholar.google.com/scholar?q=DynLLM%3A+When+Large+Language+Models+Meet+Dynamic+Graph+Recommendation

15. Personalized Elastic Embedding Learning for On-Device Recommendation — Ruiqi Zheng et al., 2023

https://scholar.google.com/scholar?q=Personalized+Elastic+Embedding+Learning+for+On-Device+Recommendation

16. A Survey on Deep Tabular Learning — Shriyank Somvanshi et al., 2024

https://scholar.google.com/scholar?q=A+Survey+on+Deep+Tabular+Learning

17. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3

18. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp3

This episode explores TransactionGPT, a Visa Research paper that argues for a foundation-model approach to consumer transaction data spanning generation, anomaly detection, and representation learning. It explains why payment histories are fundamentally different from text or simple time series: each event mixes merchant IDs, amounts, timestamps, and engineered risk signals, creating a multi-modal, temporal, tabular structure that demands more schema-aware modeling. The discussion walks through the paper’s 1D, 2D, and 3D architecture progression, highlighting how separate transformers for transaction metadata, downstream features, and behavioral sequences aim to avoid the pitfalls of flattening everything into a single embedding space. Listeners would find it interesting for its clear debate over whether TransactionGPT is a genuine reusable backbone for payments or mainly a strong engineering response to real-world constraints like heterogeneity, scale, regulation, and low-latency fraud decisioning.

Sources:

1. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Yan Zheng, Vineeth Rakesh, Huiyuan Chen, Guanchu Wang, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025

http://arxiv.org/abs/2511.08939

2. TabDPT: Scaling Tabular Foundation Models on Real Data — Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Hamidreza Kamkari, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, Maksims Volkovs, 2024

https://arxiv.org/abs/2410.18164

3. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024

https://arxiv.org/abs/2401.01641

4. Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions — Gustavo Polleti, Marlesson Santana, Eduardo Fontes, 2025

https://arxiv.org/abs/2511.12154

5. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Vineeth Rakesh, Huiyuan Chen, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025

https://arxiv.org/abs/2511.08939

6. TabTransformer: Tabular Data Modeling Using Contextual Embeddings — Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin, 2020

https://arxiv.org/abs/2012.06678

7. Tabular Transformers for Modeling Multivariate Time Series — Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, Erik Altman, 2021

https://arxiv.org/abs/2011.01843

8. FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data — Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng, Zhongfang Zhuang, Wei Zhang, 2023

https://arxiv.org/abs/2310.13818

9. Multi-modal Time Series Analysis: A Tutorial and Survey — Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, Dongjin Song, 2025

https://arxiv.org/abs/2503.13709

10. Credit card fraud detection using machine learning: A survey — Yvan Lucas, Johannes Jurgovsky, 2020

https://arxiv.org/abs/2010.06479

11. Interleaved Sequence RNNs for Fraud Detection — Bernardo Branco, Pedro Abreu, Ana Sofia Gomes, Mariana S. C. Almeida, João Tiago Ascensão, Pedro Bizarro, 2020

https://arxiv.org/abs/2002.05988

12. FraudTransformer: Time-Aware GPT for Transaction Fraud Detection — Gholamali Aminian, Andrew Elliott, Tiger Li, Timothy Cheuk Hin Wong, Victor Claude Dehon, Lukasz Szpruch, Carsten Maple, Christopher Read, Martin Brown, Gesine Reinert, Mo Mamouei, 2025

https://arxiv.org/abs/2509.23712

13. Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings — Xiran Fan, Zhimeng Jiang, Chin-Chia Michael Yeh, Yuzhong Chen, Yingtong Dou, Menghai Pan, Yan Zheng, 2025

https://scholar.google.com/scholar?q=Enhancing+Foundation+Models+in+Transaction+Understanding+with+LLM-based+Sentence+Embeddings

14. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018

https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation

15. TabLLM: Few-shot Classification of Tabular Data with Large Language Models — Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, David Sontag, 2022

https://scholar.google.com/scholar?q=TabLLM%3A+Few-shot+Classification+of+Tabular+Data+with+Large+Language+Models

16. TimeGPT-1 — Azul Garza, Cristian Challu, Max Mergenthaler-Canseco, 2023

https://scholar.google.com/scholar?q=TimeGPT-1

17. ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning — Junguang Jiang et al., 2023

https://scholar.google.com/scholar?q=ForkMerge%3A+Mitigating+Negative+Transfer+in+Auxiliary-Task+Learning

18. Identification of Negative Transfers in Multitask Learning Using Surrogate Models — Dongyue Li, Huy L. Nguyen, Hongyang R. Zhang, 2023

https://scholar.google.com/scholar?q=Identification+of+Negative+Transfers+in+Multitask+Learning+Using+Surrogate+Models

19. Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers — Gjergji Kasneci, Enkelejda Kasneci, 2024

https://scholar.google.com/scholar?q=Enriching+Tabular+Data+with+Contextual+LLM+Embeddings%3A+A+Comprehensive+Ablation+Study+for+Ensemble+Classifiers

20. LLM Embeddings Improve Test-time Adaptation to Tabular Y|X-Shifts — Yibo Zeng et al., 2024

https://scholar.google.com/scholar?q=LLM+Embeddings+Improve+Test-time+Adaptation+to+Tabular+Y%7CX-Shifts

21. LLM Embeddings for Deep Learning on Tabular Data — Boshko Koloski et al., 2025

https://scholar.google.com/scholar?q=LLM+Embeddings+for+Deep+Learning+on+Tabular+Data

22. TableGPT2: A Large Multimodal Model with Tabular Data Integration — Aofeng Su et al., 2024

https://scholar.google.com/scholar?q=TableGPT2%3A+A+Large+Multimodal+Model+with+Tabular+Data+Integration

23. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3

24. AI Post Transformers: Relational Graph Transformer for Multi-Table Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-relational-graph-transformer-for-multi-t-57cce3.mp3

25. AI Post Transformers: Unembedding Matrices as Feature Lenses for Embeddings — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-08-unembedding-matrices-as-feature-lenses-f-4dc415.mp3

Interactive Visualization: TransactionGPT as a Payments Foundation Model

This episode explores the paper When Does LeJEPA Learn a World Model? and uses it to examine what should count as a genuine world model in latent predictive learning, contrasting JEPA-style representation prediction with generative reconstruction. It explains why good probe scores are not enough: the real standard is linear identifiability, where a single global linear map recovers the environment’s hidden state well enough to support planning and compositional generalization. The discussion centers on the paper’s main theorem that, under stationary additive-noise dynamics with Gaussian latent variables, LeJEPA’s alignment objective plus SIGReg recovers the true latent state up to an orthogonal rotation, and on the sharper converse result that this universal guarantee fails for non-Gaussian latents. Listeners get a rigorous argument for when latent models are truly learning the world’s coordinates instead of merely extracting features that happen to be useful on downstream tasks.

Sources:

1. When LeJEPA Truly Learns a World Model

https://arxiv.org/pdf/2605.26379

2. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations — Francesco Locatello, Stefan Bauer, Mario Lucic, et al., 2018

https://scholar.google.com/scholar?q=Challenging+Common+Assumptions+in+the+Unsupervised+Learning+of+Disentangled+Representations

3. Variational Autoencoders and Nonlinear ICA: A Unifying Framework — Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, Aapo Hyvarinen, 2019

https://scholar.google.com/scholar?q=Variational+Autoencoders+and+Nonlinear+ICA%3A+A+Unifying+Framework

4. On Linear Identifiability of Learned Representations — Geoffrey Roeder, Luke Metz, Diederik P. Kingma, 2020

https://scholar.google.com/scholar?q=On+Linear+Identifiability+of+Learned+Representations

5. Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning — Aapo Hyvarinen, Ilyes Khemakhem, Hiroshi Morioka, 2023

https://scholar.google.com/scholar?q=Nonlinear+Independent+Component+Analysis+for+Principled+Disentanglement+in+Unsupervised+Deep+Learning

6. Auto-Encoding Variational Bayes — Diederik P. Kingma, Max Welling, 2013

https://scholar.google.com/scholar?q=Auto-Encoding+Variational+Bayes

7. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021

https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning

8. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025

https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics

9. When Does LeJEPA Learn a World Model? — David Klindt, Yann LeCun, Randall Balestriero, 2026

https://scholar.google.com/scholar?q=When+Does+LeJEPA+Learn+a+World+Model%3F

10. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero, 2026

https://scholar.google.com/scholar?q=LeWorldModel%3A+Stable+End-to-End+Joint-Embedding+Predictive+Architecture+from+Pixels

11. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning — Mido Assran et al., 2025

https://scholar.google.com/scholar?q=V-JEPA+2%3A+Self-Supervised+Video+Models+Enable+Understanding%2C+Prediction+and+Planning

12. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning — Aapo Hyvarinen, Hiroaki Sasaki, Richard E. Turner, 2018

https://scholar.google.com/scholar?q=Nonlinear+ICA+Using+Auxiliary+Variables+and+Generalized+Contrastive+Learning

13. Joint Embedding Predictive Architectures Focus on Slow Features — Vlad Sobal, Jyothir S V, Siddhartha Jalagam, Nicolas Carion, Kyunghyun Cho, Yann LeCun, 2022

https://scholar.google.com/scholar?q=Joint+Embedding+Predictive+Architectures+Focus+on+Slow+Features

14. Cross-Entropy Is All You Need To Invert the Data Generating Process — Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, David Klindt, 2024

https://scholar.google.com/scholar?q=Cross-Entropy+Is+All+You+Need+To+Invert+the+Data+Generating+Process

15. Identifiability of latent-variable and structural-equation models: from linear to nonlinear — Aapo Hyvarinen, Ilyes Khemakhem, Ricardo Monti, 2023

https://scholar.google.com/scholar?q=Identifiability+of+latent-variable+and+structural-equation+models%3A+from+linear+to+nonlinear

16. On the Identifiability of Sparse ICA without Assuming Non-Gaussianity — Ignavier Ng, Yujia Zheng, Xinshuai Dong, Kun Zhang, 2024

https://scholar.google.com/scholar?q=On+the+Identifiability+of+Sparse+ICA+without+Assuming+Non-Gaussianity

17. Adaptive World Models: Learning Behaviors by Latent Imagination Under Non-Stationarity — Emiliyan Gospodinov, Vaisakh Shaj, Philipp Becker, Stefan Geyer, Gerhard Neumann, 2024

https://scholar.google.com/scholar?q=Adaptive+World+Models%3A+Learning+Behaviors+by+Latent+Imagination+Under+Non-Stationarity

18. Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors — Yong Liu, Chenyu Li, Jianmin Wang, Mingsheng Long, 2023

https://scholar.google.com/scholar?q=Koopa%3A+Learning+Non-stationary+Time+Series+Dynamics+with+Koopman+Predictors

19. Simplifying Latent Dynamics with Softly State-Invariant World Models — Tankred Saanum, Peter Dayan, Eric Schulz, 2024

https://scholar.google.com/scholar?q=Simplifying+Latent+Dynamics+with+Softly+State-Invariant+World+Models

20. Structured World Models from Human Videos — Russell Mendonca, Shikhar Bahl, Deepak Pathak, 2023

https://scholar.google.com/scholar?q=Structured+World+Models+from+Human+Videos

21. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3

22. AI Post Transformers: Causal-JEPA for Object-Level World Models — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-15-causal-jepa-for-object-level-world-model-311a8b.mp3

23. AI Post Transformers: Learning Latent Action World Models from Video — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-09-learning-latent-action-world-models-from-1570a4.mp3

Interactive Visualization: When LeJEPA Truly Learns a World Model

Hal Turing and Dr. Ada Shannon examine an empirical study of parameter-efficient fine-tuning for large language models, centered on a practical question: when does a small task-specific update beat retraining the entire model? Using FLAN-T5-XL as the test bed, they frame PEFT as a transfer-learning strategy that freezes most of the transformer while learning a compact adaptation layer, whether through LoRA’s low-rank weight updates, adapter-style modules, IA3 scaling vectors, BitFit bias updates, or learned soft prompts. The discussion keeps returning to the real systems tradeoff: quality matters, but so do training speed, storage cost, and the burden of maintaining separate model copies for many downstream tasks.

The episode walks through the benchmark design in detail rather than treating PEFT as a vague category. The paper compares full tuning, LoRA, IA3, prompt tuning, and BitFit on the same backbone across classification tasks like AG News and CoLA, generation tasks like E2E and SAMSum, and data budgets of roughly 100, 1,000, and 10,000 examples. The hosts emphasize why those controls matter: same model, same stopping rule, and fixed method settings make it easier to see where each technique actually helps, while also limiting how far the results should be generalized to other architectures, especially decoder-only chat models.

They then dig into the paper’s uneven but useful results. In low-resource settings, LoRA and BitFit frequently outperform full tuning, with LoRA posting a notably stronger CoLA score and BitFit leading on AG News, E2E, and SAMSum, while prompt tuning performs strikingly poorly on the generation benchmarks under this setup. In medium-resource settings, IA3, LoRA, and BitFit remain competitive, but at higher data scales full tuning starts reclaiming ground on some tasks even as LoRA and IA3 still win specific cases. The takeaway is not that one PEFT method universally dominates, but that the strengths and weaknesses of each approach shift with task type, data regime, and the exact adaptation recipe.

Sources:

1. Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs — George Pu, Anirudh Jain, Jihan Yin, Russell Kaplan, 2023

http://arxiv.org/abs/2304.14999

2. Parameter-Efficient Transfer Learning for NLP — Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, et al., 2019

https://arxiv.org/abs/1902.00751

3. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, et al., 2021

https://arxiv.org/abs/2106.09685

4. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning — Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, Anna Rumshisky, 2023

https://arxiv.org/abs/2303.15647

5. Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs — George Pu, Anirudh Jain, Jihan Yin, Russell Kaplan, 2023

https://arxiv.org/abs/2304.14999

6. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021

https://arxiv.org/abs/2101.00190

7. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021

https://arxiv.org/abs/2104.08691

8. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks — Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, Jie Tang, 2021

https://arxiv.org/abs/2110.07602

9. SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer — Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, Daniel Cer, 2021

https://arxiv.org/abs/2110.07904

10. Revisiting Parameter-Efficient Tuning: Are We Really There Yet? — Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, Shangsong Liang, 2022

https://scholar.google.com/scholar?q=Revisiting+Parameter-Efficient+Tuning%3A+Are+We+Really+There+Yet%3F

11. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel, 2022

https://scholar.google.com/scholar?q=Few-Shot+Parameter-Efficient+Fine-Tuning+is+Better+and+Cheaper+than+In-Context+Learning

12. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation — Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si, 2021

https://scholar.google.com/scholar?q=On+the+Effectiveness+of+Adapter-based+Tuning+for+Pretrained+Language+Model+Adaptation

13. Scaling Instruction-Finetuned Language Models — Hyung Won Chung et al., 2022

https://scholar.google.com/scholar?q=Scaling+Instruction-Finetuned+Language+Models

14. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method — Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat, 2024

https://arxiv.org/abs/2402.17193

15. Parameter-Efficient Fine-Tuning Design Spaces — Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, Diyi Yang, 2023

https://arxiv.org/abs/2301.01821

16. AI Post Transformers: ForkKV for Multi-LoRA Agent Serving — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-13-forkkv-for-multi-lora-agent-serving-ccafa4.mp3

17. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp3

18. AI Post Transformers: ZeRO-Offload: Democratizing Billion-Scale Model Training — Hal Turing & Dr. Ada Shannon, Fri,

https://podcast.do-not-panic.com/episodes/zero-offload-democratizing-billion-scale-model-training/

Interactive Visualization: Benchmarking PEFT Techniques for Large Language Models

This episode explores Weak-SIGReg, a lightweight covariance regularizer designed to prevent representation collapse in fragile supervised training, especially for small-data Vision Transformers. It explains how the method uses a sketched covariance matrix and an identity-matching penalty to keep hidden features decorrelated and similarly scaled at much lower cost than full covariance regularization. The discussion centers on CIFAR-100 results, where Weak-SIGReg dramatically improves a deliberately unstable ViT setup and also boosts a plain MLP, while offering little change on an already stable ResNet18. It also digs into the paper’s main caveat: the biggest gains appear when the baseline training recipe is badly broken, so the most interesting question is not just whether the regularizer works, but when it adds real value beyond simply fixing optimization and initialization.

Sources:

1. Weak-SIGReg: Covariance Regularization for Stable Deep Learning — Habibullah Akbar, 2026

http://arxiv.org/abs/2603.05924

2. Reducing Overfitting in Deep Networks by Decorrelating Representations — Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra, 2015

https://scholar.google.com/scholar?q=Reducing+Overfitting+in+Deep+Networks+by+Decorrelating+Representations

3. Barlow Twins: Self-Supervised Learning via Redundancy Reduction — Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stephane Deny, 2021

https://scholar.google.com/scholar?q=Barlow+Twins%3A+Self-Supervised+Learning+via+Redundancy+Reduction

4. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021

https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning

5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025

https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics

6. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., 2021

https://scholar.google.com/scholar?q=An+Image+is+Worth+16x16+Words%3A+Transformers+for+Image+Recognition+at+Scale

7. Training data-efficient image transformers & distillation through attention — Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou, 2021

https://scholar.google.com/scholar?q=Training+data-efficient+image+transformers+%26+distillation+through+attention

8. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers — Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer, 2021

https://scholar.google.com/scholar?q=How+to+train+your+ViT%3F+Data%2C+Augmentation%2C+and+Regularization+in+Vision+Transformers

9. Early Convolutions Help Transformers See Better — Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollar, Ross Girshick, 2021

https://scholar.google.com/scholar?q=Early+Convolutions+Help+Transformers+See+Better

10. Whitening for Self-Supervised Representation Learning — Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe, 2020

https://arxiv.org/abs/2007.06346

11. Understanding Dimensional Collapse in Contrastive Self-Supervised Learning — Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian, 2021

https://arxiv.org/abs/2110.09348

12. UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures — Triet M. Le, 2026

https://arxiv.org/abs/2606.01443

13. Stability of Transformers under Layer Normalization — Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis, 2025

https://scholar.google.com/scholar?q=Stability+of+Transformers+under+Layer+Normalization

14. Transformers without Normalization — Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu, 2025

https://scholar.google.com/scholar?q=Transformers+without+Normalization

15. Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations — Yongyi Yang, Jacob Steinhardt, Wei Hu, 2023

https://scholar.google.com/scholar?q=Are+Neurons+Actually+Collapsed%3F+On+the+Fine-Grained+Structure+in+Neural+Representations

16. The Impact of Geometric Complexity on Neural Collapse in Transfer Learning — Michael Munn, Benoit Dherin, Javier Gonzalvo, 2024

https://scholar.google.com/scholar?q=The+Impact+of+Geometric+Complexity+on+Neural+Collapse+in+Transfer+Learning

17. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3

18. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp3