This episode explores Paul Werbos’s 1990 paper on Backpropagation Through Time and explains how ordinary backpropagation extends to systems whose state evolves over time. It walks through the core idea of unrolling a recurrent or dynamic system into a time-indexed computation graph, then applying reverse-mode differentiation to compute exact gradients across both layers and time steps. The discussion also places BPTT in historical context, connecting it to earlier work on backpropagation, automatic differentiation, and alternative recurrent learning methods like real-time recurrent learning. Listeners would find it interesting because it shows how a foundational training method for sequence models, control systems, and differentiable simulations emerged from a simple but powerful reframing of memory and time in neural computation.

Sources:

1. Backpropagation Through Time Explained

https://podcast.do-not-panic.com/uploaded-pdfs/2026-05-08T22-17-15-230Z-Backpropagation-through-time-what-it-does-and-how-to-do-it.pdf

2. Learning representations by back-propagating errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986

https://scholar.google.com/scholar?q=Learning+representations+by+back-propagating+errors

3. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks — Ronald J. Williams, David Zipser, 1989

https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks

4. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990

https://scholar.google.com/scholar?q=Backpropagation+Through+Time%3A+What+It+Does+and+How+to+Do+It

5. Learning long-term dependencies with gradient descent is difficult — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994

https://scholar.google.com/scholar?q=Learning+long-term+dependencies+with+gradient+descent+is+difficult

6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997

https://scholar.google.com/scholar?q=Long+Short-Term+Memory

7. Taylor expansion of the accumulated rounding error — Seppo Linnainmaa, 1976

https://scholar.google.com/scholar?q=Taylor+expansion+of+the+accumulated+rounding+error

8. Fast Exact Multiplication by the Hessian — Barak A. Pearlmutter, 1994

https://scholar.google.com/scholar?q=Fast+Exact+Multiplication+by+the+Hessian

9. Automatic Differentiation in Machine Learning: a Survey — Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind, 2018

https://scholar.google.com/scholar?q=Automatic+Differentiation+in+Machine+Learning%3A+a+Survey

10. A review of automatic differentiation and its efficient implementation — Charles C. Margossian, 2019

https://scholar.google.com/scholar?q=A+review+of+automatic+differentiation+and+its+efficient+implementation

11. Generalization of Back-Propagation to Recurrent Neural Networks — Fernando J. Pineda, 1987

https://scholar.google.com/scholar?q=Generalization+of+Back-Propagation+to+Recurrent+Neural+Networks

12. Finding Structure in Time — Jeffrey L. Elman, 1990

https://scholar.google.com/scholar?q=Finding+Structure+in+Time

13. BP(lambda): Online Learning via Synthetic Gradients — approx. anonymous from snippet / modern deep learning authors, recent

https://scholar.google.com/scholar?q=BP%28lambda%29%3A+Online+Learning+via+Synthetic+Gradients

14. Streaming Propagation Through Time: A New Computational Paradigm for Recurrent Neural Networks — approx. modern recurrent-learning authors, recent

https://scholar.google.com/scholar?q=Streaming+Propagation+Through+Time%3A+A+New+Computational+Paradigm+for+Recurrent+Neural+Networks

15. Combining Truncated BPTT and Truncated RTRL for LSTM Training — Jakob Stefan Weber, recent

https://scholar.google.com/scholar?q=Combining+Truncated+BPTT+and+Truncated+RTRL+for+LSTM+Training

16. Second-order forward-mode optimization of recurrent neural networks for neuroscience — approx. modern neuroscience/optimization authors, recent

https://scholar.google.com/scholar?q=Second-order+forward-mode+optimization+of+recurrent+neural+networks+for+neuroscience

17. Sample-Based Hybrid Mode Control: Asymptotically Optimal Switching of Algorithmic and Non-Differentiable Control Modes — approx. modern control authors, recent

https://scholar.google.com/scholar?q=Sample-Based+Hybrid+Mode+Control%3A+Asymptotically+Optimal+Switching+of+Algorithmic+and+Non-Differentiable+Control+Modes

18. On the differentiability of the value function of switched linear systems under arbitrary and controlled switching — approx. control theory authors, recent

https://scholar.google.com/scholar?q=On+the+differentiability+of+the+value+function+of+switched+linear+systems+under+arbitrary+and+controlled+switching

19. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp3

20. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3

Interactive Visualization: Backpropagation Through Time Explained

This episode explores Marvin Minsky’s 1961 paper on whether mostly random neural networks can learn useful behavior simply by reinforcing successful responses. It explains how the paper distinguishes rote memory, associative recall, pattern recognition, and true generalization, arguing that reward signals alone are not enough unless the system already has a meaningful notion of similarity between situations. The discussion places that idea in context with early machine learning work like Rosenblatt’s perceptron and Samuel’s checkers program, then connects it to later, more disciplined descendants such as echo state networks and random features. Listeners get a sharp historical view of a debate that still matters now: whether intelligence comes from discovering good representations or from selecting among structures that were already there.

Sources:

1. Learning in Random Nets and Generalization

https://stacks.stanford.edu/file/druid:yr384hg3073/yr384hg3073.pdf

2. Learning in Random Nets — Marvin Minsky, Oliver G. Selfridge, 1961

https://scholar.google.com/scholar?q=Learning+in+Random+Nets

3. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain — Frank Rosenblatt, 1958

https://scholar.google.com/scholar?q=The+Perceptron%3A+A+Probabilistic+Model+for+Information+Storage+and+Organization+in+the+Brain

4. The Echo State Approach to Analysing and Training Recurrent Neural Networks — Herbert Jaeger, 2001

https://scholar.google.com/scholar?q=The+Echo+State+Approach+to+Analysing+and+Training+Recurrent+Neural+Networks

5. Random Features for Large-Scale Kernel Machines — Ali Rahimi, Benjamin Recht, 2007

https://scholar.google.com/scholar?q=Random+Features+for+Large-Scale+Kernel+Machines

6. The Use of Multiple Measurements in Taxonomic Problems — R. A. Fisher, 1936

https://scholar.google.com/scholar?q=The+Use+of+Multiple+Measurements+in+Taxonomic+Problems

7. Nearest Neighbor Pattern Classification — Thomas M. Cover, Peter E. Hart, 1967

https://scholar.google.com/scholar?q=Nearest+Neighbor+Pattern+Classification

8. Support-Vector Networks — Corinna Cortes, Vladimir Vapnik, 1995

https://scholar.google.com/scholar?q=Support-Vector+Networks

9. Gradient-Based Learning Applied to Document Recognition — Yann LeCun, Leon Bottou, Yoshua Bengio, Patrick Haffner, 1998

https://scholar.google.com/scholar?q=Gradient-Based+Learning+Applied+to+Document+Recognition

10. Some Studies in Machine Learning Using the Game of Checkers — Arthur L. Samuel, 1959

https://scholar.google.com/scholar?q=Some+Studies+in+Machine+Learning+Using+the+Game+of+Checkers

11. Generalization of Pattern Recognition in a Self-Organizing System — B. G. Farley and W. A. Clark, 1954

https://scholar.google.com/scholar?q=Generalization+of+Pattern+Recognition+in+a+Self-Organizing+System

12. A Heterarchy of Values Determined by the Topology of Nervous Nets — Warren S. McCulloch, 1945

https://scholar.google.com/scholar?q=A+Heterarchy+of+Values+Determined+by+the+Topology+of+Nervous+Nets

13. Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime — Hong Hu, Yue M. Lu, Theodor Misiakiewicz, 2024

https://scholar.google.com/scholar?q=Asymptotics+of+Random+Feature+Regression+Beyond+the+Linear+Scaling+Regime

14. Power-Law Spectrum of the Random Feature Model — Elliot Paquette, Ke Liang Xiao, Yizhe Zhu, 2026

https://scholar.google.com/scholar?q=Power-Law+Spectrum+of+the+Random+Feature+Model

15. Local to Global: Learning Dynamics and Effect of Initialization for Transformers — Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gastpar, 2024

https://scholar.google.com/scholar?q=Local+to+Global%3A+Learning+Dynamics+and+Effect+of+Initialization+for+Transformers

16. Augmenting Language Models with Long-Term Memory — Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei, 2023

https://scholar.google.com/scholar?q=Augmenting+Language+Models+with+Long-Term+Memory

17. MemoryBank: Enhancing Large Language Models with Long-Term Memory — Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, Yanlin Wang, 2023

https://scholar.google.com/scholar?q=MemoryBank%3A+Enhancing+Large+Language+Models+with+Long-Term+Memory

18. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models — Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su, 2024

https://scholar.google.com/scholar?q=HippoRAG%3A+Neurobiologically+Inspired+Long-Term+Memory+for+Large+Language+Models

19. The Learnability of In-Context Learning — Noam Wies, Yoav Levine, Amnon Shashua, 2023

https://scholar.google.com/scholar?q=The+Learnability+of+In-Context+Learning

20. Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions — Satwik Bhattamishra, Arkil Patel, Phil Blunsom, Varun Kanade, 2023

https://scholar.google.com/scholar?q=Understanding+In-Context+Learning+in+Transformers+and+LLMs+by+Learning+to+Learn+Discrete+Functions

21. Learning without Training: The Implicit Dynamics of In-Context Learning — Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, Javier Gonzalvo, 2025

https://scholar.google.com/scholar?q=Learning+without+Training%3A+The+Implicit+Dynamics+of+In-Context+Learning

22. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp3

23. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3

24. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3

25. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3

Interactive Visualization: Learning in Random Nets and Generalization

This episode explores the 1959 paper on frog vision that argued the retina does far more than passively relay a camera-like image to the brain. It explains how experiments on single optic nerve fibers revealed specialized visual detectors tuned to ecologically relevant signals such as small moving dark objects, edges, dimming, and contrast changes, rather than raw brightness alone. The discussion connects these findings to modern machine learning ideas like preprocessing, receptive fields, sparse event-driven signals, and early feature extraction, while also emphasizing where biological retinal circuits differ sharply from engineered neural networks. A listener would find it interesting because it shows how a foundational neuroscience experiment anticipated core ideas in AI and neural coding by asking what information an animal actually needs to survive.

Sources:

1. What the Frog’s Eye Tells the Brain

https://courses.csail.mit.edu/6.803/pdf/lettvin.pdf

2. The Response of Single Optic Nerve Fibers of the Vertebrate Eye to Illumination of the Retina — H. Keffer Hartline, 1938

https://scholar.google.com/scholar?q=The+Response+of+Single+Optic+Nerve+Fibers+of+the+Vertebrate+Eye+to+Illumination+of+the+Retina

3. Discharge Patterns and Functional Organization of Mammalian Retina — Stephen W. Kuffler, 1953

https://scholar.google.com/scholar?q=Discharge+Patterns+and+Functional+Organization+of+Mammalian+Retina

4. Anatomy and Physiology of Vision in the Frog (Rana pipiens) — Humberto R. Maturana, Jerome Y. Lettvin, Warren S. McCulloch, Walter H. Pitts, 1960

https://scholar.google.com/scholar?q=Anatomy+and+Physiology+of+Vision+in+the+Frog+%28Rana+pipiens%29

5. The Dynamic Receptive Fields of Retinal Ganglion Cells — Sophia Wienbar, Gregory W. Schwartz, 2018

https://scholar.google.com/scholar?q=The+Dynamic+Receptive+Fields+of+Retinal+Ganglion+Cells

6. What the Frog's Eye Tells the Frog's Brain — Jerome Y. Lettvin, Humberto R. Maturana, Warren S. McCulloch, Walter H. Pitts, 1959

https://scholar.google.com/scholar?q=What+the+Frog%27s+Eye+Tells+the+Frog%27s+Brain

7. Summation and Inhibition in the Frog's Retina — Horace B. Barlow, 1953

https://scholar.google.com/scholar?q=Summation+and+Inhibition+in+the+Frog%27s+Retina

8. The Mechanism of Directionally Selective Units in Rabbit's Retina — Horace B. Barlow, William R. Levick, 1965

https://scholar.google.com/scholar?q=The+Mechanism+of+Directionally+Selective+Units+in+Rabbit%27s+Retina

9. The Retina Dissects the Visual Scene into Distinct Features — Botond Roska, Markus Meister, 2014

https://scholar.google.com/scholar?q=The+Retina+Dissects+the+Visual+Scene+into+Distinct+Features

10. Possible Principles Underlying the Transformations of Sensory Messages — Horace B. Barlow, 1961

https://scholar.google.com/scholar?q=Possible+Principles+Underlying+the+Transformations+of+Sensory+Messages

11. The Neural Code of the Retina — Markus Meister, Michael J. Berry II, 1999

https://scholar.google.com/scholar?q=The+Neural+Code+of+the+Retina

12. Weak Pairwise Correlations Imply Strongly Correlated Network States in a Neural Population — Elad Schneidman, Michael J. Berry II, Ronen Segev, William Bialek, 2006

https://scholar.google.com/scholar?q=Weak+Pairwise+Correlations+Imply+Strongly+Correlated+Network+States+in+a+Neural+Population

13. Spatio-temporal Correlations and Visual Signalling in a Complete Neuronal Population — Jonathan W. Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, Eero P. Simoncelli, 2008

https://scholar.google.com/scholar?q=Spatio-temporal+Correlations+and+Visual+Signalling+in+a+Complete+Neuronal+Population

14. Receptive Fields of Single Neurones in the Cat's Striate Cortex — D. H. Hubel and T. N. Wiesel, 1959

https://scholar.google.com/scholar?q=Receptive+Fields+of+Single+Neurones+in+the+Cat%27s+Striate+Cortex

15. Interpreting the retinal neural code for natural scenes: From computations to neurons — Maheswaranathan, McIntosh, Tanaka, Baccus et al., 2023

https://scholar.google.com/scholar?q=Interpreting+the+retinal+neural+code+for+natural+scenes%3A+From+computations+to+neurons

16. Spatial adaptation of primate retinal ganglion cells between artificial and natural stimuli — Vystrcilova, Sridhar, Burg, Gollisch, Ecker et al., 2025/2026

https://scholar.google.com/scholar?q=Spatial+adaptation+of+primate+retinal+ganglion+cells+between+artificial+and+natural+stimuli

17. Distributed feature representations of natural stimuli across parallel retinal pathways — Hsiang, Shen, Soto, Kerschensteiner et al., 2024

https://scholar.google.com/scholar?q=Distributed+feature+representations+of+natural+stimuli+across+parallel+retinal+pathways

18. Retinal motion statistics during natural locomotion — Muller, Matthis, Bonnen, Cormack, Huk, Hayhoe, 2023

https://scholar.google.com/scholar?q=Retinal+motion+statistics+during+natural+locomotion

19. Natural visual behavior and active sensing in the mouse — review by members of the Niell lab and colleagues, 2024

https://scholar.google.com/scholar?q=Natural+visual+behavior+and+active+sensing+in+the+mouse

20. A genetically defined tecto-thalamic pathway drives a system of superior-colliculus-dependent visual cortices — Brenner, Beltramo, Gerfen, Ruediger, Scanziani, 2023

https://scholar.google.com/scholar?q=A+genetically+defined+tecto-thalamic+pathway+drives+a+system+of+superior-colliculus-dependent+visual+cortices

Interactive Visualization: What the Frog’s Eye Tells the Brain

This episode explores a mechanistic interpretability study asking whether a language model can detect when a concept has been injected into its hidden activations and, in some cases, identify what that concept was. It explains the difference between detection and identification, walks through activation steering in the residual stream, and highlights the paper’s controlled experiments on Gemma3-27B across 500 concepts, including a strong result of moderate detection with zero false positives under several prompt styles. The discussion also focuses on the paper’s argument that this reporting behavior emerges mainly during post-training, especially preference optimization, rather than from pretraining alone. Listeners would find it interesting because it turns a provocative claim about model “introspection” into a concrete circuit-level question about what internal features and gates may be doing.

Sources:

1. Mechanisms of Introspective Awareness — Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, Jack Lindsey, 2026

http://arxiv.org/abs/2603.21396

2. Emergent Introspective Awareness in Large Language Models — Jack Lindsey, 2025

https://scholar.google.com/scholar?q=Emergent+Introspective+Awareness+in+Large+Language+Models

3. Looking Inward: Language Models Can Learn About Themselves by Introspection — Felix J. Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, Owain Evans, 2024

https://scholar.google.com/scholar?q=Looking+Inward%3A+Language+Models+Can+Learn+About+Themselves+by+Introspection

4. Activation Addition: Steering Language Models Without Optimization — Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, Monte MacDiarmid, Chris Olah, 2023

https://scholar.google.com/scholar?q=Activation+Addition%3A+Steering+Language+Models+Without+Optimization

5. Representation Engineering: A Top-Down Approach to AI Transparency — Andy Zou, Long Phan, Sarah Chen, James Campbell, Richard Ngo, Adam Jermyn, Stephen McAleer, Alexander Tamkin, 2023

https://scholar.google.com/scholar?q=Representation+Engineering%3A+A+Top-Down+Approach+to+AI+Transparency

6. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, and others, 2024

https://scholar.google.com/scholar?q=Scaling+Monosemanticity%3A+Extracting+Interpretable+Features+from+Claude+3+Sonnet

7. Circuit Tracing: Revealing Computational Graphs in Language Models — Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, and others, 2025

https://scholar.google.com/scholar?q=Circuit+Tracing%3A+Revealing+Computational+Graphs+in+Language+Models

8. Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=Steering+Vector+Fields+for+Context-Aware+Inference-Time+Control+in+Large+Language+Models

9. No Training Wheels: Steering Vectors for Bias Correction at Inference Time — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=No+Training+Wheels%3A+Steering+Vectors+for+Bias+Correction+at+Inference+Time

10. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=A+mechanistic+understanding+of+alignment+algorithms%3A+A+case+study+on+DPO+and+toxicity

11. How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=How+Does+DPO+Reduce+Toxicity%3F+A+Mechanistic+Neuron-Level+Analysis

12. Refusal in language models is mediated by a single direction — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=Refusal+in+language+models+is+mediated+by+a+single+direction

13. Beyond I'm Sorry, I Can't: Dissecting Large-Language-Model Refusal — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=Beyond+I%27m+Sorry%2C+I+Can%27t%3A+Dissecting+Large-Language-Model+Refusal

14. Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=Surgical%2C+cheap%2C+and+flexible%3A+Mitigating+false+refusal+in+language+models+via+single+vector+ablation

15. Residual stream analysis with multi-layer saes — authors unclear from snippet, 2025/2026

https://scholar.google.com/scholar?q=Residual+stream+analysis+with+multi-layer+saes

16. AI Post Transformers: Anthropic: Introspective Awareness in LLMs — Hal Turing & Dr. Ada Shannon, 2025

https://podcast.do-not-panic.com/episodes/anthropic-introspective-awareness-in-llms/

17. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp3

18. AI Post Transformers: Advancing Mechanistic Interpretability with Sparse Autoencoders — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/advancing-mechanistic-interpretability-with-sparse-autoencoders/

19. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp3

20. AI Post Transformers: Self-Improving Pretraining With Post-Trained Models — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-02-self-improving-pretraining-with-post-tra-e37460.mp3

Interactive Visualization: How Models Detect Hidden Activation Steering

This episode explores an FPGA-based time-to-digital converter that combines careful delay-line layout with machine-learning-based calibration to achieve very fine timing measurements on real hardware. It explains how tapped-delay-line TDCs work, why real FPGA implementations suffer from nonuniform time bins and bubble errors, and why those imperfections matter for applications like LiDAR, medical imaging, particle physics, and high-speed communications. The discussion compares the new approach against earlier FPGA TDC work, arguing that the real contribution is not flashy AI but a practical learned decoder that maps a 940-bit raw hardware output into a more accurate time estimate after physical design has reduced as much noise as possible. Listeners would find it interesting because it gets specific about where machine learning genuinely helps in instrumentation: not replacing physics, but reducing calibration effort while preserving picosecond-level precision.

Sources:

1. Machine Learning Self-Calibrated FPGA Time-to-Digital Converter

https://podcast.do-not-panic.com/uploaded-pdfs/2026-05-08T03-07-35-153Z-1-s2.0-S2667305326000190-main.pdf

2. A 19.6 ps, FPGA-Based TDC With Multiple Channels for Open Source Applications — Matthew W. Fishburn, L. Harmen Menninga, Claudio Favi, Edoardo Charbon, 2013

https://scholar.google.com/scholar?q=A+19.6+ps%2C+FPGA-Based+TDC+With+Multiple+Channels+for+Open+Source+Applications

3. A low nonlinearity, missing-code free time-to-digital converter based on 28nm FPGAs with embedded bin-width calibrations — Haochang Chen, Yongliang Zhang, David Day-Uei Li, 2017

https://scholar.google.com/scholar?q=A+low+nonlinearity%2C+missing-code+free+time-to-digital+converter+based+on+28nm+FPGAs+with+embedded+bin-width+calibrations

4. A 19 ps Precision and 170 M Samples/s Time-to-Digital Converter Implemented in FPGA with Online Calibration — Mengdi Zhang, Ye Zhao, Zhengsheng Han, Fazhan Zhao, 2022

https://scholar.google.com/scholar?q=A+19+ps+Precision+and+170+M+Samples%2Fs+Time-to-Digital+Converter+Implemented+in+FPGA+with+Online+Calibration

5. Low-Resource Time-to-Digital Converters for Field Programmable Gate Arrays: A Review — Diego Real, David Calvo, 2024

https://scholar.google.com/scholar?q=Low-Resource+Time-to-Digital+Converters+for+Field+Programmable+Gate+Arrays%3A+A+Review

6. Calibration Methods for Time-to-Digital Converters — Wassim Khaddour, Wilfried Uhring, Foudil Dadouche, Norbert Dumas, Morgan Madec, 2023

https://scholar.google.com/scholar?q=Calibration+Methods+for+Time-to-Digital+Converters

7. Time Resolution Improvement Using Dual Delay Lines for Field-Programmable-Gate-Array-Based Time-to-Digital Converters with Real-Time Calibration — Yuan-Ho Chen, 2019

https://scholar.google.com/scholar?q=Time+Resolution+Improvement+Using+Dual+Delay+Lines+for+Field-Programmable-Gate-Array-Based+Time-to-Digital+Converters+with+Real-Time+Calibration

8. Novel machine learning-driven optimizing decoding solutions for FPGA-based time-to-digital converters — Fabio Garzetti, Nicola Lusardi, Enrico Ronconi, Andrea Costa, Angelo Geraci, 2024

https://scholar.google.com/scholar?q=Novel+machine+learning-driven+optimizing+decoding+solutions+for+FPGA-based+time-to-digital+converters

9. A novel FPGA-based time-to-digital converter featuring machine learning-aided self-calibration — Arash Amini Bardpareh, Eleonora Vacca, Davide Nicolini, Corrado De Sio, Sarah Azimi, Luca Sterpone, Elisa Fiorina, Emanuele Maria Data, Felix Mas Milian, 2026

https://scholar.google.com/scholar?q=A+novel+FPGA-based+time-to-digital+converter+featuring+machine+learning-aided+self-calibration

10. Multiple-tapped-delay-line hardware-linearisation technique based on wire load regulation — Dariusz Chaberski, Robert Frankowski, Marek Zielinski, Lukasz Zaworski, 2016

https://scholar.google.com/scholar?q=Multiple-tapped-delay-line+hardware-linearisation+technique+based+on+wire+load+regulation

11. 5.7 ps Resolution Time-to-Digital Converter Implementation Using Routing Path Delays — Roza Teklehaimanot Siecha, Getachew Alemu, Jeffrey Prinzie, Paul Leroux, 2023

https://scholar.google.com/scholar?q=5.7+ps+Resolution+Time-to-Digital+Converter+Implementation+Using+Routing+Path+Delays

12. Tapped delay line for compact time-to-digital converter on UltraScale FPGA and its coding method — Min Zhu, Xihan Qi, Tang Cui, Qiang Gao, 2023

https://scholar.google.com/scholar?q=Tapped+delay+line+for+compact+time-to-digital+converter+on+UltraScale+FPGA+and+its+coding+method

13. A High-Resolution (<10 ps RMS) 48-Channel Time-to-Digital Converter (TDC) Implemented in a Field Programmable Gate Array (FPGA) — Eugen Bayer, Michael Traxler, 2011

https://scholar.google.com/scholar?q=A+High-Resolution+%28%3C10+ps+RMS%29+48-Channel+Time-to-Digital+Converter+%28TDC%29+Implemented+in+a+Field+Programmable+Gate+Array+%28FPGA%29

14. A Low Nonlinearity, Missing-Code Free Time-to-Digital Converter Based on 28-nm FPGAs With Embedded Bin-Width Calibrations — Haochang Chen, Yongliang Zhang, David Day-Uei Li, 2017

https://scholar.google.com/scholar?q=A+Low+Nonlinearity%2C+Missing-Code+Free+Time-to-Digital+Converter+Based+on+28-nm+FPGAs+With+Embedded+Bin-Width+Calibrations

15. High-Performance Time-to-Digital Converter IP-Core for Xilinx Ultrascale/Ultrascale+ FPGAs — Nicola Lusardi, Fabio Garzetti, E. Ronconi, Nicola Corna, Andrea Costa, Angelo Geraci, 2022

https://scholar.google.com/scholar?q=High-Performance+Time-to-Digital+Converter+IP-Core+for+Xilinx+Ultrascale%2FUltrascale%2B+FPGAs

16. An FPGA-Based Time-to-Digital Converter With Online Dual-Chain Calibration — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=An+FPGA-Based+Time-to-Digital+Converter+With+Online+Dual-Chain+Calibration

17. A review of advancements and trends in time-to-digital converters based on FPGA — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=A+review+of+advancements+and+trends+in+time-to-digital+converters+based+on+FPGA

18. A study on the effect of temperature variations on FPGA-based multi-channel time-to-digital converters — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=A+study+on+the+effect+of+temperature+variations+on+FPGA-based+multi-channel+time-to-digital+converters

19. Heterogeneous Tapped Delay-Line Time-to-Digital Converter on Artix-7 FPGA — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=Heterogeneous+Tapped+Delay-Line+Time-to-Digital+Converter+on+Artix-7+FPGA

20. A New Implementation of Tapped-Delay-Line Based High-Performance Time-to-Digital Converter on Xilinx Kintex-7 FPGA — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=A+New+Implementation+of+Tapped-Delay-Line+Based+High-Performance+Time-to-Digital+Converter+on+Xilinx+Kintex-7+FPGA

21. NN-Based Encoding Applied to a Time-to-Digital Converter for Time Interval Measurements — authors unclear from snippet, recent; exact year unclear

https://scholar.google.com/scholar?q=NN-Based+Encoding+Applied+to+a+Time-to-Digital+Converter+for+Time+Interval+Measurements

22. AI Post Transformers: FPGA Neural Network Accelerators for Space — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-26-fpga-neural-network-accelerators-for-spa-3087ae.mp3

23. AI Post Transformers: Caffeine: A Unified FPGA for CNNs — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-06-caffeine-a-unified-fpga-for-cnns-e8acbe.mp3

Interactive Visualization: Machine Learning Self-Calibrated FPGA Time-to-Digital Converter

This episode explores the TensorFlow paper as a systems argument for unifying the full machine learning lifecycle, from mobile inference to large-scale distributed training, within a single stateful dataflow framework. It explains how TensorFlow represents computation as graphs with mutable state, why that mattered for device placement, parameter storage, checkpointing, and heterogeneous hardware, and how it aimed to improve on the limitations of DistBelief. The discussion also places the paper in the broader lineage of MapReduce, Dryad, Naiad, and parameter-server training, while debating whether TensorFlow truly generalized machine learning workflows or mainly fit the kinds of static, graph-friendly workloads large organizations like Google already needed. Listeners would find it interesting for its mix of technical history, distributed systems insight, and a clear-eyed look at the tradeoff between organizational scale, portability, and usability for everyday researchers.

Sources:

1. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng, 2016

http://arxiv.org/abs/1603.04467

2. MapReduce: Simplified Data Processing on Large Clusters — Jeffrey Dean and Sanjay Ghemawat, 2004

https://scholar.google.com/scholar?q=MapReduce%3A+Simplified+Data+Processing+on+Large+Clusters

3. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks — Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly, 2007

https://scholar.google.com/scholar?q=Dryad%3A+Distributed+Data-Parallel+Programs+from+Sequential+Building+Blocks

4. Large Scale Distributed Deep Networks — Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng, 2012

https://scholar.google.com/scholar?q=Large+Scale+Distributed+Deep+Networks

5. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Jeffrey Dean, Rajat Monga, and many others, 2016

https://scholar.google.com/scholar?q=TensorFlow%3A+Large-Scale+Machine+Learning+on+Heterogeneous+Distributed+Systems

6. Naiad: A Timely Dataflow System — Frank McSherry, Derek G. Murray, Rebecca Isaacs, and Michael Isard, 2013

https://scholar.google.com/scholar?q=Naiad%3A+A+Timely+Dataflow+System

7. Project Adam: Building an Efficient and Scalable Deep Learning Training System — Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman, 2014

https://scholar.google.com/scholar?q=Project+Adam%3A+Building+an+Efficient+and+Scalable+Deep+Learning+Training+System

8. Parameter Server for Distributed Machine Learning — Mu Li, David G. Andersen, Alexander J. Smola, and Kai Yu, 2014

https://scholar.google.com/scholar?q=Parameter+Server+for+Distributed+Machine+Learning

9. Caffe: Convolutional Architecture for Fast Feature Embedding — Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell, 2014

https://scholar.google.com/scholar?q=Caffe%3A+Convolutional+Architecture+for+Fast+Feature+Embedding

10. Theano: A CPU and GPU Math Compiler in Python — James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio, 2010

https://scholar.google.com/scholar?q=Theano%3A+A+CPU+and+GPU+Math+Compiler+in+Python

11. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems — Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang, 2015

https://scholar.google.com/scholar?q=MXNet%3A+A+Flexible+and+Efficient+Machine+Learning+Library+for+Heterogeneous+Distributed+Systems

12. SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement — Shihong Gao, Yiming Li, Xin Zhang, Yanyan Shen, Yingxia Shao, Lei Chen, 2024

https://scholar.google.com/scholar?q=SIMPLE%3A+Efficient+Temporal+Graph+Neural+Network+Training+at+Scale+with+Dynamic+Data+Placement

13. Strategy-Switch: From All-Reduce to Parameter Server for Faster Efficient Training — Nikodimos Provatas, Iasonas Chalas, Ioannis Konstantinou, Nectarios Koziris, 2025

https://scholar.google.com/scholar?q=Strategy-Switch%3A+From+All-Reduce+to+Parameter+Server+for+Faster+Efficient+Training

14. Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models — Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, Xiaowen Chu, 2023

https://scholar.google.com/scholar?q=Dissecting+the+Runtime+Performance+of+the+Training%2C+Fine-tuning%2C+and+Inference+of+Large+Language+Models

15. AI Post Transformers: ONNX Ecosystem, Optimization, and Deployment — Hal Turing & Dr. Ada Shannon, 2025

https://podcast.do-not-panic.com/episodes/onnx-ecosystem-optimization-and-deployment/

Interactive Visualization: TensorFlow for Distributed Machine Learning Systems

This episode explores SGLang, a system for making complex language model workflows run faster by treating them as full programs rather than single prompt-response calls. It explains how modern LLM applications involve branching, tool use, retries, and structured outputs, then examines SGLang’s co-design of a Python-embedded language with a specialized runtime that can optimize those patterns directly. The discussion highlights ideas like KV-cache reuse through RadixAttention, grammar-constrained decoding for reliable JSON output, and why these systems techniques matter more than just nicer prompt scripting. Listeners would find it interesting because it connects practical agent-style LLM engineering to deeper questions about compilers, serving infrastructure, and whether headline speedups really hold across real workloads.

Sources:

1. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023

http://arxiv.org/abs/2312.07104

2. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning — Saibo Geng, Martin Josifoski, Maxime Peyrard, Robert West, 2023

https://scholar.google.com/scholar?q=Grammar-Constrained+Decoding+for+Structured+NLP+Tasks+without+Finetuning

3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs

4. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models — Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, Tianqi Chen, 2024

https://scholar.google.com/scholar?q=XGrammar%3A+Flexible+and+Efficient+Structured+Generation+Engine+for+Large+Language+Models

5. Generating Structured Outputs from Language Models: Benchmark and Studies — Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, Harsha Nori, 2025

https://scholar.google.com/scholar?q=Generating+Structured+Outputs+from+Language+Models%3A+Benchmark+and+Studies

6. Language Model Cascades — David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-Dickstein, Kevin Murphy, Charles Sutton, 2022

https://scholar.google.com/scholar?q=Language+Model+Cascades

7. Prompting Is Programming: A Query Language for Large Language Models — Luca Beurer-Kellner, Marc Fischer, Martin Vechev, 2023

https://scholar.google.com/scholar?q=Prompting+Is+Programming%3A+A+Query+Language+for+Large+Language+Models

8. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines — Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts, 2023

https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+Self-Improving+Pipelines

9. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023

https://scholar.google.com/scholar?q=vLLM%3A+Easy%2C+Fast%2C+and+Cheap+LLM+Serving+with+PagedAttention

10. Guidance: A Guidance Language for Controlling Large Language Models — Microsoft Research and collaborators, 2023

https://scholar.google.com/scholar?q=Guidance%3A+A+Guidance+Language+for+Controlling+Large+Language+Models

11. LMQL: A Programming Language for Large Language Models — Luca Beurer-Kellner, Marc Fischer, Martin Vechev, 2023

https://scholar.google.com/scholar?q=LMQL%3A+A+Programming+Language+for+Large+Language+Models

12. Outlines — Thibault Glaunec and contributors, 2023

https://scholar.google.com/scholar?q=Outlines

13. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, Shiyu Chang, 2025

https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse

14. Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques — Neusha Javidnia, Bita Rouhani, Farinaz Koushanfar, 2025

https://scholar.google.com/scholar?q=Key%2C+Value%2C+Compress%3A+A+Systematic+Exploration+of+KV+Cache+Compression+Techniques

15. KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models — Sourjya Roy, Shrihari Sridharan, Surya Selvam, Anand Raghunathan, 2025

https://scholar.google.com/scholar?q=KV-CAR%3A+KV+Cache+Compression+using+Autoencoders+and+KV+Reuse+in+Large+Language+Models

16. Grammar-Aligned Decoding — Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, Loris D'Antoni, 2024

https://scholar.google.com/scholar?q=Grammar-Aligned+Decoding

17. Grammar-Constrained Decoding Makes Large Language Models Better Logical Parsers — Federico Raspanti, Tanir Ozcelebi, Mike J. Holenderski, 2025

https://scholar.google.com/scholar?q=Grammar-Constrained+Decoding+Makes+Large+Language+Models+Better+Logical+Parsers

18. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, Ravi Netravali, 2025

https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs

19. Towards Efficient Agents: A Co-Design of Inference Architecture and System — Weizhe Lin, Hui-Ling Zhen, Shuai Yang, Xian Wang, Renxi Liu, Hanting Chen, Wangze Zhang, Chuansai Zhou, Yiming Li, Chen Chen, Xing Li, Zhiyuan Yang, Xiaosong Li, Xianzhi Yu, Zhenhua Dong, Mingxuan Yuan, Yunhe Wang, 2025

https://scholar.google.com/scholar?q=Towards+Efficient+Agents%3A+A+Co-Design+of+Inference+Architecture+and+System

20. Optimizing Agentic Language Model Inference via Speculative Tool Calls — Daniel Nichols, Prajwal Singhania, Charles Jekel, Abhinav Bhatele, Harshitha Menon, 2025

https://scholar.google.com/scholar?q=Optimizing+Agentic+Language+Model+Inference+via+Speculative+Tool+Calls

21. AI Post Transformers: SGLang: Efficient Language Model Program Execution — Hal Turing & Dr. Ada Shannon, Sun,

https://podcast.do-not-panic.com/episodes/sglang-efficient-language-model-program-execution/

22. AI Post Transformers: Breaking the Prefix Barrier with Shared KV Cache — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-24-breaking-the-prefix-barrier-with-shared-a5e5a6.mp3

23. AI Post Transformers: Continuous Batching for LLM Inference: Throughput and Latency Gains — Hal Turing & Dr. Ada Shannon, Mon,

https://podcast.do-not-panic.com/episodes/continuous-batching-for-llm-inference-throughput-and-latency-gains/

24. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3

25. AI Post Transformers: KV Cache TTL for Multi-Turn Agent Scheduling — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-09-kv-cache-ttl-for-multi-turn-agent-schedu-996bf1.mp3

26. AI Post Transformers: TokenDance for Multi-Agent KV Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-22-tokendance-for-multi-agent-kv-cache-shar-aa9b99.mp3

Interactive Visualization: SGLang for Faster Structured LLM Programs

This episode explores CL-BENCH, a benchmark designed to test whether language models can actually learn task-specific knowledge from long, messy context and then reason with it, rather than merely retrieving facts or mimicking examples. It explains the distinction between long-context understanding, in-context learning, and the stronger notion of context learning, using examples like legal codes, product manuals, and experimental notebooks to show what real-world adaptation demands. The discussion highlights how the benchmark’s 500 contexts, 1,899 tasks, and dense binary verification rubrics are built to stress models on rule-following, procedural reasoning, and inferring governing relationships from data. Listeners would find it interesting because it gets at a central question in modern AI: whether bigger context windows actually make systems more capable, or just better at holding more text without truly learning from it.

Sources:

1. CL-bench: A Benchmark for Context Learning — Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, Tao Gui, Zuxuan Wu, Xipeng Qiu, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Di Wang, Shunyu Yao, 2026

http://arxiv.org/abs/2602.03587

2. Language Models are Few-Shot Learners — Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan and others, 2020

https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners

3. MetaICL: Learning to Learn In Context — Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi, 2021

https://scholar.google.com/scholar?q=MetaICL%3A+Learning+to+Learn+In+Context

4. Transformers learn in-context by gradient descent — Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov, 2022

https://scholar.google.com/scholar?q=Transformers+learn+in-context+by+gradient+descent

5. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

6. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding — Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2023

https://scholar.google.com/scholar?q=LongBench%3A+A+Bilingual%2C+Multitask+Benchmark+for+Long+Context+Understanding

7. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Yurii Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev, 2024

https://scholar.google.com/scholar?q=BABILong%3A+Testing+the+Limits+of+LLMs+with+Long+Context+Reasoning-in-a-Haystack

8. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024

https://scholar.google.com/scholar?q=LongBench+v2%3A+Towards+Deeper+Understanding+and+Reasoning+on+Realistic+Long-context+Multitasks

9. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, David Seunghyun Yoon, Hinrich Schutze, 2025

https://scholar.google.com/scholar?q=NoLiMa%3A+Long-Context+Evaluation+Beyond+Literal+Matching

10. LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion — Zhan Ling et al., 2025

https://scholar.google.com/scholar?q=LongReason%3A+A+Synthetic+Long-Context+Reasoning+Benchmark+via+Context+Expansion

11. DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities — Tianyi Zhuang et al., 2025

https://scholar.google.com/scholar?q=DocPuzzle%3A+A+Process-Aware+Benchmark+for+Evaluating+Realistic+Long-Context+Reasoning+Capabilities

12. In-Context Learning Creates Task Vectors — Roee Hendel, Mor Geva, Amir Globerson, 2023

https://scholar.google.com/scholar?q=In-Context+Learning+Creates+Task+Vectors

13. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering — Sheng Liu, Haotian Ye, Lei Xing, James Zou, 2024

https://scholar.google.com/scholar?q=In-context+Vectors%3A+Making+In+Context+Learning+More+Effective+and+Controllable+Through+Latent+Space+Steering

14. Task Vectors in In-Context Learning: Emergence, Formation, and Benefit — Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, Robert Nowak, 2025

https://scholar.google.com/scholar?q=Task+Vectors+in+In-Context+Learning%3A+Emergence%2C+Formation%2C+and+Benefit

15. Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory — Guangyue Peng, Tao Ge, Wen Luo, Wei Li, Houfeng Wang, 2025

https://scholar.google.com/scholar?q=Learn+to+Memorize%3A+Scalable+Continual+Learning+in+Semiparametric+Models+with+Mixture-of-Neighbors+Induction+Memory

16. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp3

17. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp3

18. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp3

19. AI Post Transformers: In-Place Test-Time Training for Transformers — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-09-in-place-test-time-training-for-transfor-d0b976.mp3

20. AI Post Transformers: Training LLMs for Divide-and-Conquer Reasoning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-04-training-llms-for-divide-and-conquer-rea-ea6e22.mp3

21. AI Post Transformers: Inverse IFEval: Unlearning LLM Cognitive Inertia — Hal Turing & Dr. Ada Shannon, 2025

https://podcast.do-not-panic.com/episodes/inverse-ifeval-unlearning-llm-cognitive-inertia/

Interactive Visualization: Can Models Learn from Long Context?

This episode explores the 1987 paper on synchronous data flow and how it turns stream-processing programs into analyzable graphs with fixed token production and consumption rates. It explains how those fixed rates let a compiler precompute a repeating execution schedule, prove steady-state consistency through balance equations, and allocate bounded buffers ahead of time instead of relying on expensive runtime scheduling. The discussion highlights why that tradeoff works so well for digital signal processing workloads like filtering, resampling, and codecs, while also showing why the model is too restrictive for messier software with irregular control flow. Listeners would find it interesting because it shows how a carefully limited programming model can unlock strong guarantees about performance, memory use, and parallel execution.

Sources:

1. Synchronous Data Flow for Signal Processing

https://ptolemy.berkeley.edu/publications/papers/87/synchdataflow/synchdataflow.pdf

2. Synchronous Data Flow — Edward A. Lee and David G. Messerschmitt, 1987

https://scholar.google.com/scholar?q=Synchronous+Data+Flow

3. Dataflow Process Networks — Edward A. Lee and Thomas M. Parks, 1995

https://scholar.google.com/scholar?q=Dataflow+Process+Networks

4. Cycle-Static Dataflow — Greet Bilsen, Marc Engels, Rudy Lauwereins, and Jean Peperstraete, 1996

https://scholar.google.com/scholar?q=Cycle-Static+Dataflow

5. Static Scheduling of Synchronous Data Flow Programs for Digital Signal Processing — Edward A. Lee and David G. Messerschmitt, 1987

https://scholar.google.com/scholar?q=Static+Scheduling+of+Synchronous+Data+Flow+Programs+for+Digital+Signal+Processing

6. Bounded Scheduling of Process Networks — Thomas M. Parks, 1995

https://scholar.google.com/scholar?q=Bounded+Scheduling+of+Process+Networks

7. Synthesis of Embedded Software from Synchronous Dataflow Specifications — Shuvra S. Bhattacharyya, Praveen K. Murthy, and Edward A. Lee, 1999

https://scholar.google.com/scholar?q=Synthesis+of+Embedded+Software+from+Synchronous+Dataflow+Specifications

8. StreamIt: A Language for Streaming Applications — William Thies, Michal Karczmarek, and Saman Amarasinghe, 2002

https://scholar.google.com/scholar?q=StreamIt%3A+A+Language+for+Streaming+Applications

9. Memory Management for Dataflow Programming of Multirate Signal Processing Algorithms — Shuvra S. Bhattacharyya and Edward A. Lee, 1994

https://scholar.google.com/scholar?q=Memory+Management+for+Dataflow+Programming+of+Multirate+Signal+Processing+Algorithms

10. Joint Minimization of Code and Data for Synchronous Dataflow Programs — Praveen K. Murthy, Shuvra S. Bhattacharyya, and Edward A. Lee, 1994

https://scholar.google.com/scholar?q=Joint+Minimization+of+Code+and+Data+for+Synchronous+Dataflow+Programs

11. Buffer Merging: A Powerful Technique for Reducing Memory Requirements of Synchronous Dataflow Specifications — Praveen K. Murthy and Shuvra S. Bhattacharyya, 2000

https://scholar.google.com/scholar?q=Buffer+Merging%3A+A+Powerful+Technique+for+Reducing+Memory+Requirements+of+Synchronous+Dataflow+Specifications

12. Pipeline Interleaved Programmable DSP's: Synchronous Data Flow Programming — Edward A. Lee and David G. Messerschmitt, 1987

https://scholar.google.com/scholar?q=Pipeline+Interleaved+Programmable+DSP%27s%3A+Synchronous+Data+Flow+Programming

13. Multirate Digital Filters, Filter Banks, Polyphase Networks, and Applications: A Tutorial — P. P. Vaidyanathan, 1990

https://scholar.google.com/scholar?q=Multirate+Digital+Filters%2C+Filter+Banks%2C+Polyphase+Networks%2C+and+Applications%3A+A+Tutorial

14. The Semantics of a Simple Language for Parallel Programming — Gilles Kahn, 1974

https://scholar.google.com/scholar?q=The+Semantics+of+a+Simple+Language+for+Parallel+Programming

15. First Version of a Data Flow Procedure Language — Jack B. Dennis, 1974

https://scholar.google.com/scholar?q=First+Version+of+a+Data+Flow+Procedure+Language

16. On the Boundedness of Process Networks — Gilles Kahn and David B. MacQueen, 1977

https://scholar.google.com/scholar?q=On+the+Boundedness+of+Process+Networks

17. Algorithm Design for Signal Processing — Charles S. Burrus, 1982

https://scholar.google.com/scholar?q=Algorithm+Design+for+Signal+Processing

18. DynVec: An End-to-End Framework for Efficient Vector-Dataflow Execution — approximate; recent systems/compiler authors, recent

https://scholar.google.com/scholar?q=DynVec%3A+An+End-to-End+Framework+for+Efficient+Vector-Dataflow+Execution

19. Compiler discovered dynamic scheduling of irregular code in high-level synthesis — approximate; recent HLS/compiler authors, recent

https://scholar.google.com/scholar?q=Compiler+discovered+dynamic+scheduling+of+irregular+code+in+high-level+synthesis

20. Dataflow Models of computation for programming heterogeneous multicores — approximate; recent embedded/parallel-systems authors, recent

https://scholar.google.com/scholar?q=Dataflow+Models+of+computation+for+programming+heterogeneous+multicores

21. Heuristic & Expert-Guided Buffer Sizing for Neural Network Inference Applications on FPGAs — approximate; recent FPGA/dataflow authors, recent

https://scholar.google.com/scholar?q=Heuristic+%26+Expert-Guided+Buffer+Sizing+for+Neural+Network+Inference+Applications+on+FPGAs

22. Sgcn: Exploiting compressed-sparse features in deep graph convolutional network accelerators — approximate; recent accelerator authors, recent

https://scholar.google.com/scholar?q=Sgcn%3A+Exploiting+compressed-sparse+features+in+deep+graph+convolutional+network+accelerators

23. Safe shared state in dataflow systems — approximate; recent programming-systems authors, recent

https://scholar.google.com/scholar?q=Safe+shared+state+in+dataflow+systems

24. An Intermediate Representation for Stateful Dataflows — approximate; recent systems authors, recent

https://scholar.google.com/scholar?q=An+Intermediate+Representation+for+Stateful+Dataflows

25. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp3

26. AI Post Transformers: Caffeine: A Unified FPGA for CNNs — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-06-caffeine-a-unified-fpga-for-cnns-e8acbe.mp3

27. AI Post Transformers: Caffe and the Rise of CNN Frameworks — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-06-caffe-and-the-rise-of-cnn-frameworks-cf15f3.mp3

Interactive Visualization: Synchronous Data Flow for Signal Processing

This episode explores FP-DNN, a 2017 framework that aims to compile TensorFlow-era neural networks onto FPGAs automatically, reducing the need for hand-designed accelerators for each model. It explains how the system maps convolutional layers, fully connected layers, and parts of LSTM computation into a shared matrix-multiplication core, while combining hand-tuned RTL for performance-critical components with HLS-generated logic for orchestration and layer-specific handling. The discussion highlights why this hybrid design matters for performance-per-watt, latency, and communication efficiency, especially as deeper CNNs and recurrent models were pushing hardware limits. Listeners would find it interesting for its clear look at an early attempt to turn FPGA deployment from an expert-only craft into a more reusable compiler-driven workflow, while also showing where the paper’s claims about broad model coverage may be too optimistic.

Sources:

1. Automating DNN Compilation for FPGA Accelerators

https://ceca.pku.edu.cn/media/lw/e3d0e0cd92452e0504b148220d442b9a.pdf

2. A Survey of FPGA-based Neural Network Inference Accelerators — Kaiyuan Guo, Shulin Zeng, Jincheng Yu, Yu Wang, Huazhong Yang, 2019

https://scholar.google.com/scholar?q=A+Survey+of+FPGA-based+Neural+Network+Inference+Accelerators

3. DeepBurning: Automatic Generation of FPGA-based Learning Accelerators for the Neural Network Family — Ying Wang, Jie Xu, Yudeng Sun, Baohua Cao, Chunyuan Xu, Yibo Kong, Chundao Han, Xuan Wang, 2016

https://scholar.google.com/scholar?q=DeepBurning%3A+Automatic+Generation+of+FPGA-based+Learning+Accelerators+for+the+Neural+Network+Family

4. FP-DNN: An Automated Framework for Mapping Deep Neural Networks onto FPGAs with RTL-HLS Hybrid Templates — Yijin Guan, Hao Liang, Ningyi Xu, Wenqiang Wang, Shaoshuai Shi, Xi Chen, Guangyu Sun, Wei Zhang, Jason Cong, 2017

https://scholar.google.com/scholar?q=FP-DNN%3A+An+Automated+Framework+for+Mapping+Deep+Neural+Networks+onto+FPGAs+with+RTL-HLS+Hybrid+Templates

5. DNNBuilder: An Automated Tool for Building High-Performance DNN Hardware Accelerators for FPGAs — Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin, Jinjun Xiong, Wen-Mei Hwu, Deming Chen, 2018

https://scholar.google.com/scholar?q=DNNBuilder%3A+An+Automated+Tool+for+Building+High-Performance+DNN+Hardware+Accelerators+for+FPGAs

6. From High-Level Deep Neural Models to FPGAs — Hardik Sharma, Jongse Park, Emmanuel Amaro, Bradley Thwaites, Priyanka Kotha, Anmol Gupta, Joon Kyung Kim, Asit Mishra, and Hsien-Hsin S. Lee, 2016

https://scholar.google.com/scholar?q=From+High-Level+Deep+Neural+Models+to+FPGAs

7. Caffeine: Towards Uniformed Representation and Acceleration for Deep Convolutional Neural Networks — Chen Zhang, Zhenman Fang, Peipei Zhou, Peichen Pan, and Jason Cong, 2016

https://scholar.google.com/scholar?q=Caffeine%3A+Towards+Uniformed+Representation+and+Acceleration+for+Deep+Convolutional+Neural+Networks

8. Throughput-Optimized OpenCL-Based FPGA Accelerator for Large-Scale Convolutional Neural Networks — Naveen Suda, Vikas Chandra, Ganesh Dasika, Abinash Mohanty, Yufei Ma, Sarita Vrudhula, Jae-sun Seo, and Yu Cao, 2016

https://scholar.google.com/scholar?q=Throughput-Optimized+OpenCL-Based+FPGA+Accelerator+for+Large-Scale+Convolutional+Neural+Networks

9. Going Deeper with Embedded FPGA Platform for Convolutional Neural Network — Jiantao Qiu, Jie Wang, Song Yao, Kai Guo, Boxun Li, Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, and Song Wang, 2016

https://scholar.google.com/scholar?q=Going+Deeper+with+Embedded+FPGA+Platform+for+Convolutional+Neural+Network

10. Optimizing FPGA-Based Accelerator Design for Deep Convolutional Neural Networks — Chen Zhang, Peng Li, Guangyu Sun, Yijin Guan, Bingjun Xiao, and Jason Cong, 2015

https://scholar.google.com/scholar?q=Optimizing+FPGA-Based+Accelerator+Design+for+Deep+Convolutional+Neural+Networks

11. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding — Song Han, Huizi Mao, and William J. Dally, 2015

https://scholar.google.com/scholar?q=Deep+Compression%3A+Compressing+Deep+Neural+Networks+with+Pruning%2C+Trained+Quantization+and+Huffman+Coding

12. BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach — Zhen Zheng et al., 2023

https://scholar.google.com/scholar?q=BladeDISC%3A+Optimizing+Dynamic+Shape+Machine+Learning+Workloads+via+Compiler+Approach

13. TSCompiler: efficient compilation framework for dynamic-shape models — Xiang Luo, Chen Zhang, Chenbo Geng, Yanzhi Yi, Jiahui Hu, Renwei Zhang, Zhen Zhang, Gianpietro Consolaro, Fan Yang, Tun Lu, Ning Gu, Li Shang, 2024

https://scholar.google.com/scholar?q=TSCompiler%3A+efficient+compilation+framework+for+dynamic-shape+models

14. TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture — Jiajun Wu, Mo Song, Jingmin Zhao, Yizhao Gao, Jia Li, Hayden Kwok-Hay So, 2024

https://scholar.google.com/scholar?q=TATAA%3A+Programmable+Mixed-Precision+Transformer+Acceleration+with+a+Transformable+Arithmetic+Architecture

15. FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models — Woohong Byun, Jongseok Woo, Saibal Mukhopadhyay, 2025

https://scholar.google.com/scholar?q=FPGA+Acceleration+With+Hessian-Based+Comprehensive+Intra-Layer+Mixed-Precision+Quantization+for+Transformer+Models

16. Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference — Zifan He, Rui Ma, Yizhou Sun, Jason Cong, 2026

https://scholar.google.com/scholar?q=Understand+and+Accelerate+Memory+Processing+Pipeline+for+Disaggregated+LLM+Inference

17. AI Post Transformers: FPGA Neural Network Accelerators for Space — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-26-fpga-neural-network-accelerators-for-spa-3087ae.mp3

18. AI Post Transformers: Continuous Batching for LLM Inference: Throughput and Latency Gains — Hal Turing & Dr. Ada Shannon, 2025

https://podcast.do-not-panic.com/episodes/continuous-batching-for-llm-inference-throughput-and-latency-gains/

19. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/advancements-in-efficient-kv-cache-quantization-and-management/

Interactive Visualization: Automating DNN Compilation for FPGA Accelerators