Sign up to save your podcasts
Or
Share Tokenization Bias: The Hidden Flaw Breaking Language Models
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Tokenization Bias: The Hidden Flaw Breaking Language Models
Hal Turing and Dr. Ada Shannon open the episode by confronting a structural flaw that has been hiding in plain sight since the transformer era began: tokenization bias. The episode centers on "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich (ICLR 2025), which formally proves that a tokenized model and a byte-level model can be statistically equivalent and still produce wildly different predictions for the same next character. The hosts trace the origins of the problem through BPE's introduction by Rico Sennrich, Barry Haddow, and Alexandra Birch in 2016 and its industrialization via Kudo and Richardson's SentencePiece in 2018 — a library now frozen into the spine of LLaMA, Mistral, Gemma, and most open-source models not in the OpenAI lineage.
The discussion sharpens around fill-in-the-middle prompting, the paradigm introduced by Mohammad Bavarian and colleagues at OpenAI in 2022 and now embedded in every major code completion tool from GitHub Copilot to StarCoder. Shannon walks through the paper's central example: a code completion scenario where the correct next character receives a probability of exactly zero — not a rounding artifact but a structural impossibility, because the tokenizer has carved up the prompt in a way that makes the right answer unreachable in token-space. Turing challenges the framing, arguing that byte-level alternatives like ByT5 and MegaByte existed and BPE was an informed trade-off against the three-to-eight times sequence length penalty that raw bytes impose on attention compute. Shannon holds the line: the point is not that BPE was a mistake but that its systematic bias was never formally characterized until now, and the Byte-Token Representation Lemma finally gives the field the mathematical language to name and measure it.
The episode closes by introducing the second paper from the episode's pairing — Minixhofer et al.'s NeurIPS 2025 work on cross-tokenizer knowledge distillation — which attacks the tokenizer barrier from the training side rather than the inference side. Where Phan et al. offer a zero-shot correction algorithm that recovers 18% on fill-in-the-middle coding benchmarks without any retraining, Minixhofer et al. enable knowledge transfer between models with fundamentally incompatible vocabularies, breaking the assumption that distillation requires shared tokenization. Together the two papers sketch a trajectory where tokenization becomes a transparent implementation detail rather than an architectural constraint that determines what a model can and cannot express.
Sources:
1. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles — Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, Karen Ullrich, 2024
http://arxiv.org/abs/2410.09303
2. https://arxiv.org/pdf/2503.20083
3. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units
4. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Taku Kudo, John Richardson, 2018
https://scholar.google.com/scholar?q=SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+Neural+Text+Processing
5. Toward a Theory of Tokenization in LLMs — Nived Rajaraman, Jiantao Jiao, Kannan Ramchandran, 2024
https://scholar.google.com/scholar?q=Toward+a+Theory+of+Tokenization+in+LLMs
6. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models — Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, 2021
https://scholar.google.com/scholar?q=How+Good+is+Your+Tokenizer%3F+On+the+Monolingual+Performance+of+Multilingual+Language+Models
7. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel, 2022
https://scholar.google.com/scholar?q=ByT5%3A+Towards+a+Token-Free+Future+with+Pre-trained+Byte-to-Byte+Models
8. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis, 2023
https://scholar.google.com/scholar?q=MEGABYTE%3A+Predicting+Million-byte+Sequences+with+Multiscale+Transformers
9. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation — Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting, 2022
https://scholar.google.com/scholar?q=CANINE%3A+Pre-training+an+Efficient+Tokenization-Free+Encoder+for+Language+Representation
10. Charformer: Fast Character Transformers via Gradient-Based Subword Tokenization — Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Kornblith, Cecelia Zhang, Donald Metzler, Mostafa Dehghani, 2022
https://scholar.google.com/scholar?q=Charformer%3A+Fast+Character+Transformers+via+Gradient-Based+Subword+Tokenization
11. Efficient Training of Language Models to Fill in the Middle — Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen, 2022
https://scholar.google.com/scholar?q=Efficient+Training+of+Language+Models+to+Fill+in+the+Middle
12. Evaluating Large Language Models Trained on Code — Mark Chen, Jerry Tworek, Heewoo Jun, and a large team at OpenAI, 2021
https://scholar.google.com/scholar?q=Evaluating+Large+Language+Models+Trained+on+Code
13. Code Llama: Open Foundation Models for Code — Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, and others at Meta AI, 2023
https://scholar.google.com/scholar?q=Code+Llama%3A+Open+Foundation+Models+for+Code
14. StarCoder: May the Source Be With You! — Raymond Li, Loubna Ben Allal, and a large BigCode / HuggingFace / ServiceNow collaboration, 2023
https://scholar.google.com/scholar?q=StarCoder%3A+May+the+Source+Be+With+You%21
15. Training Products of Experts by Minimizing Contrastive Divergence — Geoffrey E. Hinton, 2002
https://scholar.google.com/scholar?q=Training+Products+of+Experts+by+Minimizing+Contrastive+Divergence
16. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2023
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
17. Knowledge Fusion of Large Language Models — Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi, 2024
https://scholar.google.com/scholar?q=Knowledge+Fusion+of+Large+Language+Models
18. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023
https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion
19. Fill in the Middle: A New Training Objective for Language Models — Bavarian et al., 2022
https://scholar.google.com/scholar?q=Fill+in+the+Middle%3A+A+New+Training+Objective+for+Language+Models
20. CodeFusion: A Pre-trained Diffusion Model for Code Generation — Dagan et al., 2024
https://scholar.google.com/scholar?q=CodeFusion%3A+A+Pre-trained+Diffusion+Model+for+Code+Generation
21. Is Tokenization More Than Compression? — Rajaraman et al., 2024
https://scholar.google.com/scholar?q=Is+Tokenization+More+Than+Compression%3F
22. SpaceByte: Towards Deleting Tokenization from Large Language Modeling — Slagle, 2024
https://scholar.google.com/scholar?q=SpaceByte%3A+Towards+Deleting+Tokenization+from+Large+Language+Modeling
23. StarCoder 2 and The Stack v2: The Next Generation — Lozhkov et al., 2024
https://scholar.google.com/scholar?q=StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation
24. Byte Latent Transformer: Patches Scale Better Than Tokens — Pagnoni et al. (Meta AI), 2024
https://scholar.google.com/scholar?q=Byte+Latent+Transformer%3A+Patches+Scale+Better+Than+Tokens
25. Token-level Ensembling of Models with Different Vocabularies — various, 2024
https://scholar.google.com/scholar?q=Token-level+Ensembling+of+Models+with+Different+Vocabularies
26. Bridging the Gap Between Different Vocabularies for LLM Ensemble — various, 2024
https://scholar.google.com/scholar?q=Bridging+the+Gap+Between+Different+Vocabularies+for+LLM+Ensemble
27. Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies — various, 2024
https://scholar.google.com/scholar?q=Accelerating+LLM+Inference+with+Lossless+Speculative+Decoding+Algorithms+for+Heterogeneous+Vocabularies
28. Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm — various, 2024
https://scholar.google.com/scholar?q=Bridging+Developer+Instructions+and+Code+Completion+Through+Instruction-Aware+Fill-in-the-Middle+Paradigm
29. AI Post Transformers: Fast Inference from Transformers via Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Fast-Inference-from-Transformers-via-Speculative-Decoding-e3foclv
30. AI Post Transformers: Multiagent Debate Improves Language Model Reasoning — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Multiagent-Debate-Improves-Language-Model-Reasoning-e36kfd4
Interactive Visualization: Tokenization Bias: The Hidden Flaw Breaking Language Models
View all episodes
By mcgrofHal Turing and Dr. Ada Shannon open the episode by confronting a structural flaw that has been hiding in plain sight since the transformer era began: tokenization bias. The episode centers on "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich (ICLR 2025), which formally proves that a tokenized model and a byte-level model can be statistically equivalent and still produce wildly different predictions for the same next character. The hosts trace the origins of the problem through BPE's introduction by Rico Sennrich, Barry Haddow, and Alexandra Birch in 2016 and its industrialization via Kudo and Richardson's SentencePiece in 2018 — a library now frozen into the spine of LLaMA, Mistral, Gemma, and most open-source models not in the OpenAI lineage.
The discussion sharpens around fill-in-the-middle prompting, the paradigm introduced by Mohammad Bavarian and colleagues at OpenAI in 2022 and now embedded in every major code completion tool from GitHub Copilot to StarCoder. Shannon walks through the paper's central example: a code completion scenario where the correct next character receives a probability of exactly zero — not a rounding artifact but a structural impossibility, because the tokenizer has carved up the prompt in a way that makes the right answer unreachable in token-space. Turing challenges the framing, arguing that byte-level alternatives like ByT5 and MegaByte existed and BPE was an informed trade-off against the three-to-eight times sequence length penalty that raw bytes impose on attention compute. Shannon holds the line: the point is not that BPE was a mistake but that its systematic bias was never formally characterized until now, and the Byte-Token Representation Lemma finally gives the field the mathematical language to name and measure it.
The episode closes by introducing the second paper from the episode's pairing — Minixhofer et al.'s NeurIPS 2025 work on cross-tokenizer knowledge distillation — which attacks the tokenizer barrier from the training side rather than the inference side. Where Phan et al. offer a zero-shot correction algorithm that recovers 18% on fill-in-the-middle coding benchmarks without any retraining, Minixhofer et al. enable knowledge transfer between models with fundamentally incompatible vocabularies, breaking the assumption that distillation requires shared tokenization. Together the two papers sketch a trajectory where tokenization becomes a transparent implementation detail rather than an architectural constraint that determines what a model can and cannot express.
Sources:
1. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles — Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, Karen Ullrich, 2024
http://arxiv.org/abs/2410.09303
2. https://arxiv.org/pdf/2503.20083
3. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units
4. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Taku Kudo, John Richardson, 2018
https://scholar.google.com/scholar?q=SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+Neural+Text+Processing
5. Toward a Theory of Tokenization in LLMs — Nived Rajaraman, Jiantao Jiao, Kannan Ramchandran, 2024
https://scholar.google.com/scholar?q=Toward+a+Theory+of+Tokenization+in+LLMs
6. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models — Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, 2021
https://scholar.google.com/scholar?q=How+Good+is+Your+Tokenizer%3F+On+the+Monolingual+Performance+of+Multilingual+Language+Models
7. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel, 2022
https://scholar.google.com/scholar?q=ByT5%3A+Towards+a+Token-Free+Future+with+Pre-trained+Byte-to-Byte+Models
8. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis, 2023
https://scholar.google.com/scholar?q=MEGABYTE%3A+Predicting+Million-byte+Sequences+with+Multiscale+Transformers
9. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation — Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting, 2022
https://scholar.google.com/scholar?q=CANINE%3A+Pre-training+an+Efficient+Tokenization-Free+Encoder+for+Language+Representation
10. Charformer: Fast Character Transformers via Gradient-Based Subword Tokenization — Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Kornblith, Cecelia Zhang, Donald Metzler, Mostafa Dehghani, 2022
https://scholar.google.com/scholar?q=Charformer%3A+Fast+Character+Transformers+via+Gradient-Based+Subword+Tokenization
11. Efficient Training of Language Models to Fill in the Middle — Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen, 2022
https://scholar.google.com/scholar?q=Efficient+Training+of+Language+Models+to+Fill+in+the+Middle
12. Evaluating Large Language Models Trained on Code — Mark Chen, Jerry Tworek, Heewoo Jun, and a large team at OpenAI, 2021
https://scholar.google.com/scholar?q=Evaluating+Large+Language+Models+Trained+on+Code
13. Code Llama: Open Foundation Models for Code — Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, and others at Meta AI, 2023
https://scholar.google.com/scholar?q=Code+Llama%3A+Open+Foundation+Models+for+Code
14. StarCoder: May the Source Be With You! — Raymond Li, Loubna Ben Allal, and a large BigCode / HuggingFace / ServiceNow collaboration, 2023
https://scholar.google.com/scholar?q=StarCoder%3A+May+the+Source+Be+With+You%21
15. Training Products of Experts by Minimizing Contrastive Divergence — Geoffrey E. Hinton, 2002
https://scholar.google.com/scholar?q=Training+Products+of+Experts+by+Minimizing+Contrastive+Divergence
16. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2023
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
17. Knowledge Fusion of Large Language Models — Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi, 2024
https://scholar.google.com/scholar?q=Knowledge+Fusion+of+Large+Language+Models
18. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023
https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion
19. Fill in the Middle: A New Training Objective for Language Models — Bavarian et al., 2022
https://scholar.google.com/scholar?q=Fill+in+the+Middle%3A+A+New+Training+Objective+for+Language+Models
20. CodeFusion: A Pre-trained Diffusion Model for Code Generation — Dagan et al., 2024
https://scholar.google.com/scholar?q=CodeFusion%3A+A+Pre-trained+Diffusion+Model+for+Code+Generation
21. Is Tokenization More Than Compression? — Rajaraman et al., 2024
https://scholar.google.com/scholar?q=Is+Tokenization+More+Than+Compression%3F
22. SpaceByte: Towards Deleting Tokenization from Large Language Modeling — Slagle, 2024
https://scholar.google.com/scholar?q=SpaceByte%3A+Towards+Deleting+Tokenization+from+Large+Language+Modeling
23. StarCoder 2 and The Stack v2: The Next Generation — Lozhkov et al., 2024
https://scholar.google.com/scholar?q=StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation
24. Byte Latent Transformer: Patches Scale Better Than Tokens — Pagnoni et al. (Meta AI), 2024
https://scholar.google.com/scholar?q=Byte+Latent+Transformer%3A+Patches+Scale+Better+Than+Tokens
25. Token-level Ensembling of Models with Different Vocabularies — various, 2024
https://scholar.google.com/scholar?q=Token-level+Ensembling+of+Models+with+Different+Vocabularies
26. Bridging the Gap Between Different Vocabularies for LLM Ensemble — various, 2024
https://scholar.google.com/scholar?q=Bridging+the+Gap+Between+Different+Vocabularies+for+LLM+Ensemble
27. Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies — various, 2024
https://scholar.google.com/scholar?q=Accelerating+LLM+Inference+with+Lossless+Speculative+Decoding+Algorithms+for+Heterogeneous+Vocabularies
28. Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm — various, 2024
https://scholar.google.com/scholar?q=Bridging+Developer+Instructions+and+Code+Completion+Through+Instruction-Aware+Fill-in-the-Middle+Paradigm
29. AI Post Transformers: Fast Inference from Transformers via Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Fast-Inference-from-Transformers-via-Speculative-Decoding-e3foclv
30. AI Post Transformers: Multiagent Debate Improves Language Model Reasoning — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Multiagent-Debate-Improves-Language-Model-Reasoning-e36kfd4
Interactive Visualization: Tokenization Bias: The Hidden Flaw Breaking Language Models