Sign up to save your podcasts
Or
Share When Combining Language Models Stops Helping
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
When Combining Language Models Stops Helping
This episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why the real gain over a strong single model only exists on queries where that model fails but another succeeds. The discussion walks through results from a 15-model routing setup and a 67-model frontier-model study, showing that even calibrated copula-based estimates systematically understate shared failure and that learned routers capture only a small fraction of the available oracle gain. A listener would find it interesting because it cuts through ensemble hype with a concrete argument about when multi-model orchestration is worth the added cost and complexity, plus a practical way to estimate headroom before building a router at all.
Sources:
1. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026
http://arxiv.org/abs/2606.27288
2. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023
https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion
3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024
https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities
4. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? — Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin, 2025
https://scholar.google.com/scholar?q=Rethinking+Mixture-of-Agents%3A+Is+Mixing+Different+Large+Language+Models+Beneficial%3F
5. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026
https://scholar.google.com/scholar?q=When+Does+Combining+Language+Models+Help%3F+A+Co-Failure+Ceiling+on+Routing%2C+Voting%2C+and+Mixture-of-Agents+Across+67+Frontier+Models
6. A Unified Approach to Routing and Cascading for LLMs — Jasper Dekoninck, Maximilian Baader, and Martin Vechev, 2024
https://scholar.google.com/scholar?q=A+Unified+Approach+to+Routing+and+Cascading+for+LLMs
7. When Does Confidence-Based Cascade Deferral Suffice? — Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar, 2023
https://scholar.google.com/scholar?q=When+Does+Confidence-Based+Cascade+Deferral+Suffice%3F
8. Correlated Errors in Large Language Models — Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg, 2025
https://scholar.google.com/scholar?q=Correlated+Errors+in+Large+Language+Models
9. Don't Always Pick the Highest-Performing Model: An Information-Theoretic View of LLM Ensemble Selection — Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu, 2026
https://scholar.google.com/scholar?q=Don%27t+Always+Pick+the+Highest-Performing+Model%3A+An+Information-Theoretic+View+of+LLM+Ensemble+Selection
10. PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier — Yuhua Jiang et al., 2025
https://arxiv.org/abs/2506.10406
11. S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ruotian Ma et al., 2025
https://arxiv.org/abs/2502.12853
12. Small Language Models Need Strong Verifiers to Self-Correct Reasoning — Yunxiang Zhang et al., 2024
https://arxiv.org/abs/2404.17140
13. CP-Router: An Uncertainty-Aware Router Between LLM and LRM — Jiayuan Su et al., 2025
https://arxiv.org/abs/2505.19970
14. Leveraging Uncertainty Estimation for Efficient LLM Routing — Tuo Zhang et al., 2025
https://arxiv.org/abs/2502.11021
15. Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization — Yu-Neng Chuang et al., 2025
https://arxiv.org/abs/2502.04428
16. Wisdom and Delusion of LLM Ensembles for Code Generation and Repair — Fernando Vallecillos Ruiz et al., 2025
https://arxiv.org/abs/2510.21513
17. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang et al., 2026
https://arxiv.org/abs/2602.03794
18. LLM Chemistry Estimation for Multi-LLM Recommendation — Huascar Sanchez and Briland Hitaj, 2025
https://arxiv.org/abs/2510.03930
19. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp3
20. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp3
View all episodes
By mcgrofThis episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why the real gain over a strong single model only exists on queries where that model fails but another succeeds. The discussion walks through results from a 15-model routing setup and a 67-model frontier-model study, showing that even calibrated copula-based estimates systematically understate shared failure and that learned routers capture only a small fraction of the available oracle gain. A listener would find it interesting because it cuts through ensemble hype with a concrete argument about when multi-model orchestration is worth the added cost and complexity, plus a practical way to estimate headroom before building a router at all.
Sources:
1. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026
http://arxiv.org/abs/2606.27288
2. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023
https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion
3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024
https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities
4. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? — Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin, 2025
https://scholar.google.com/scholar?q=Rethinking+Mixture-of-Agents%3A+Is+Mixing+Different+Large+Language+Models+Beneficial%3F
5. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026
https://scholar.google.com/scholar?q=When+Does+Combining+Language+Models+Help%3F+A+Co-Failure+Ceiling+on+Routing%2C+Voting%2C+and+Mixture-of-Agents+Across+67+Frontier+Models
6. A Unified Approach to Routing and Cascading for LLMs — Jasper Dekoninck, Maximilian Baader, and Martin Vechev, 2024
https://scholar.google.com/scholar?q=A+Unified+Approach+to+Routing+and+Cascading+for+LLMs
7. When Does Confidence-Based Cascade Deferral Suffice? — Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar, 2023
https://scholar.google.com/scholar?q=When+Does+Confidence-Based+Cascade+Deferral+Suffice%3F
8. Correlated Errors in Large Language Models — Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg, 2025
https://scholar.google.com/scholar?q=Correlated+Errors+in+Large+Language+Models
9. Don't Always Pick the Highest-Performing Model: An Information-Theoretic View of LLM Ensemble Selection — Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu, 2026
https://scholar.google.com/scholar?q=Don%27t+Always+Pick+the+Highest-Performing+Model%3A+An+Information-Theoretic+View+of+LLM+Ensemble+Selection
10. PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier — Yuhua Jiang et al., 2025
https://arxiv.org/abs/2506.10406
11. S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ruotian Ma et al., 2025
https://arxiv.org/abs/2502.12853
12. Small Language Models Need Strong Verifiers to Self-Correct Reasoning — Yunxiang Zhang et al., 2024
https://arxiv.org/abs/2404.17140
13. CP-Router: An Uncertainty-Aware Router Between LLM and LRM — Jiayuan Su et al., 2025
https://arxiv.org/abs/2505.19970
14. Leveraging Uncertainty Estimation for Efficient LLM Routing — Tuo Zhang et al., 2025
https://arxiv.org/abs/2502.11021
15. Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization — Yu-Neng Chuang et al., 2025
https://arxiv.org/abs/2502.04428
16. Wisdom and Delusion of LLM Ensembles for Code Generation and Repair — Fernando Vallecillos Ruiz et al., 2025
https://arxiv.org/abs/2510.21513
17. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang et al., 2026
https://arxiv.org/abs/2602.03794
18. LLM Chemistry Estimation for Multi-LLM Recommendation — Huascar Sanchez and Briland Hitaj, 2025
https://arxiv.org/abs/2510.03930
19. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp3
20. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp3