Sign up to save your podcasts
Or
Share Scaling Test-Time Compute for Reasoning Models
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Scaling Test-Time Compute for Reasoning Models
This episode explores how test-time compute should be allocated in large language models, using a recent study that compares parallel sampling, majority voting, shortest- and longest-trace selection, and beam-style search under a common evaluation setup. It explains the paper’s central argument that there is no single best inference-time strategy: some model families behave like short-horizon reasoners that benefit from several concise attempts, while others act like long-horizon reasoners that can make productive use of longer sequential reasoning. The discussion also examines how the authors benchmark eight open models across demanding datasets such as AIME and GPQA Diamond, and why harder problems reveal whether extra trace length produces real progress or just more verbose failure. Listeners would find it interesting because it turns a vague idea of “letting models think longer” into a concrete engineering question about how reasoning systems should spend their runtime budget.
Sources:
1. The Art of Scaling Test-Time Compute for Large Language Models — Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty, 2025
http://arxiv.org/abs/2512.02008
2. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024
https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters
3. Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning — Michael Hassid, Gabriel Synnaeve, Yossi Adi, Roy Schwartz, 2025
https://scholar.google.com/scholar?q=Don%27t+Overthink+it.+Preferring+Shorter+Thinking+Chains+for+Improved+LLM+Reasoning
4. Inverse Scaling in Test-Time Compute — Aryo Pradipta Gema, Alexander Hagele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, Ethan Perez, 2025
https://scholar.google.com/scholar?q=Inverse+Scaling+in+Test-Time+Compute
5. Qwen3 Technical Report — An Yang and the Qwen team, 2025
https://scholar.google.com/scholar?q=Qwen3+Technical+Report
6. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang et al., 2023
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao et al., 2023
https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models
8. Graph of Thoughts: Solving Elaborate Problems with Large Language Models — Maciej Besta et al., 2024
https://scholar.google.com/scholar?q=Graph+of+Thoughts%3A+Solving+Elaborate+Problems+with+Large+Language+Models
9. short-m@k — Ranit Hassid et al., 2025
https://scholar.google.com/scholar?q=short-m%40k
10. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — approx. recent process-verifier work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Rewarding+Progress%3A+Scaling+Automated+Process+Verifiers+for+LLM+Reasoning
11. Improving LLM Reasoning Through Scaling Inference Computation With Collaborative Verification — approx. recent collaborative-verification work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Improving+LLM+Reasoning+Through+Scaling+Inference+Computation+With+Collaborative+Verification
12. Graph of Verification: Structured Verification of LLM Reasoning With Directed Acyclic Graphs — approx. recent verification-structure work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Graph+of+Verification%3A+Structured+Verification+of+LLM+Reasoning+With+Directed+Acyclic+Graphs
13. Dynamic Parallel Tree Search for Efficient LLM Reasoning — approx. recent tree-search work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Dynamic+Parallel+Tree+Search+for+Efficient+LLM+Reasoning
14. Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfalls — approx. recent tree-search analysis work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Don%27t+Get+Lost+in+the+Trees%3A+Streamlining+LLM+Reasoning+by+Overcoming+Tree+Search+Exploration+Pitfalls
15. REST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search — approx. recent MCTS/process-reward work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=REST-MCTS%2A%3A+LLM+Self-Training+via+Process+Reward+Guided+Tree+Search
16. Large Language Models Cannot Self-Correct Reasoning Yet — approx. recent self-correction evaluation work, exact authors not confirmed from snippet, 2024
https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet
17. AI Post Transformers: Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/test-time-scaling/
18. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp3
19. AI Post Transformers: Agentic Aggregation for Long-Horizon AI Tasks — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-agentic-aggregation-for-long-horizon-ai-4c1a71.mp3
20. AI Post Transformers: TUMIX Multi-Agent Test-Time Scaling with Tools — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-tumix-multi-agent-test-time-scaling-with-40671c.mp3
21. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3
22. AI Post Transformers: The Art of Scaling Reinforcement Learning Compute for LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/the-art-of-scaling-reinforcement-learning-compute-for-llms/
View all episodes
By mcgrofThis episode explores how test-time compute should be allocated in large language models, using a recent study that compares parallel sampling, majority voting, shortest- and longest-trace selection, and beam-style search under a common evaluation setup. It explains the paper’s central argument that there is no single best inference-time strategy: some model families behave like short-horizon reasoners that benefit from several concise attempts, while others act like long-horizon reasoners that can make productive use of longer sequential reasoning. The discussion also examines how the authors benchmark eight open models across demanding datasets such as AIME and GPQA Diamond, and why harder problems reveal whether extra trace length produces real progress or just more verbose failure. Listeners would find it interesting because it turns a vague idea of “letting models think longer” into a concrete engineering question about how reasoning systems should spend their runtime budget.
Sources:
1. The Art of Scaling Test-Time Compute for Large Language Models — Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty, 2025
http://arxiv.org/abs/2512.02008
2. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024
https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters
3. Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning — Michael Hassid, Gabriel Synnaeve, Yossi Adi, Roy Schwartz, 2025
https://scholar.google.com/scholar?q=Don%27t+Overthink+it.+Preferring+Shorter+Thinking+Chains+for+Improved+LLM+Reasoning
4. Inverse Scaling in Test-Time Compute — Aryo Pradipta Gema, Alexander Hagele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, Ethan Perez, 2025
https://scholar.google.com/scholar?q=Inverse+Scaling+in+Test-Time+Compute
5. Qwen3 Technical Report — An Yang and the Qwen team, 2025
https://scholar.google.com/scholar?q=Qwen3+Technical+Report
6. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang et al., 2023
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao et al., 2023
https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models
8. Graph of Thoughts: Solving Elaborate Problems with Large Language Models — Maciej Besta et al., 2024
https://scholar.google.com/scholar?q=Graph+of+Thoughts%3A+Solving+Elaborate+Problems+with+Large+Language+Models
9. short-m@k — Ranit Hassid et al., 2025
https://scholar.google.com/scholar?q=short-m%40k
10. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — approx. recent process-verifier work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Rewarding+Progress%3A+Scaling+Automated+Process+Verifiers+for+LLM+Reasoning
11. Improving LLM Reasoning Through Scaling Inference Computation With Collaborative Verification — approx. recent collaborative-verification work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Improving+LLM+Reasoning+Through+Scaling+Inference+Computation+With+Collaborative+Verification
12. Graph of Verification: Structured Verification of LLM Reasoning With Directed Acyclic Graphs — approx. recent verification-structure work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Graph+of+Verification%3A+Structured+Verification+of+LLM+Reasoning+With+Directed+Acyclic+Graphs
13. Dynamic Parallel Tree Search for Efficient LLM Reasoning — approx. recent tree-search work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Dynamic+Parallel+Tree+Search+for+Efficient+LLM+Reasoning
14. Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfalls — approx. recent tree-search analysis work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=Don%27t+Get+Lost+in+the+Trees%3A+Streamlining+LLM+Reasoning+by+Overcoming+Tree+Search+Exploration+Pitfalls
15. REST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search — approx. recent MCTS/process-reward work, exact authors not confirmed from snippet, 2025
https://scholar.google.com/scholar?q=REST-MCTS%2A%3A+LLM+Self-Training+via+Process+Reward+Guided+Tree+Search
16. Large Language Models Cannot Self-Correct Reasoning Yet — approx. recent self-correction evaluation work, exact authors not confirmed from snippet, 2024
https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet
17. AI Post Transformers: Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/test-time-scaling/
18. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp3
19. AI Post Transformers: Agentic Aggregation for Long-Horizon AI Tasks — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-agentic-aggregation-for-long-horizon-ai-4c1a71.mp3
20. AI Post Transformers: TUMIX Multi-Agent Test-Time Scaling with Tools — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-tumix-multi-agent-test-time-scaling-with-40671c.mp3
21. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3
22. AI Post Transformers: The Art of Scaling Reinforcement Learning Compute for LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/the-art-of-scaling-reinforcement-learning-compute-for-llms/