Sign up to save your podcasts
Or
Share Self-Improving Pretraining With Post-Trained Models
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Self-Improving Pretraining With Post-Trained Models
This episode explores the idea of “self-improving pretraining,” where already post-trained models are used to shape the pretraining of new models rather than waiting to add safety, reasoning, and factuality later. It explains how the approach rewrites training continuations, uses stronger models as judges, and compares original corpus text, teacher-generated suffixes, and learner rollouts to push model preferences upstream into training. The discussion also situates the paper against earlier work like Constitutional AI, STaR, and Quiet-STaR, while debating whether this is a genuine shift in training philosophy or mainly a more aggressive form of distilling a stronger model’s preferences. Listeners would find it interesting because it gets at a central question in modern AI: whether better behavior can be built into a model’s foundations instead of patched on after the fact.
Sources:
1. Self-Improving Pretraining: using post-trained models to pretrain better models — Ellen Xiaoqing Tan, Jack Lanchantin, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, Ping Yu, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Xian Li, Olga Golovneva, 2026
http://arxiv.org/abs/2601.21343
2. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, et al., 2022
https://scholar.google.com/scholar?q=Constitutional+AI%3A+Harmlessness+from+AI+Feedback
3. STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, and Percy Liang, 2022
https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner+Bootstrapping+Reasoning+With+Reasoning
4. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking — Eric Zelikman, Yuhuai Wu, Noah D. Goodman, and Jakob Foerster, 2024
https://scholar.google.com/scholar?q=Quiet-STaR%3A+Language+Models+Can+Teach+Themselves+to+Think+Before+Speaking
5. Self-Rewarding Language Models — Natasha Shumailov, John Adler, Ming-Wei Chang, Sharan Narang, and Yi Tay, 2024
https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models
6. Confronting Reward Model Overoptimization with Constrained RLHF — Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla, 2024
https://scholar.google.com/scholar?q=Confronting+Reward+Model+Overoptimization+with+Constrained+RLHF
7. On the Diversity of Synthetic Data and its Impact on Training Large Language Models — Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, Marah I. Abdin, 2024
https://scholar.google.com/scholar?q=On+the+Diversity+of+Synthetic+Data+and+its+Impact+on+Training+Large+Language+Models
8. In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners — Jaehoon Kim, Kwangwook Seo, Dongha Lee, 2025
https://scholar.google.com/scholar?q=In+Their+Own+Words%3A+Reasoning+Traces+Tailored+for+Small+Models+Make+Them+Better+Reasoners
9. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks — William F. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Barbieri, Timon Willi, Akhil Mathur, Ilias Leontiadis, 2026
https://scholar.google.com/scholar?q=Rethinking+Rubric+Generation+for+Improving+LLM+Judge+and+Reward+Modeling+for+Open-ended+Tasks
10. Ask a Strong LLM Judge when Your Reward Model is Uncertain — Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, Hyokun Yun, Tuo Zhao, 2025
https://scholar.google.com/scholar?q=Ask+a+Strong+LLM+Judge+when+Your+Reward+Model+is+Uncertain
11. FOL-Traces: Verified First-Order Logic Reasoning Traces at Scale — Isabelle Lee, Sarah Liaw, Dani Yogatama, 2025
https://scholar.google.com/scholar?q=FOL-Traces%3A+Verified+First-Order+Logic+Reasoning+Traces+at+Scale
12. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3
13. AI Post Transformers: Distilling Multi-Agent Reasoning into a Single LLM — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-distilling-multi-agent-reasoning-into-a-143263.mp3
14. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp3
15. AI Post Transformers: MASA: Meta-Awareness via Self-Alignment Reinforcement Learning — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/masa-meta-awareness-via-self-alignment-reinforcement-learning/
16. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp3
Interactive Visualization: Self-Improving Pretraining With Post-Trained Models
View all episodes
By mcgrofThis episode explores the idea of “self-improving pretraining,” where already post-trained models are used to shape the pretraining of new models rather than waiting to add safety, reasoning, and factuality later. It explains how the approach rewrites training continuations, uses stronger models as judges, and compares original corpus text, teacher-generated suffixes, and learner rollouts to push model preferences upstream into training. The discussion also situates the paper against earlier work like Constitutional AI, STaR, and Quiet-STaR, while debating whether this is a genuine shift in training philosophy or mainly a more aggressive form of distilling a stronger model’s preferences. Listeners would find it interesting because it gets at a central question in modern AI: whether better behavior can be built into a model’s foundations instead of patched on after the fact.
Sources:
1. Self-Improving Pretraining: using post-trained models to pretrain better models — Ellen Xiaoqing Tan, Jack Lanchantin, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, Ping Yu, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Xian Li, Olga Golovneva, 2026
http://arxiv.org/abs/2601.21343
2. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, et al., 2022
https://scholar.google.com/scholar?q=Constitutional+AI%3A+Harmlessness+from+AI+Feedback
3. STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, and Percy Liang, 2022
https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner+Bootstrapping+Reasoning+With+Reasoning
4. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking — Eric Zelikman, Yuhuai Wu, Noah D. Goodman, and Jakob Foerster, 2024
https://scholar.google.com/scholar?q=Quiet-STaR%3A+Language+Models+Can+Teach+Themselves+to+Think+Before+Speaking
5. Self-Rewarding Language Models — Natasha Shumailov, John Adler, Ming-Wei Chang, Sharan Narang, and Yi Tay, 2024
https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models
6. Confronting Reward Model Overoptimization with Constrained RLHF — Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla, 2024
https://scholar.google.com/scholar?q=Confronting+Reward+Model+Overoptimization+with+Constrained+RLHF
7. On the Diversity of Synthetic Data and its Impact on Training Large Language Models — Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, Marah I. Abdin, 2024
https://scholar.google.com/scholar?q=On+the+Diversity+of+Synthetic+Data+and+its+Impact+on+Training+Large+Language+Models
8. In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners — Jaehoon Kim, Kwangwook Seo, Dongha Lee, 2025
https://scholar.google.com/scholar?q=In+Their+Own+Words%3A+Reasoning+Traces+Tailored+for+Small+Models+Make+Them+Better+Reasoners
9. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks — William F. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Barbieri, Timon Willi, Akhil Mathur, Ilias Leontiadis, 2026
https://scholar.google.com/scholar?q=Rethinking+Rubric+Generation+for+Improving+LLM+Judge+and+Reward+Modeling+for+Open-ended+Tasks
10. Ask a Strong LLM Judge when Your Reward Model is Uncertain — Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, Hyokun Yun, Tuo Zhao, 2025
https://scholar.google.com/scholar?q=Ask+a+Strong+LLM+Judge+when+Your+Reward+Model+is+Uncertain
11. FOL-Traces: Verified First-Order Logic Reasoning Traces at Scale — Isabelle Lee, Sarah Liaw, Dani Yogatama, 2025
https://scholar.google.com/scholar?q=FOL-Traces%3A+Verified+First-Order+Logic+Reasoning+Traces+at+Scale
12. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3
13. AI Post Transformers: Distilling Multi-Agent Reasoning into a Single LLM — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-distilling-multi-agent-reasoning-into-a-143263.mp3
14. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp3
15. AI Post Transformers: MASA: Meta-Awareness via Self-Alignment Reinforcement Learning — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/masa-meta-awareness-via-self-alignment-reinforcement-learning/
16. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp3
Interactive Visualization: Self-Improving Pretraining With Post-Trained Models