Sign up to save your podcasts
Or
Share Scaling Laws for Multilingual Code Pretraining
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Scaling Laws for Multilingual Code Pretraining
This episode explores whether code models should allocate training data evenly across programming languages or tune the mix based on how each language scales. It walks through the paper’s core experiments on monolingual scaling, bilingual transfer, translation-style code pairing, and multilingual token allocation, with close attention to languages like Python, JavaScript, Rust, and TypeScript. The discussion highlights the paper’s main argument that programming languages differ in scaling behavior and cross-lingual usefulness, which could make non-uniform token budgets more effective than treating all code equally. Listeners would find it interesting for its concrete take on how multilingual software ecosystems challenge simple scaling-law assumptions and for its careful scrutiny of whether the evidence really supports those stronger optimization claims.
Sources:
1. Scaling Laws for Multilingual Code Pretraining
https://arxiv.org/pdf/2512.13472
2. Unsupervised Translation of Programming Languages — Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample, 2020
https://scholar.google.com/scholar?q=Unsupervised+Translation+of+Programming+Languages
3. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence — Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, Chandan K. Reddy, 2022
https://scholar.google.com/scholar?q=XLCoST%3A+A+Benchmark+Dataset+for+Cross-lingual+Code+Intelligence
4. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, Abhinav Jangda, 2022
https://scholar.google.com/scholar?q=MultiPL-E%3A+A+Scalable+and+Extensible+Approach+to+Benchmarking+Neural+Code+Generation
5. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, Jie Tang, 2023
https://scholar.google.com/scholar?q=CodeGeeX%3A+A+Pre-Trained+Model+for+Code+Generation+with+Multilingual+Benchmarking+on+HumanEval-X
6. Scaling Laws for Code: A More Data-Hungry Regime — Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che, 2025
https://scholar.google.com/scholar?q=Scaling+Laws+for+Code%3A+A+More+Data-Hungry+Regime
7. Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study — Razan Baltaji, Saurabh Pujar, Louis Mandel, Martin Hirzel, Luca Buratti, Lav Varshney, 2025
https://scholar.google.com/scholar?q=Cross-lingual+Transfer+in+Programming+Languages%3A+An+Extensive+Empirical+Study
8. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution — Ruiyang Xu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun, 2024
https://scholar.google.com/scholar?q=CRUXEval-X%3A+A+Benchmark+for+Multilingual+Code+Reasoning%2C+Understanding+and+Execution
9. RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation — Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, Zibin Zheng, 2024
https://scholar.google.com/scholar?q=RepoTransBench%3A+A+Real-World+Benchmark+for+Repository-Level+Code+Translation
10. Towards Multi-Language Repository-Level Code Generation: From-Scratch to Guided Tasks — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Towards+Multi-Language+Repository-Level+Code+Generation%3A+From-Scratch+to+Guided+Tasks
11. Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Do+Not+Treat+Code+as+Natural+Language%3A+Implications+for+Repository-Level+Code+Generation+and+Beyond
12. Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Code-Switching+In-Context+Learning+for+Cross-Lingual+Transfer+of+Large+Language+Models
13. Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Bridging+the+Language+Gap%3A+Enhancing+Multilingual+Prompt-Based+Code+Generation+in+LLMs+via+Zero-Shot+Cross-Lingual+Transfer
14. Advancing Code Translation With Context-Aware Pre-Training in Data-Scarce Environments — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Advancing+Code+Translation+With+Context-Aware+Pre-Training+in+Data-Scarce+Environments
15. Semi-Supervised Code Translation Overcoming the Scarcity of Parallel Code Data — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Semi-Supervised+Code+Translation+Overcoming+the+Scarcity+of+Parallel+Code+Data
16. Scaling Laws for Predicting Downstream Performance in LLMs — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Scaling+Laws+for+Predicting+Downstream+Performance+in+LLMs
17. AI Post Transformers: CODEGEN: Open Language Model for Code Synthesis — Hal Turing & Dr. Ada Shannon, Fri,
https://podcast.do-not-panic.com/episodes/codegen-open-language-model-for-code-synthesis/
18. AI Post Transformers: Llama 3: Architecture, Capabilities, and Safety — Hal Turing & Dr. Ada Shannon, Sun,
https://podcast.do-not-panic.com/episodes/llama-3-architecture-capabilities-and-safety/
19. AI Post Transformers: Program Synthesis with Large Language Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-program-synthesis-with-large-language-mo-b962ec.mp3
20. AI Post Transformers: MTEB & MMTEB: The Massive Text Embedding Benchmark — Hal Turing & Dr. Ada Shannon, Fri,
https://podcast.do-not-panic.com/episodes/mteb-mmteb-the-massive-text-embedding-benchmark/
Interactive Visualization: Scaling Laws for Multilingual Code Pretraining
View all episodes
By mcgrofThis episode explores whether code models should allocate training data evenly across programming languages or tune the mix based on how each language scales. It walks through the paper’s core experiments on monolingual scaling, bilingual transfer, translation-style code pairing, and multilingual token allocation, with close attention to languages like Python, JavaScript, Rust, and TypeScript. The discussion highlights the paper’s main argument that programming languages differ in scaling behavior and cross-lingual usefulness, which could make non-uniform token budgets more effective than treating all code equally. Listeners would find it interesting for its concrete take on how multilingual software ecosystems challenge simple scaling-law assumptions and for its careful scrutiny of whether the evidence really supports those stronger optimization claims.
Sources:
1. Scaling Laws for Multilingual Code Pretraining
https://arxiv.org/pdf/2512.13472
2. Unsupervised Translation of Programming Languages — Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample, 2020
https://scholar.google.com/scholar?q=Unsupervised+Translation+of+Programming+Languages
3. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence — Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, Chandan K. Reddy, 2022
https://scholar.google.com/scholar?q=XLCoST%3A+A+Benchmark+Dataset+for+Cross-lingual+Code+Intelligence
4. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, Abhinav Jangda, 2022
https://scholar.google.com/scholar?q=MultiPL-E%3A+A+Scalable+and+Extensible+Approach+to+Benchmarking+Neural+Code+Generation
5. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, Jie Tang, 2023
https://scholar.google.com/scholar?q=CodeGeeX%3A+A+Pre-Trained+Model+for+Code+Generation+with+Multilingual+Benchmarking+on+HumanEval-X
6. Scaling Laws for Code: A More Data-Hungry Regime — Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che, 2025
https://scholar.google.com/scholar?q=Scaling+Laws+for+Code%3A+A+More+Data-Hungry+Regime
7. Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study — Razan Baltaji, Saurabh Pujar, Louis Mandel, Martin Hirzel, Luca Buratti, Lav Varshney, 2025
https://scholar.google.com/scholar?q=Cross-lingual+Transfer+in+Programming+Languages%3A+An+Extensive+Empirical+Study
8. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution — Ruiyang Xu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun, 2024
https://scholar.google.com/scholar?q=CRUXEval-X%3A+A+Benchmark+for+Multilingual+Code+Reasoning%2C+Understanding+and+Execution
9. RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation — Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, Zibin Zheng, 2024
https://scholar.google.com/scholar?q=RepoTransBench%3A+A+Real-World+Benchmark+for+Repository-Level+Code+Translation
10. Towards Multi-Language Repository-Level Code Generation: From-Scratch to Guided Tasks — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Towards+Multi-Language+Repository-Level+Code+Generation%3A+From-Scratch+to+Guided+Tasks
11. Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Do+Not+Treat+Code+as+Natural+Language%3A+Implications+for+Repository-Level+Code+Generation+and+Beyond
12. Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Code-Switching+In-Context+Learning+for+Cross-Lingual+Transfer+of+Large+Language+Models
13. Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Bridging+the+Language+Gap%3A+Enhancing+Multilingual+Prompt-Based+Code+Generation+in+LLMs+via+Zero-Shot+Cross-Lingual+Transfer
14. Advancing Code Translation With Context-Aware Pre-Training in Data-Scarce Environments — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Advancing+Code+Translation+With+Context-Aware+Pre-Training+in+Data-Scarce+Environments
15. Semi-Supervised Code Translation Overcoming the Scarcity of Parallel Code Data — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Semi-Supervised+Code+Translation+Overcoming+the+Scarcity+of+Parallel+Code+Data
16. Scaling Laws for Predicting Downstream Performance in LLMs — authors unclear from snippet, unknown
https://scholar.google.com/scholar?q=Scaling+Laws+for+Predicting+Downstream+Performance+in+LLMs
17. AI Post Transformers: CODEGEN: Open Language Model for Code Synthesis — Hal Turing & Dr. Ada Shannon, Fri,
https://podcast.do-not-panic.com/episodes/codegen-open-language-model-for-code-synthesis/
18. AI Post Transformers: Llama 3: Architecture, Capabilities, and Safety — Hal Turing & Dr. Ada Shannon, Sun,
https://podcast.do-not-panic.com/episodes/llama-3-architecture-capabilities-and-safety/
19. AI Post Transformers: Program Synthesis with Large Language Models — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-program-synthesis-with-large-language-mo-b962ec.mp3
20. AI Post Transformers: MTEB & MMTEB: The Massive Text Embedding Benchmark — Hal Turing & Dr. Ada Shannon, Fri,
https://podcast.do-not-panic.com/episodes/mteb-mmteb-the-massive-text-embedding-benchmark/
Interactive Visualization: Scaling Laws for Multilingual Code Pretraining