Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team’s 2024 NeurIPS paper on how distinguishing between useful and “noisy” tokens in pretraining can improve token efficiency and model performance.<a href="https://www.microsoft.com/en-us/research/publication/not-all-tokens-are-what-you-need-for-pretraining/" style="color:rgb(70,120,134);background-color:rgb(255,255,255);">Read the paper</a><a href="https://github.com/microsoft/rho" style="color:rgb(70,120,134);background-color:rgb(255,255,255);">Get the code</a>

Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team’s 2024 NeurIPS paper on how distinguishing between useful and “noisy” tokens in pretraining can improve token efficiency and model performance. Read the paper Get the code

Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team’s 2024 NeurIPS paper on how distinguishing between useful and “noisy” tokens in pretraining can improve token efficiency and model performance.<a href="https://www.microsoft.com/en-us/research/publication/not-all-tokens-are-what-you-need-for-pretraining/" rel="noopener noreferrer">Read the paper</a><a href="https://github.com/microsoft/rho" rel="noopener noreferrer">Get the code</a>

Abstracts: NeurIPS 2024 with Weizhu Chen

An ongoing series of conversations bringing you right up to the cutting edge of Microsoft Research.

Share Abstracts: NeurIPS 2024 with Weizhu Chen

Sign up to save your podcasts

Abstracts: NeurIPS 2024 with Weizhu Chen

Abstracts: NeurIPS 2024 with Weizhu Chen

More shows like Microsoft Research Podcast

The Daily

Dwarkesh Podcast

Hard Fork

No Priors: Artificial Intelligence | Technology | Startups