The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Dynamic Token Merging for Efficient Byte-level Language Models with Julie Kallini - #724


Listen Later

Today, we're joined by Julie Kallini, PhD student at Stanford University to discuss her recent papers, “MrT5: Dynamic Token Merging for Efficient Byte-level Language Models” and “Mission: Impossible Language Models.” For the MrT5 paper, we explore the importance and failings of tokenization in large language models—including inefficient compression rates for under-resourced languages—and dig into byte-level modeling as an alternative. We discuss the architecture of MrT5, its ability to learn language-specific compression rates, its performance on multilingual benchmarks and character-level manipulation tasks, and its performance and efficiency. For the “Mission: Impossible Language Models” paper, we review the core idea behind the research, the definition and creation of impossible languages, the creation of impossible language training datasets, and explore the bias of language model architectures towards natural language.


The complete show notes for this episode can be found at https://twimlai.com/go/724.

...more
View all episodesView all episodes
Download on the App Store

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)By Sam Charrington

  • 4.7
  • 4.7
  • 4.7
  • 4.7
  • 4.7

4.7

412 ratings


More shows like The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

View all
The AI in Business Podcast by Daniel Faggella

The AI in Business Podcast

158 Listeners

Data Skeptic by Kyle Polich

Data Skeptic

475 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

295 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

312 Listeners

AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion by AI & Data Today

AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion

149 Listeners

Practical AI by Practical AI LLC

Practical AI

196 Listeners

Last Week in AI by Skynet Today

Last Week in AI

271 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

92 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

315 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

106 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

178 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

70 Listeners

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

397 Listeners

AI + a16z by a16z

AI + a16z

26 Listeners

Training Data by Sequoia Capital

Training Data

31 Listeners