Machine Learning Street Talk (MLST)

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer


Listen Later

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten chat about Large-scale Transfer Learning in Natural Language Processing. The Text-to-Text Transfer Transformer (T5) model from Google AI does an exhaustive survey of what’s important for Transfer Learning in NLP and what’s not. In this conversation, we go through the key takeaways of the paper, text-to-text input/output format, architecture choice, dataset size and composition, fine-tuning strategy, and how to best use more computation.

Beginning with these topics, we diverge into exciting ideas such as embodied cognition, meta-learning, and the measure of intelligence. We are still beginning our podcast journey and really appreciate any feedback from our listeners. Is the chat too technical? Do you prefer group discussions, interviewing experts, or chats between the three of us? Thanks for watching and if you haven’t already, Please Subscribe!

Paper Links discussed in the chat:

Text-to-Text Transfer Transformer: https://arxiv.org/abs/1910.10683

Experience Grounds Language (relevant to divergent discussion about embodied cognition): https://arxiv.org/pdf/2004.10151.pdf

On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Train Large, Then Compress: https://arxiv.org/pdf/2002.11794.pdf

Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361.pdf

The Illustrated Transformer: http://jalammar.github.io/illustrated...

ELECTRA: https://arxiv.org/pdf/2003.10555.pdf

Transformer-XL: https://arxiv.org/pdf/1901.02860.pdf

Reformer: The Efficient Transformer: https://openreview.net/pdf?id=rkgNKkHtvB

The Evolved Transformer: https://arxiv.org/pdf/1901.11117.pdf

DistilBERT: https://arxiv.org/pdf/1910.01108.pdf

How to generate text (HIGHLY RECOMMEND): https://huggingface.co/blog/how-to-ge...

Tokenizers: https://blog.floydhub.com/tokenization-nlp/

...more
View all episodesView all episodes
Download on the App Store

Machine Learning Street Talk (MLST)By Machine Learning Street Talk (MLST)

  • 4.7
  • 4.7
  • 4.7
  • 4.7
  • 4.7

4.7

84 ratings


More shows like Machine Learning Street Talk (MLST)

View all
Data Skeptic by Kyle Polich

Data Skeptic

481 Listeners

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

441 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

298 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

323 Listeners

Machine Learning Guide by OCDevel

Machine Learning Guide

764 Listeners

Practical AI by Practical AI LLC

Practical AI

190 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

87 Listeners

Google DeepMind: The Podcast by Hannah Fry

Google DeepMind: The Podcast

199 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

371 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

122 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

199 Listeners

Unsupervised Learning by by Redpoint Ventures

Unsupervised Learning

39 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

76 Listeners

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

441 Listeners

Training Data by Sequoia Capital

Training Data

36 Listeners