Linear Digressions

Pre-training language models for natural language processing problems


Listen Later

When you build a model for natural language processing (NLP), such as a recurrent neural network, it helps a ton if you’re not starting from zero. In other words, if you can draw upon other datasets for building your understanding of word meanings, and then use your training dataset just for subject-specific refinements, you’ll get farther than just using your training dataset for everything. This idea of starting with some pre-trained resources has an analogue in computer vision, where initializations from ImageNet used for the first few layers of a CNN have become the new standard. There’s a similar progression under way in NLP, where simple(r) embeddings like word2vec are giving way to more advanced pre-processing methods that aim to capture more sophisticated understanding of word meanings, contexts, language structure, and more.
Relevant links:
https://thegradient.pub/nlp-imagenet/
...more
View all episodesView all episodes
Download on the App Store

Linear DigressionsBy Ben Jaffe and Katie Malone

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

352 ratings


More shows like Linear Digressions

View all
Global News Podcast by BBC World Service

Global News Podcast

7,894 Listeners

Data Skeptic by Kyle Polich

Data Skeptic

480 Listeners

The Daily by The New York Times

The Daily

111,746 Listeners

Up First from NPR by NPR

Up First from NPR

56,180 Listeners

What's That Rash? by ABC listen

What's That Rash?

243 Listeners

The Ezra Klein Show by New York Times Opinion

The Ezra Klein Show

15,220 Listeners

Prof G Markets by Vox Media Podcast Network

Prof G Markets

1,048 Listeners