Data Science at Home

More powerful deep learning with transformers (Ep. 84)

Listen Later

Some of the most powerful NLP models like BERT and GPT-2 have one thing in common: they all use the transformer architecture.

Such architecture is built on top of another important concept already known to the community: self-attention.
In this episode I explain what these mechanisms are, how they work and why they are so powerful.

Don't forget to subscribe to our Newsletter or join the discussion on our Discord server

 

References
  • Attention is all you need 
https://arxiv.org/abs/1706.03762
  • The illustrated transformer 
    • https://jalammar.github.io/illustrated-transformer
  • Self-attention for generative models 
    • http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf
    ...more
    View all episodesView all episodes
    Download on the App Store
    Data Science at HomeBy Francesco Gadaleta
    • 4.2
    • 4.2
    • 4.2
    • 4.2
    • 4.2

    4.2

    72 ratings

    More shows like Data Science at Home

    View all
    On Point with Meghna Chakrabarti by WBUR

    On Point with Meghna Chakrabarti

    4,022 Listeners

    Making Sense with Sam Harris by Sam Harris

    Making Sense with Sam Harris

    26,380 Listeners

    Nature Podcast by Springer Nature Limited

    Nature Podcast

    756 Listeners

    Software Engineering Daily by Software Engineering Daily

    Software Engineering Daily

    626 Listeners

    Science Vs by Spotify Studios

    Science Vs

    12,130 Listeners

    Science Friday by Science Friday and WNYC Studios

    Science Friday

    6,467 Listeners

    Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

    Super Data Science: ML & AI Podcast with Jon Krohn

    306 Listeners

    The Daily by The New York Times

    The Daily

    113,121 Listeners

    Up First from NPR by NPR

    Up First from NPR

    56,944 Listeners

    The Atlantic Interview by The Atlantic

    The Atlantic Interview

    14 Listeners

    Modern Wisdom by Chris Williamson

    Modern Wisdom

    4,025 Listeners

    The Peter Attia Drive by Peter Attia, MD

    The Peter Attia Drive

    8,043 Listeners

    Practical AI by Practical AI LLC

    Practical AI

    212 Listeners

    Consider This from NPR by NPR

    Consider This from NPR

    6,462 Listeners

    The Ezra Klein Show by New York Times Opinion

    The Ezra Klein Show

    16,525 Listeners