Data Science at Home

More powerful deep learning with transformers (Ep. 84)

Listen Later

Some of the most powerful NLP models like BERT and GPT-2 have one thing in common: they all use the transformer architecture.

Such architecture is built on top of another important concept already known to the community: self-attention.
In this episode I explain what these mechanisms are, how they work and why they are so powerful.

Don't forget to subscribe to our Newsletter or join the discussion on our Discord server

 

References
  • Attention is all you need 
https://arxiv.org/abs/1706.03762
  • The illustrated transformer 
    • https://jalammar.github.io/illustrated-transformer
  • Self-attention for generative models 
    • http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf
    ...more
    View all episodesView all episodes
    Download on the App Store
    Data Science at HomeBy Francesco Gadaleta
    • 4.2
    • 4.2
    • 4.2
    • 4.2
    • 4.2

    4.2

    72 ratings

    More shows like Data Science at Home

    View all
    Radiolab by WNYC Studios

    Radiolab

    43,851 Listeners

    TED Talks Daily by TED

    TED Talks Daily

    11,267 Listeners

    Learning English Conversations by BBC Radio

    Learning English Conversations

    1,063 Listeners

    Stuff You Should Know by iHeartPodcasts

    Stuff You Should Know

    77,233 Listeners

    Data Skeptic by Kyle Polich

    Data Skeptic

    474 Listeners

    Talk Python To Me by Michael Kennedy

    Talk Python To Me

    584 Listeners

    AWS Podcast by Amazon Web Services

    AWS Podcast

    200 Listeners

    Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

    Super Data Science: ML & AI Podcast with Jon Krohn

    295 Listeners

    Learning English from the News by BBC Radio

    Learning English from the News

    249 Listeners

    DataFramed by DataCamp

    DataFramed

    267 Listeners

    Practical AI by Practical AI LLC

    Practical AI

    196 Listeners

    The Intelligence from The Economist by The Economist

    The Intelligence from The Economist

    2,537 Listeners

    Raport o stanie świata Dariusza Rosiaka by Dariusz Rosiak

    Raport o stanie świata Dariusza Rosiaka

    42 Listeners

    The Ancients by History Hit

    The Ancients

    2,820 Listeners

    Hard Fork by The New York Times

    Hard Fork

    5,367 Listeners