Data Science at Home

More powerful deep learning with transformers (Ep. 84) (Rebroadcast)

Listen Later

Some of the most powerful NLP models like BERT and GPT-2 have one thing in common: they all use the transformer architecture.

Such architecture is built on top of another important concept already known to the community: self-attention.
In this episode I explain what these mechanisms are, how they work and why they are so powerful.

Don't forget to subscribe to our Newsletter or join the discussion on our Discord server

 

References
  • Attention is all you need 
https://arxiv.org/abs/1706.03762
  • The illustrated transformer 
    • https://jalammar.github.io/illustrated-transformer
  • Self-attention for generative models 
    • http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf
    ...more
    View all episodesView all episodes
    Download on the App Store
    Data Science at HomeBy Francesco Gadaleta
    • 4.2
    • 4.2
    • 4.2
    • 4.2
    • 4.2

    4.2

    72 ratings

    More shows like Data Science at Home

    View all
    More or Less by BBC Radio 4

    More or Less

    887 Listeners

    WSJ Tech News Briefing by The Wall Street Journal

    WSJ Tech News Briefing

    1,642 Listeners

    Software Engineering Daily by Software Engineering Daily

    Software Engineering Daily

    625 Listeners

    Talk Python To Me by Michael Kennedy

    Talk Python To Me

    585 Listeners

    BBC Inside Science by BBC Radio 4

    BBC Inside Science

    412 Listeners

    Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

    Super Data Science: ML & AI Podcast with Jon Krohn

    303 Listeners

    FT Tech Tonic by Financial Times

    FT Tech Tonic

    99 Listeners

    Worklife with Adam Grant by TED

    Worklife with Adam Grant

    9,167 Listeners

    Practical AI by Practical AI LLC

    Practical AI

    208 Listeners

    Last Week in AI by Skynet Today

    Last Week in AI

    306 Listeners

    Hard Fork by The New York Times

    Hard Fork

    5,507 Listeners

    This Day in AI Podcast by Michael Sharkey, Chris Sharkey

    This Day in AI Podcast

    227 Listeners

    The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

    The AI Daily Brief: Artificial Intelligence News and Analysis

    608 Listeners

    Unhedged by Financial Times & Pushkin Industries

    Unhedged

    181 Listeners

    The Last Invention by Longview

    The Last Invention

    1,088 Listeners