Sign up to save your podcasts
Or
Share [QA] Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
[QA] Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers
This paper explores how Rotary Positional Embeddings (RoPE) affect Transformer model dynamics, introducing phase shifts that influence embeddings, information retention, and attention through oscillatory behaviors and frequency components.
https://arxiv.org/abs//2410.18067
YouTube: https://www.youtube.com/@ArxivPapers
TikTok: https://www.tiktok.com/@arxiv_papers
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
View all episodes
By Igor Melnyk
5
33 ratings
This paper explores how Rotary Positional Embeddings (RoPE) affect Transformer model dynamics, introducing phase shifts that influence embeddings, information retention, and attention through oscillatory behaviors and frequency components.
https://arxiv.org/abs//2410.18067
YouTube: https://www.youtube.com/@ArxivPapers
TikTok: https://www.tiktok.com/@arxiv_papers
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
More shows like Arxiv Papers
View all
Exchanges
960 Listeners
Odd Lots
1,930 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
432 Listeners
The Daily
112,200 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,925 Listeners
Hard Fork
5,512 Listeners
UnHerd with Freddie Sayers
211 Listeners
Unsupervised Learning with Jacob Effron
49 Listeners
Latent Space: The AI Engineer Podcast
93 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
466 Listeners