Sign up to save your podcasts
Or
Share Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
In this episode:
• Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper.
• The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information.
• Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where.
• Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling.
• Length Extrapolation and Conclusion: Analyzing PoPE's zero-shot length extrapolation capabilities compared to YaRN, followed by episode wrap-up.
• Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper.
• The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information.
• Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where.
• Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling.
• Length Extrapolation and Conclusion: Analyzing PoPE's zero-shot length extrapolation capabilities compared to YaRN, followed by episode wrap-up.
View all episodes
By Mechanical DirkIn this episode:
• Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper.
• The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information.
• Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where.
• Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling.
• Length Extrapolation and Conclusion: Analyzing PoPE's zero-shot length extrapolation capabilities compared to YaRN, followed by episode wrap-up.
• Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper.
• The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information.
• Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where.
• Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling.
• Length Extrapolation and Conclusion: Analyzing PoPE's zero-shot length extrapolation capabilities compared to YaRN, followed by episode wrap-up.