Sign up to save your podcasts
Or
Share Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
In this episode:
• To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate.
• The Inductive Bias Paradox: Linda explains the paper's first major observation: Positional Embeddings (PEs) are necessary scaffolding for fast training convergence but become a straitjacket for zero-shot generalization. They discuss the theoretical findings regarding 'attention positional bias' and why NoPE models struggle to learn initially.
• Why Scaling Frequency Breaks Meaning: The hosts dive into the technical critique of current context-extension methods like YaRN and RoPE-scaling. Linda details how compressing low frequencies to fit longer contexts distorts 'semantic heads'—the parts of the model that match content rather than position—causing failures in retrieval tasks.
• The DroPE Solution: Remove the Training Wheels: They discuss the proposed method: training with RoPE, then stripping it away and doing a quick recalibration. Norris warms up to the analogy of PEs acting as 'scaffolding' that should be removed once the building (the model) is self-supporting.
• Needles in Haystacks and Future Architectures: Reviewing the empirical results, including the massive improvements on Needle-in-a-Haystack benchmarks compared to YaRN. The episode concludes with a discussion on whether future foundation models will all be trained with this 'drop-and-recalibrate' paradigm.
• To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate.
• The Inductive Bias Paradox: Linda explains the paper's first major observation: Positional Embeddings (PEs) are necessary scaffolding for fast training convergence but become a straitjacket for zero-shot generalization. They discuss the theoretical findings regarding 'attention positional bias' and why NoPE models struggle to learn initially.
• Why Scaling Frequency Breaks Meaning: The hosts dive into the technical critique of current context-extension methods like YaRN and RoPE-scaling. Linda details how compressing low frequencies to fit longer contexts distorts 'semantic heads'—the parts of the model that match content rather than position—causing failures in retrieval tasks.
• The DroPE Solution: Remove the Training Wheels: They discuss the proposed method: training with RoPE, then stripping it away and doing a quick recalibration. Norris warms up to the analogy of PEs acting as 'scaffolding' that should be removed once the building (the model) is self-supporting.
• Needles in Haystacks and Future Architectures: Reviewing the empirical results, including the massive improvements on Needle-in-a-Haystack benchmarks compared to YaRN. The episode concludes with a discussion on whether future foundation models will all be trained with this 'drop-and-recalibrate' paradigm.
View all episodes
By Mechanical DirkIn this episode:
• To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate.
• The Inductive Bias Paradox: Linda explains the paper's first major observation: Positional Embeddings (PEs) are necessary scaffolding for fast training convergence but become a straitjacket for zero-shot generalization. They discuss the theoretical findings regarding 'attention positional bias' and why NoPE models struggle to learn initially.
• Why Scaling Frequency Breaks Meaning: The hosts dive into the technical critique of current context-extension methods like YaRN and RoPE-scaling. Linda details how compressing low frequencies to fit longer contexts distorts 'semantic heads'—the parts of the model that match content rather than position—causing failures in retrieval tasks.
• The DroPE Solution: Remove the Training Wheels: They discuss the proposed method: training with RoPE, then stripping it away and doing a quick recalibration. Norris warms up to the analogy of PEs acting as 'scaffolding' that should be removed once the building (the model) is self-supporting.
• Needles in Haystacks and Future Architectures: Reviewing the empirical results, including the massive improvements on Needle-in-a-Haystack benchmarks compared to YaRN. The episode concludes with a discussion on whether future foundation models will all be trained with this 'drop-and-recalibrate' paradigm.
• To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate.
• The Inductive Bias Paradox: Linda explains the paper's first major observation: Positional Embeddings (PEs) are necessary scaffolding for fast training convergence but become a straitjacket for zero-shot generalization. They discuss the theoretical findings regarding 'attention positional bias' and why NoPE models struggle to learn initially.
• Why Scaling Frequency Breaks Meaning: The hosts dive into the technical critique of current context-extension methods like YaRN and RoPE-scaling. Linda details how compressing low frequencies to fit longer contexts distorts 'semantic heads'—the parts of the model that match content rather than position—causing failures in retrieval tasks.
• The DroPE Solution: Remove the Training Wheels: They discuss the proposed method: training with RoPE, then stripping it away and doing a quick recalibration. Norris warms up to the analogy of PEs acting as 'scaffolding' that should be removed once the building (the model) is self-supporting.
• Needles in Haystacks and Future Architectures: Reviewing the empirical results, including the massive improvements on Needle-in-a-Haystack benchmarks compared to YaRN. The episode concludes with a discussion on whether future foundation models will all be trained with this 'drop-and-recalibrate' paradigm.