Sign up to save your podcasts
Or
Share When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
Source: LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
Paper was published on May 27, 2026
This episode was AI-generated on May 28, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.
Unplug a top AI search agent's internet connection and it still answers 44% of questions on a benchmark designed to require browsing. That uncomfortable result is the opening move in a paper that argues current search agents aren't really searching — they're verifying what they already know — and that the field's leaderboards have been measuring the wrong capability.
Key Takeaways
Pulling search tools off frontier agents reveals they already answer a large fraction of 'requires browsing' questions from memory alone.
The distinction between leaked benchmark questions and broad world knowledge covering the answer territory — and why decontamination can't fix the latter.
Removing the supporting documents from the index while leaving hard negatives in place causes performance to collapse below the no-tools floor across every model tested.
Why the failure pattern looks like a confident student rubber-stamping a textbook rather than reading it — and what that means for robustness.
Measuring where query entities come from and how often agents actually use retrieved evidence, leading to the paper's named failure mode: memory-backed verification rather than evidence-driven discovery.
The recent-plus-obscure construction across six structured sources, designed to push answers outside any model's parametric memory.
Why human solve rates and timing on both benchmarks are nearly identical, anchoring the claim that agent collapse on LiveBrowseComp reflects suppressed IKD rather than harder questions.
Where the evidence-blocking setup is adversarial, where the IKD inference is underdetermined, and what survives the strongest version of the skeptic's case.
Why these agents are most reliable in the regime where you don't need them and least reliable — silently — in the regime where search is the whole point.
Recommended Reading
View all episodes
By paperdive.aiWhen Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
Source: LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
Paper was published on May 27, 2026
This episode was AI-generated on May 28, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.
Unplug a top AI search agent's internet connection and it still answers 44% of questions on a benchmark designed to require browsing. That uncomfortable result is the opening move in a paper that argues current search agents aren't really searching — they're verifying what they already know — and that the field's leaderboards have been measuring the wrong capability.
Key Takeaways
Pulling search tools off frontier agents reveals they already answer a large fraction of 'requires browsing' questions from memory alone.
The distinction between leaked benchmark questions and broad world knowledge covering the answer territory — and why decontamination can't fix the latter.
Removing the supporting documents from the index while leaving hard negatives in place causes performance to collapse below the no-tools floor across every model tested.
Why the failure pattern looks like a confident student rubber-stamping a textbook rather than reading it — and what that means for robustness.
Measuring where query entities come from and how often agents actually use retrieved evidence, leading to the paper's named failure mode: memory-backed verification rather than evidence-driven discovery.
The recent-plus-obscure construction across six structured sources, designed to push answers outside any model's parametric memory.
Why human solve rates and timing on both benchmarks are nearly identical, anchoring the claim that agent collapse on LiveBrowseComp reflects suppressed IKD rather than harder questions.
Where the evidence-blocking setup is adversarial, where the IKD inference is underdetermined, and what survives the strongest version of the skeptic's case.
Why these agents are most reliable in the regime where you don't need them and least reliable — silently — in the regime where search is the whole point.
Recommended Reading