Data Science in the Wild

Pinterest's fine-tuned XLM-RoBERTa relevance judge made stratified sampling affordable, cutting search A/B-test MDE from 1.3–1.5% to ≤0.25%.

Key takeaways:

• A small, fast judge. An XLM-RoBERTa-large cross-encoder fine-tuned on ~2.6M human-labeled query–Pin pairs labels 150,000 rows in 30 minutes on one A10G GPU, using text only (incl. BLIP image captions). Llama-3-8B was slightly better but ~6x costlier.
• Sampling is the payoff. Bigger samples, strata by query interest × popularity, Neyman allocation and a stratified estimator took MDE from 1.3–1.5% to ≤0.25%. At 2,000 queries, stratification alone cut the standard deviation 52%.
• Close to humans. 73.7% exact label match, 91.7% within one point; US sDCG@25 correlations τ 0.652, ρ 0.817. Humans still grade relevance-model tests.
• Our take. Novelty ~3/5: a standard model, smart measurement engineering. Watch for a judge that favors treatments built like itself.
• Note. τ and ρ compare how LLM- and human-based scores rank queries, not the order of Pins on a page. The 67% cut is the full design at 5,000 queries vs. 2,000 random ones. France/Germany (τ ≈ 0.47) is lower but still “strong” per the paper.

Chapters:

00:00 When an A/B test says nothing changed

01:07 Pinterest's LLM relevance paper

02:30 MDE: the rusty bathroom scale

04:01 Choosing the judge: Llama-3-8B vs. XLM-RoBERTa

06:26 A text-only judge on a visual platform

08:36 sDCG@25 and the long-tail problem

10:33 Stratified sampling and Neyman allocation

13:19 Results: MDE ≤0.25%, and agreement with humans

15:52 Our take: novelty and the language gap

18:12 Judge bias and the human safety net

19:34 Related work, recap, closing question

Sources:

• LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest (Wang et al.; RecSys ’25 EARL workshop, Sep 2025)
• Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects (related, Oct 2026)
• The Quest to Understand Metric Movements (Pinterest Engineering, 2025)

Related episode: E15: Netflix — Grading the Graders: Can an AI Judge Your Synopsis?

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Token budgets for RAG: which questions deserve extra retrieval? On FinanceBench, the cost of an extra call is easy to predict (R² 0.93) but its benefit isn't (AUROC 0.60). Gain-per-token escalation still uses 46.6% fewer tokens than top-5 everywhere.

Key takeaways:

• Escalate selectively. Every question gets a cheap top-1 answer. Only the calls with the best predicted gain per token get a top-5 retry.
• Cost is predictable; benefit isn't. Extra-call cost is predicted with R² 0.93. Spotting which questions will improve reaches only AUROC 0.60, where 0.5 is a coin flip.
• It still pays off. At a 10% budget, adjudicated quality is 0.034 higher than with gain-only ranking, using 46.6% fewer tokens than top-5 on every question.
• It's uplift targeting. Spend on “persuadable” queries that top-5 can fix. Skip the sure things and the lost causes.
• Our caveats. The test uses only 150 questions, and a weak predictor will starve some queries that needed more context.

Chapters:

00:00 Why every retrieved token has a price

01:26 The paper, and the cost of top-5 everywhere

04:01 FinanceBench setup and gain per token

06:10 Cost R² 0.93 vs. benefit AUROC 0.60

08:32 Results: +0.034 quality, 46.6% fewer tokens

09:43 What's new: uplift targeting

11:49 Related work on agentic RAG budgets

14:14 Limitations, and when triage makes sense

16:42 Recap: should we pay for uplift?

Sources:

• Token-Budgeted Escalation for Financial Document QA: Cost Is Predictable, Benefit Is the Bottleneck (Zhu, Yang, Ding, Qi; arXiv, Oct 2026)
• FinanceBench (benchmark)
• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (related)
• Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks (related)

Related episode: E32: Agentic RAG Evaluation — More Questions Beat More Repeats

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

With a fixed token budget for evaluating an agentic RAG system, should you ask more questions, sample more search trajectories, or re-grade answers? This paper finds that more questions win, cutting standard error by up to 33%.

Key takeaways:

• Split the noise. Generalizability theory separates evaluation variance into questions, search trajectories and repeated reads, tested on HotpotQA and MuSiQue.
• Go wide. At about 34 million tokens, broader question coverage lowered standard error by 33% compared with five reads, and by 12.6% compared with three trajectories.
• One read is usually enough. Reading each answer once cost only 0–9.9% in variance relative to the optimal allocation.
• Set the judge to temperature zero. It cut answer disagreement from 14.3% to 3.4%, with similar comparison precision.
• Plan ahead. Forecasts from a small pilot predicted the allocations within 3.5–4.0%, so you can budget before spending. Results are on academic benchmarks.

Chapters:

00:00 Wasted evaluation budgets

01:31 Why agentic RAG evals are noisy

03:05 Brute-force repeats

03:52 Generalizability theory

05:24 Judge disagreement and temperature

06:56 Spending 34 million tokens

08:30 Why one read is enough

10:01 Borrowing from A/B power analysis

11:32 Benchmark caveats

13:07 Planning your token spend

14:37 Recap

Sources:

• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (Jingjie Ning, Xueqi Li and Yibo Kong; arXiv, Oct 2026)

Related episode: E33: Token-Budgeted Escalation — Predicting Cost Is Easy, Predicting Benefit Is Hard

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

When platforms can't randomize users, they switch whole regions between treatment and control over time. A new design from Zeng, Johari, Wager and colleagues re-randomizes every period to keep switchback groups balanced.

Key takeaways:

• Why switchbacks struggle. Few units, big differences between markets, conditions that change over time, and carryover from earlier periods all make standard A/B tests unreliable.
• Re-randomize every period. SRSB redraws assignments each period to balance prognostic variables built from past data, such as lagged outcomes and covariates.
• Valid inference. The authors provide finite-sample randomization inference under a sharp null, plus asymptotic inference as the number of periods grows.
• Handling carryover. Blocked SRSB re-randomizes within strata defined by the previous treatment, creating stable “stay” groups to compare.
• Evidence so far. Extensive simulations show gains over standard switchback designs. Real-world deployment will test whether it holds up under fast-changing conditions.

Chapters:

00:00 Not as clear as an X-ray

00:45 The paper

01:32 Why marketplaces use switchbacks

03:06 Few units, big differences

04:37 The carryover problem

05:24 SRSB: re-randomizing each period

07:43 Blocked SRSB for carryover

10:00 Tighter confidence intervals

11:30 Pairing with regression adjustment

13:51 Choosing the switch cadence

14:36 Production caveats

16:07 Recap

Sources:

• Sequentially-Rerandomized Switchback Experiments (Zeng, Adjaho, Bucarey, Qin, Zhang, Hoban, Johari and Wager; arXiv, Apr 2026)

Related episode: E30: Jeunen (arXiv) — Faster A/B Tests by Ignoring Where Policies Agree

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

When the treatment and control in an A/B test pick the same action, that traffic adds noise but no signal. Olivier Jeunen shows how to exploit this policy overlap for unbiased, lower-variance tests of recommenders and rankers.

Key takeaways:

• The hidden inefficiency. Two ranking policies often agree on most decisions. Outcomes where they agree inflate confidence intervals without saying anything about the treatment effect.
• Assignment as a meta-policy. Treat the randomized assignment itself as a policy, log the probability of each action under both policies, and estimate the effect with Δ-off-policy estimation.
• Provably no worse. In the general case the method reduces to standard A/B testing. Its variance scales with how much the policies diverge, not with raw outcome noise, so it beats difference-in-means whenever the policies overlap.
• Complements CUPED. CUPED uses historical data to cut noise. This uses the structure of the two policies, so the two can in principle be stacked.
• Caveat. You need action probabilities from both policies, which is harder for black-box generative systems.

Chapters:

00:00 A great algorithm, a stalled test

00:45 Jeunen's paper

01:30 The A/B testing bottleneck

02:16 When policies agree

04:34 Logging probabilities, not just actions

05:19 Measuring policy overlap

06:51 Off-policy estimation

08:23 Does it work in practice?

09:54 Comparison with CUPED

12:14 The LLM problem

13:46 Related work

17:34 Caveats

19:07 Recap

Sources:

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap (Olivier Jeunen; arXiv, Jul 2026)

Related episode: E31: Stanford & Marketplace Researchers — Rerandomizing Switchback Experiments

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Kuaishou E-commerce built A/B Agent, an LLM agent that proposes recommendation strategies, learns from live A/B results and gets better each round. In production it lifted GMV 4.829% without hurting guardrail metrics.

Key takeaways:

• Institutional memory. Knowledge from past experiments is usually scattered. A/B Agent organizes historical strategies into a hierarchical experience tree by business scenario, recommendation stage, objective and context.
• Tree-RAG. Multi-path retrieval walks the tree to find transferable evidence, avoiding the mismatches of flat retrieval.
• Generate, test, evolve. The agent writes executable strategies, analyzes online A/B feedback to tune them, and updates the tree.
• Results. Built on a benchmark of 310 historical strategies from three Kuaishou scenarios. Online it delivered a 4.829% GMV lift with positive guardrail metrics.
• Caveat. The lift is author-reported, and the best of many automated tests can overstate the true effect.

Chapters:

00:00 The restaurant analogy

01:32 When AI runs the experiments

02:18 Hundreds of A/B tests at once

03:50 Experts don't scale

04:35 Historical amnesia

05:22 The experience tree

07:42 Tree-RAG retrieval

08:27 Generating strategies

09:58 Self-evolution and guardrails

10:43 Results: GMV up 4.829%

12:17 How novel is it?

14:36 Caveats: lucky winners

16:07 Closing question

Sources:

• A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing (Jiang et al., Kuaishou; arXiv, Aug 2026)

Related episode: E9: NetEase — The Agent That Audits Its Own Recommender

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Single-step retrieval is fast but weak on multi-hop questions, and step-by-step search is accurate but slow. Databricks trained a search agent to decide when it has searched enough: frontier-level quality in 5.8 seconds.

Key takeaways:

• The trade-off. Instructed-Retriever-1 handled most requests with fast single-step search, but multi-hop questions need iterative searching that adds latency.
• Learn when to stop. With online RL (CISPO, Clipped Importance Sampling Policy Optimization), the agent is rewarded for good results and penalized for extra steps that don't pay off.
• Built-in guardrails. A fixed upper bound caps sequential steps, and synthetic multi-hop questions teach it when more searching helps.
• Pick your operating point. Tuning the step penalty traces a Pareto frontier from faster to more thorough.
• Results. It matches leading models' quality while answering in 5.8 seconds on average, more than 2x faster than Claude Sonnet 5, DeepSeek-V4-Flash and GPT-5.6 Luna. These are vendor-reported benchmarks.

Chapters:

00:00 The hidden wait in enterprise AI

01:32 More compute isn't the fix

03:04 Why single-step fails multi-hop

04:36 The cost of sequential search

05:22 Online RL and CISPO

06:55 Synthetic multi-hop questions

07:42 Deciding when to stop

08:27 A hard cap on steps

09:13 The Pareto frontier

10:47 5.8 seconds

12:17 How novel is it?

13:50 Fan-out vs. adaptive search

17:39 Missing details

19:12 Recap

Sources:

• Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency (Databricks Blog, Sep 2026)

Related episode: E24: Google — Making AI Search Ask Better Questions, 20x Faster

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Grab's worry wasn't blatant AI hallucination but subtle plausibility: SQL that looks right but changes the metric, a tool call with drifted parameters. Grab Bench tests models on Grab-shaped work and explains every failure.

Key takeaways:

• Plausible isn't correct. Public leaderboards answer a different question from whether a model keeps a metric definition, obeys an internal tool contract, or avoids breaking hidden behavior.
• Safe but realistic cases. Synthetic or redacted cases keep the hard parts of real work, like distractors, stale context and ambiguous evidence, with no live user data.
• Score contracts, not confidence. Each task owns its scoring contract. Deterministic scorers are used wherever possible and LLM judges only for open-ended tasks like SQL.
• Make shortcuts fail. Shortcut baselines, such as citing every piece of evidence or passing only the visible tests, must fail. If one passes, the benchmark isn't ready.
• The failure taxonomy matters most. More reasoning helped some tool use but hurt literal schema discipline. Synthetic evals still don't prove production uplift.

Chapters:

00:00 The flawless-looking candidate

01:30 Moving past leaderboards

02:16 When plausibility becomes a threat

04:34 How Grab Bench works

05:20 Realistic synthetic cases

06:54 Contract-based scoring

08:28 Gaming the rubric

10:46 Row-level failure taxonomies

12:19 More reasoning isn't always better

14:36 Measurement hygiene

15:22 Industry context

17:40 Limitations

19:59 Closing thoughts

Sources:

• Grab Bench: Evaluating AI on Grab-shaped production work (Grab Engineering, 2026)

Related episode: E22: IBM Research — Why Your AI Agent Might Not Ace the Same Task Twice

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

“p < 0.05, ship it” answers the wrong question, argue Max Farrell, Malika Korganbekova and Sanjog Misra. Their robust A/B decision rule is as easy to compute as a t-test and cut regret across 552 real ad experiments.

Key takeaways:

• The wrong question. A t-test asks whether two arms differ in your sample. A business needs to know which arm pays off where it will actually be deployed.
• Ambiguity aversion. Each arm is judged by its worst plausible value over outcome distributions close to the one observed in the test, so high-variance “wins” get penalized.
• A closed-form shortcut. Thanks to the Donsker-Varadhan representation, the rule needs only standard A/B outcome data and one trust parameter for how much you trust the experiment.
• Real-world test. On an archive of 552 advertising experiments from a US online platform, the rule substantially reduced regret compared with significance testing.
• Caveat. It improves the decision step; it doesn't fix a badly run experiment, and the trust parameter still takes judgment.

Chapters:

00:00 A significant winner, then a flop

00:45 The paper

02:18 The standard ship-it workflow

03:04 What a t-test was built for

04:37 Why winners don't transfer

06:09 A decision-theory view

06:55 The ambiguity-averse traveler

08:27 Making it computable

09:59 The closed-form shortcut

11:32 The trust parameter

13:04 552 real ad experiments

13:50 Regret and fragile winners

16:07 How novel is it?

19:10 Caveats

20:40 Recap

Sources:

• Robust A/B Decisions (Max H. Farrell, Malika Korganbekova and Sanjog Misra; arXiv, Sep 2026)

Related episode: E17: Spotify — Why Bayesian A/B Testing Doesn't Pay Off

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Instacart rounded every card-authorization hold up to the next $5, so orders a cent apart could get very different holds. Its economists used that quirk as a natural experiment to measure how holds drive card declines.

Key takeaways:

• The quirk. Each order gets a temporary hold above the order total to cover changes, and the buffer was rounded up to the next $5 multiple. A cent above a threshold meant a bigger hold.
• Why not just A/B test? Experimenting with live payment flows is risky, so the team first looked for a causal answer in historical data.
• Regression discontinuity. Customers just below and just above a $5 cutoff are essentially the same. Fitting local regressions on each side isolates the jump in declines caused by the bigger hold.
• Validated by experiment. A follow-up A/B test that reduced the upward rounding confirmed the finding: fewer card declines and a significant increase in order volume.
• Know the limits. RDD estimates a local effect for customers near the cutoff, not a universal law.

Chapters:

00:00 A tiny rounding rule

01:34 Authorization holds explained

03:08 Rounding to the nearest $5

03:53 The pain of a declined card

05:24 Why not A/B test payments?

06:10 An accidental experiment

06:55 Regression discontinuity design

09:12 Fitting each side of the cutoff

10:44 What the data showed

11:30 Confirming with an A/B test

13:01 Our take on novelty

15:19 Thresholds hiding in your product

16:51 RDD vs. other methods

19:10 Strengths and limits

21:27 Recap

Sources:

• Optimizing at the Edge: Using Regression Discontinuity Designs to Power Decision-Making (Levi Boxell, Robert Fletcher and Tilman Drerup, Instacart Tech)

Related episode: E31: Stanford & Marketplace Researchers — Rerandomizing Switchback Experiments

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.