Data Science in the Wild

Uber's PRD Evaluator is an AI first-pass reviewer that assembles company context, calibrates review depth by risk, and returns a scorecard with write-ready fixes—before human checkpoints.

Key takeaways:

• Upstream of checkpoints. The tool strengthens the PRD before Uber's review forums; it does not approve launches or replace senior judgment (Lakshmi Ashok, May 2026).
• Four steps. Assemble context (linked docs, decks, notes, prior experiments, Uber principles/metrics/JTBD); classify light→moderate→full→specialized (policy/pricing/marketplace); assess Opportunity & Hypothesis, Product Scope, UX & Impact, Metric & Data Rigor; emit a scorecard with a "start here" fix, write-ready text, and critical vs optimization items.
• Early use. Dozens of PMs used it to find blind spots, pressure-test headroom assumptions, and surface adjacent impacts; Uber reports sharper, faster reviews (qualitative only).
• Hard boundaries. Critical-gap floors are meant to stop sycophantic "looks good" rubber stamps when fundamentals are missing.
• Our caveats. The post gives no accuracy, false-positive, or human-agreement numbers. Hosts' novelty 3/5 and Airbnb LLM-eval contrast are commentary, not from the Uber post. Score labels like "ready / not ready" are the hosts' framing of the launch-readiness rating.

Chapters:

00:00 Cold open: AI flags bad math before the room

01:08 Mission: Uber's May 2026 PRD Evaluator

01:44 Why cross-functional review bogged down

03:55 Raise the baseline, don't replace editors

04:37 Four steps: context, classify, assess, scorecard

07:04 Risk-based classification and blast radius

08:12 Dimensions and the actionable scorecard

10:07 Early use by dozens of PMs

11:58 Novelty, hard floors, and sycophancy

14:15 Related: Airbnb grades AI, Uber grades PRDs

15:50 Our critique: no quantitative metrics

18:39 Recap and closing question

Sources:

• Lessons from Building a First-Pass AI PRD Reviewer at Uber (Lakshmi Ashok; Uber Engineering Blog, May 12, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

ByteDance's OneTrans-V2 runs retrieval, pre-rank and fine-rank as one Transformer: +9.74% GMV per user and 3.2× QPS on the same hardware in a 50/50 A/B test.

Key takeaways:

• One model, cascade kept. The three stages become tasks of one causal Transformer that encodes the user's behavior sequence once per request. Each stage keeps its own candidate features and budget, and fine-rank distills into pre-rank inside the model.
• Results. Against the deployed cascade (OneTrans-GR + OneTrans-Lite + OneTrans), user-level 50/50 A/B: GMV per user +9.74%, active days +0.25%, 3.2× QPS on the same hardware despite more FLOPs per request.
• DCGR retrieval. The model first predicts a decision prefix (purchase level, value, discovery, organic vs sponsored) as non-linguistic tokens, then generates item semantic IDs. One model replaced three retrieval channels: 1.5× main-channel GMV, 2.2× new categories, 2.1× sponsored GPM.
• Where the speed comes from. Kernel fusion + FP8 (2.16× on fine-rank) times one shared user context (1.53×). SNT is a training change: 4.4× faster training, 95% less sequence storage.
• Note. The paper doesn't name the platform or give CIs, but it does define the baseline and A/B design. The text-like prefix and per-click delta cache were hosts' illustrations.

Chapters:

00:00 A 10% GMV lift from one model

01:26 How the recsys cascade works

03:59 Three models, three encodings

04:56 One shared Transformer + distillation

07:05 Mixture of experts and µP

09:38 Decision-conditioned generative retrieval

11:35 Sequence-native training

13:08 Results: GMV and throughput

15:00 Novelty score

16:00 Related: LinkedIn Feed SR

16:42 Caveats and the offline-testing dilemma

19:18 Recap and closing thought

Sources:

• OneTrans-V2: Unifying Retrieval, Pre-rank, and Fine-rank with One Transformer in Industrial Recommender (Cao et al., ByteDance; arXiv 2609.28589, Sep 2026)
• An Industrial-Scale Sequential Recommender for LinkedIn Feed Ranking (Feed SR) (LinkedIn, related; arXiv 2602.12354, Feb 2026)

Related episode: S2E3: Meta — Expected Bayesian Loss for Neutral Launches

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Meta's Expected Bayesian Loss turns neutral-launch safety from a CI-overlap yes/no into a tunable downside-risk number, matched to experts on 17 experiments.

Key takeaways:

• CI overlap is broken for neutral launches. Containing zero is permissive with scarce/noisy data and oversensitive with abundant data — absence of evidence ≠ harmlessness.
• EBL = P(harm) × severity. Closed form from frequentist m and SE s (Jeffreys prior): EBL = s φ(m/s) − m Φ(−m/s). Noisy experiments get higher loss.
• Expert-matched thresholds. On 17 similar-precision experiments (14 launch / 3 block), τ_B = 100 bp (~80% regression probability); review at 30 bp / 40%. LOO: 14/14 launches, 2/3 blocks; lower-bound CI review <80% vs EBL >95%.
• Governance, not a magic dial. τ_B states how much expected metric loss the org will absorb. Multi-metric needs Min-Max or weighted loss — don't aggregate naively.
• Note. Hosts dated Statsig's multiple-comparisons post Oct 2026 (it's Sep 29) and Meta's Ladder Sep 2026 (it's Oct 2025). EBL doesn't by itself fix multiplicity or sit on that ladder.

Chapters:

00:00 Why “safe” updates still break apps

02:14 What a neutral launch actually is

03:14 CI overlap: abundant vs scarce data

07:09 Expected Bayesian Loss

08:43 Bayesian risk from frequentist SE

11:50 Expert validation and the risk dial

14:14 Novelty score and caveats

17:08 Related work in the ecosystem

20:06 Recap and closing question

Sources:

• Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches (Alcain et al., Meta; arXiv 2610.10223, Oct 2026)
• Multiple comparisons: More comparisons, more problems (Statsig, related, Sep 2026)
• Ladder of evidence… new products — Part I (Meta Analytics, related, Oct 2025)
• Accelerating A/B-Tests with Counterfactual Estimation (related, arXiv 2607.14604)

Related episode: S2E2: Etsy — Using Post-Treatment Data to Sharpen A/B Tests

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Etsy and UC Berkeley stack 23 screened in-experiment covariates on top of CUPAC, cutting A/B-test variance a further ~2.5–14% across 29 experiments.

Key takeaways:

• Two stages. Stage 1 is Etsy's unchanged CUPAC (one LightGBM model, 117 pre-experiment covariates). Stage 2 is a linear adjustment on 23 in-experiment covariates, fit to CUPAC's residuals.
• Mean equivalence, not independence. A post-treatment covariate is usable if its mean matches across arms. That's testable; if it fails by Δ, the bias is −γᵀΔ.
• What Etsy ran. Drop too-sparse count variables, Mann–Whitney U per experiment, Fisher's method across experiments. No FWER correction, no TOST. The screen is a diagnostic, not proof, paired with domain knowledge.
• Results. On conversion rate, √R² rose 0.02 to over 0.14 vs. CUPAC; the extra variance cut was ~2.5–14% per experiment on top of CUPAC's ~10–25% (Fig. 2).
• Note. The paper is from CLeaR 2026 (PMLR 323), not ICLR. ByteDance's CUPED paper (Jun 2026) is parallel work; Etsy's predates it.

Chapters:

00:00 Why Etsy's traffic still isn't enough

02:40 CUPED, CUPAC and “forbidden” data

04:57 Two stages: CUPAC plus a linear add-on

07:15 Mean equivalence, a testable condition

08:06 What Etsy ran, and what it didn't

11:06 A diagnostic, not a proof; the −γᵀΔ bias

13:38 Results across 29 experiments

15:56 Caveat: covariates can drift

16:28 What's new, and related work

18:44 Recap and closing question

Sources:

• Variance reduction combining pre-experiment and in-experiment data (Lin & Crespo; CLeaR 2026, PMLR 323; arXiv 2410.09027)
• Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED (ByteDance, related, Jun 2026)
• CUPED on Steroids: Multivariate Covariate Adjustment for Switchback Experiments (related, Aug 2026)

Related episode: S2E1: Pinterest — A Cheap LLM Judge Plus Smarter Sampling Cuts Search-Experiment MDE

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Pinterest's fine-tuned XLM-RoBERTa relevance judge made stratified sampling affordable, cutting search A/B-test MDE from 1.3–1.5% to ≤0.25%.

Key takeaways:

• A small, fast judge. An XLM-RoBERTa-large cross-encoder fine-tuned on ~2.6M human-labeled query–Pin pairs labels 150,000 rows in 30 minutes on one A10G GPU, using text only (incl. BLIP image captions). Llama-3-8B was slightly better but ~6x costlier.
• Sampling is the payoff. Bigger samples, strata by query interest × popularity, Neyman allocation and a stratified estimator took MDE from 1.3–1.5% to ≤0.25%. At 2,000 queries, stratification alone cut the standard deviation 52%.
• Close to humans. 73.7% exact label match, 91.7% within one point; US sDCG@25 correlations τ 0.652, ρ 0.817. Humans still grade relevance-model tests.
• Our take. Novelty ~3/5: a standard model, smart measurement engineering. Watch for a judge that favors treatments built like itself.
• Note. τ and ρ compare how LLM- and human-based scores rank queries, not the order of Pins on a page. The 67% cut is the full design at 5,000 queries vs. 2,000 random ones. France/Germany (τ ≈ 0.47) is lower but still “strong” per the paper.

Chapters:

00:00 When an A/B test says nothing changed

01:07 Pinterest's LLM relevance paper

02:30 MDE: the rusty bathroom scale

04:01 Choosing the judge: Llama-3-8B vs. XLM-RoBERTa

06:26 A text-only judge on a visual platform

08:36 sDCG@25 and the long-tail problem

10:33 Stratified sampling and Neyman allocation

13:19 Results: MDE ≤0.25%, and agreement with humans

15:52 Our take: novelty and the language gap

18:12 Judge bias and the human safety net

19:34 Related work, recap, closing question

Sources:

• LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest (Wang et al.; RecSys ’25 EARL workshop, Sep 2025)
• Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects (related, Oct 2026)
• The Quest to Understand Metric Movements (Pinterest Engineering, 2025)

Related episode: E15: Netflix — Grading the Graders: Can an AI Judge Your Synopsis?

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Token budgets for RAG: which questions deserve extra retrieval? On FinanceBench, the cost of an extra call is easy to predict (R² 0.93) but its benefit isn't (AUROC 0.60). Gain-per-token escalation still uses 46.6% fewer tokens than top-5 everywhere.

Key takeaways:

• Escalate selectively. Every question gets a cheap top-1 answer. Only the calls with the best predicted gain per token get a top-5 retry.
• Cost is predictable; benefit isn't. Extra-call cost is predicted with R² 0.93. Spotting which questions will improve reaches only AUROC 0.60, where 0.5 is a coin flip.
• It still pays off. At a 10% budget, adjudicated quality is 0.034 higher than with gain-only ranking, using 46.6% fewer tokens than top-5 on every question.
• It's uplift targeting. Spend on “persuadable” queries that top-5 can fix. Skip the sure things and the lost causes.
• Our caveats. The test uses only 150 questions, and a weak predictor will starve some queries that needed more context.

Chapters:

00:00 Why every retrieved token has a price

01:26 The paper, and the cost of top-5 everywhere

04:01 FinanceBench setup and gain per token

06:10 Cost R² 0.93 vs. benefit AUROC 0.60

08:32 Results: +0.034 quality, 46.6% fewer tokens

09:43 What's new: uplift targeting

11:49 Related work on agentic RAG budgets

14:14 Limitations, and when triage makes sense

16:42 Recap: should we pay for uplift?

Sources:

• Token-Budgeted Escalation for Financial Document QA: Cost Is Predictable, Benefit Is the Bottleneck (Zhu, Yang, Ding, Qi; arXiv, Oct 2026)
• FinanceBench (benchmark)
• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (related)
• Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks (related)

Related episode: E32: Agentic RAG Evaluation — More Questions Beat More Repeats

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

With a fixed token budget for evaluating an agentic RAG system, should you ask more questions, sample more search trajectories, or re-grade answers? This paper finds that more questions win, cutting standard error by up to 33%.

Key takeaways:

• Split the noise. Generalizability theory separates evaluation variance into questions, search trajectories and repeated reads, tested on HotpotQA and MuSiQue.
• Go wide. At about 34 million tokens, broader question coverage lowered standard error by 33% compared with five reads, and by 12.6% compared with three trajectories.
• One read is usually enough. Reading each answer once cost only 0–9.9% in variance relative to the optimal allocation.
• Set the judge to temperature zero. It cut answer disagreement from 14.3% to 3.4%, with similar comparison precision.
• Plan ahead. Forecasts from a small pilot predicted the allocations within 3.5–4.0%, so you can budget before spending. Results are on academic benchmarks.

Chapters:

00:00 Wasted evaluation budgets

01:31 Why agentic RAG evals are noisy

03:05 Brute-force repeats

03:52 Generalizability theory

05:24 Judge disagreement and temperature

06:56 Spending 34 million tokens

08:30 Why one read is enough

10:01 Borrowing from A/B power analysis

11:32 Benchmark caveats

13:07 Planning your token spend

14:37 Recap

Sources:

• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (Jingjie Ning, Xueqi Li and Yibo Kong; arXiv, Oct 2026)

Related episode: E33: Token-Budgeted Escalation — Predicting Cost Is Easy, Predicting Benefit Is Hard

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

When platforms can't randomize users, they switch whole regions between treatment and control over time. A new design from Zeng, Johari, Wager and colleagues re-randomizes every period to keep switchback groups balanced.

Key takeaways:

• Why switchbacks struggle. Few units, big differences between markets, conditions that change over time, and carryover from earlier periods all make standard A/B tests unreliable.
• Re-randomize every period. SRSB redraws assignments each period to balance prognostic variables built from past data, such as lagged outcomes and covariates.
• Valid inference. The authors provide finite-sample randomization inference under a sharp null, plus asymptotic inference as the number of periods grows.
• Handling carryover. Blocked SRSB re-randomizes within strata defined by the previous treatment, creating stable “stay” groups to compare.
• Evidence so far. Extensive simulations show gains over standard switchback designs. Real-world deployment will test whether it holds up under fast-changing conditions.

Chapters:

00:00 Not as clear as an X-ray

00:45 The paper

01:32 Why marketplaces use switchbacks

03:06 Few units, big differences

04:37 The carryover problem

05:24 SRSB: re-randomizing each period

07:43 Blocked SRSB for carryover

10:00 Tighter confidence intervals

11:30 Pairing with regression adjustment

13:51 Choosing the switch cadence

14:36 Production caveats

16:07 Recap

Sources:

• Sequentially-Rerandomized Switchback Experiments (Zeng, Adjaho, Bucarey, Qin, Zhang, Hoban, Johari and Wager; arXiv, Apr 2026)

Related episode: E30: Jeunen (arXiv) — Faster A/B Tests by Ignoring Where Policies Agree

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

When the treatment and control in an A/B test pick the same action, that traffic adds noise but no signal. Olivier Jeunen shows how to exploit this policy overlap for unbiased, lower-variance tests of recommenders and rankers.

Key takeaways:

• The hidden inefficiency. Two ranking policies often agree on most decisions. Outcomes where they agree inflate confidence intervals without saying anything about the treatment effect.
• Assignment as a meta-policy. Treat the randomized assignment itself as a policy, log the probability of each action under both policies, and estimate the effect with Δ-off-policy estimation.
• Provably no worse. In the general case the method reduces to standard A/B testing. Its variance scales with how much the policies diverge, not with raw outcome noise, so it beats difference-in-means whenever the policies overlap.
• Complements CUPED. CUPED uses historical data to cut noise. This uses the structure of the two policies, so the two can in principle be stacked.
• Caveat. You need action probabilities from both policies, which is harder for black-box generative systems.

Chapters:

00:00 A great algorithm, a stalled test

00:45 Jeunen's paper

01:30 The A/B testing bottleneck

02:16 When policies agree

04:34 Logging probabilities, not just actions

05:19 Measuring policy overlap

06:51 Off-policy estimation

08:23 Does it work in practice?

09:54 Comparison with CUPED

12:14 The LLM problem

13:46 Related work

17:34 Caveats

19:07 Recap

Sources:

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap (Olivier Jeunen; arXiv, Jul 2026)

Related episode: E31: Stanford & Marketplace Researchers — Rerandomizing Switchback Experiments

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Kuaishou E-commerce built A/B Agent, an LLM agent that proposes recommendation strategies, learns from live A/B results and gets better each round. In production it lifted GMV 4.829% without hurting guardrail metrics.

Key takeaways:

• Institutional memory. Knowledge from past experiments is usually scattered. A/B Agent organizes historical strategies into a hierarchical experience tree by business scenario, recommendation stage, objective and context.
• Tree-RAG. Multi-path retrieval walks the tree to find transferable evidence, avoiding the mismatches of flat retrieval.
• Generate, test, evolve. The agent writes executable strategies, analyzes online A/B feedback to tune them, and updates the tree.
• Results. Built on a benchmark of 310 historical strategies from three Kuaishou scenarios. Online it delivered a 4.829% GMV lift with positive guardrail metrics.
• Caveat. The lift is author-reported, and the best of many automated tests can overstate the true effect.

Chapters:

00:00 The restaurant analogy

01:32 When AI runs the experiments

02:18 Hundreds of A/B tests at once

03:50 Experts don't scale

04:35 Historical amnesia

05:22 The experience tree

07:42 Tree-RAG retrieval

08:27 Generating strategies

09:58 Self-evolution and guardrails

10:43 Results: GMV up 4.829%

12:17 How novel is it?

14:36 Caveats: lucky winners

16:07 Closing question

Sources:

• A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing (Jiang et al., Kuaishou; arXiv, Aug 2026)

Related episode: E9: NetEase — The Agent That Audits Its Own Recommender

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.