Data Science in the Wild

From Prompts to Posteriors (Filbry, Melo & Gal, Oxford): LLMs break Bayes' rule without reasoning, but reasoning models compute simple posteriors almost exactly, and with code even intractable ones.

Key takeaways:

• Three tests. Dutch-book coherence (Gemini-2.5-Pro, Gemma-4-31B-it, Qwen-2.5-32B-Instruct), then posterior fidelity (KL to a Bayesian oracle) and experimental design (posterior-mass gap) on 20 Questions and the new ProteinWorld, with five open models.
• Coherence depends on the chat. In separate chats Gemini gave 50% to a Democrat winning in 2028 and 55% to a non-Democrat; violations top 50% of scenarios for almost all model and axiom pairs. Within one dialogue they fall to 10% for Gemini and 1 of 30 for Gemma.
• No reasoning, no Bayes. Without reasoning, the mass gap on the true answer exceeds 30% for every model in both environments; strong models often pick the right answer but with the wrong probability.
• Reasoning, then code. With reasoning alone, KL is near 0 on closed-form ProteinWorld and Gemma-4-31B-it matches the oracle, doing Bayes in its reasoning, not Python. On intractable ProteinWorld-hard, Gemini-2.5-Pro faked simulation output; with Python it wrote a correct nested Monte Carlo estimator and nearly matched the oracle.
• Our take. A plain reasoning prompt matched the BED-LLM harness on 20Q, so strict harnesses may no longer be needed. Caveats: close, not perfect; 20Q still falls short of the oracle; 40 runs per environment.

Chapters:

00:00 Cold open: the paper and the question

01:20 Bayesian inference, detective-style

02:16 The old assumption and its harnesses

02:58 Coherence: 50% vs 55% in separate chats

06:00 Updating without reasoning

07:38 Reasoning, no tools: KL near 0

10:06 ProteinWorld-hard: intractable posteriors

11:47 Code execution: nested Monte Carlo

13:21 Related work and auditing

15:10 Caveats: close, not perfect

15:58 Recap and closing question

Sources:

• From Prompts to Posteriors: Can LLM Agents Perform Bayesian Inference in Context? (Filbry, Melo, Gal, Oxford; arXiv, Oct 21, 2026)
• BED-LLM (Choudhury et al.; ICLR 2026)
• BayesBench (Samanta et al.; arXiv, 2026)
• LLMs are not (consistently) Bayesian (Chen et al.; arXiv, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Instagram Feed's ranker stops deleting "bias": Meta keeps the raw engagement prediction and adds its debiased, incremental part as a second, separately weighted term.

Key takeaways:

• Baseline vs incremental. Each value-model prediction mixes a baseline (a serial liker, a popular creator, a strong batch) with the candidate's own excess. Most prior work removes the baseline; Meta keeps the raw term and weights the incremental term separately (Eq. 1–2). No causal claim.
• One operator, three axes. Viewer, item and request debiasing are one operator (Eq. 3; per-term forms in Table 4). The request baseline is the p75 or mean of predictions over the request's own candidates, not time of day or device.
• Six four-arm live ablations (none/raw/incremental/both, 3% of traffic per arm, one week). Both beats each single term on point estimates in 11 of 13 pre-assigned (term, metric) pairs, 7 with both contrasts significant; 17 of 26 contrasts significantly for, 1 against (impressions).
• Beyond Feed. Separate Reels launches (Table 6): sessions +0.029% to +0.083%, DAU +0.026% to +0.042%.
• Our caveats. A Meta-authored case series of six terms; production weights unpublished; single-term arms not re-tuned; small session effects. Offline diagnostics should screen signals, not rank them.

Chapters:

00:00 Cold open: why your feed shows what it does

00:58 The paper: Baseline and Incremental

01:56 The value model's two hidden parts

04:10 Removing the baseline vs keeping it

06:30 Two terms and one operator (Eq. 1–3)

07:58 Request context = the live candidate pool

10:26 The four-arm live ablation

11:58 Reels launches: sessions and DAU

13:15 Judged pairs and multiplicity

15:00 Results: 11 of 13, 17 of 26, one loss

16:11 Novelty and our caveats

17:54 Offline diagnostics: screen, don't rank

18:53 Related work: testing in the wild

20:16 Recap and closing question

Sources:

• Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking (Chen, Wang, Zhou et al., Meta; arXiv, Oct 20, 2026)
• Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations (Tannenbaum; arXiv, 2026; our S2E9)
• Don't Rewrite the SID Map: Topology Preservation for Generative Retrieval in Dynamic E-Commerce Search (Zhu et al., Alibaba; arXiv, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Selling in the Wild: in a 14.8M-visitor field trial, the Javo AI sales agent lifted revenue per visitor 10% and held its own against human chat staff at 1.8% of the cost.

Key takeaways:

• Four live arms. 100 sites (e-commerce, travel, high-consideration services incl. B2B logistics), May–Sep 2026: no agent 2.6M, Javo 5.6M, frontier agent 5.7M, human chat 0.9M. Both bots disclosed they were AI. Javo (Qwen3-32B tuned on 4.6M consented sales chats; 2,240 H100 GPU-hours) raised revenue per visitor +10.0% (95% CI +5.1 to +14.9) on 85 transactional sites.
• Javo vs GPT-5.2, same widget. Engaged conversion +21.6% (+16.9 to +26.3); revenue per visitor +16.9%. Identical launcher, greeting and AI line, and the visitor writes first, so the model can't change who engages: 8.07% vs 8.06% (p = 0.536).
• Javo vs typical human associates (20 staffed sites): engaged conversion 11.26% vs 11.20% (+0.5%, −5.0 to +6.0), non-inferior (margin −7%; −10% in the design), not superior. $0.06 vs $3.40 per engaged conversation. Widgets differed (Javo proactive; 8.07% engaged overall vs 5.80% for humans); trimming bounds are planned but not reported.
• Guardrails held over five months: unsupported claims 0.15% of turns (frontier 0.56%), no unauthorized offers; returns, complaints in margin.
• Our caveats. The sole author is Founder/CEO of AISO Boost Ltd., which operates Javo; no independent review, registration or ethics approval is claimed. Sites came partly from his client network; one frozen GPT-5.2 snapshot.

Chapters:

00:00 Cold open: wind tunnel vs rush hour

01:48 The paper: Selling in the Wild

02:51 Javo: 4.6M sales chats, 2,240 GPU-hours

04:19 14.8M visitors and four studies

06:28 Revenue lift and intention-to-treat

08:23 Skeptic check: who chooses to chat?

10:14 Javo vs humans: conversion and cost

11:48 Human widgets and trimming bounds

13:21 Guardrails and novelty

14:53 Related work

17:00 Caveats: conflict of interest

19:04 Recap and closing question

Sources:

• Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations (Benjamin Tannenbaum; arXiv, Oct 18, 2026)
• Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches (Alcain et al.; arXiv, 2026)
• Measuring Gift Card Program Incrementality via Causal Data Fusion (Whitehouse et al.; arXiv, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Conformal Policy Selection (Cao, Gao, Saar-Tsechansky & Yin) gates AI decisions person by person—zero harmful EHR treatments deployed for q ≤ 0.85.

Key takeaways:

• Per unit, not on average. Each candidate is a test: will the outcome under the AI policy clear a threshold? Weighted conformal p-values correct for the behavior-to-target policy shift; a Benjamini–Hochberg-style rule plus randomized pruning bounds the FDR (the expected share of false selections, not a per-person guarantee).
• Doubly robust, within limits. Correct propensity weights give finite-sample FDR ≤ q; a correct outcome model with wrong weights gives asymptotic control. In linear synthetic tests FDR held when either model was right; with both wrong it reached 0.29 at a 0.15 target.
• EHR safety gate. On real covariates (5,993 patient-years, age 60+, BMI 30+) with simulated outcomes, treatment harms nearly everyone, yet a standard risk rule treats the top-risk decile. The gate deployed none of those treatments for q ≤ 0.85 (FDR 0.046 at q = 0.75). Semi-synthetic: no claims about real patients.
• Our caveat: power. Safety can mean few selections. The paper reports ~0.80 power at q = 0.95 (synthetic) but 0.012 at q = 0.15 on LaLonde (≈1.9 of ~185 selected).

Chapters:

00:00 Cold open: the 80/20 drug

01:25 Who should get the AI decision?

03:17 A personalized guess vs a guarantee

04:28 Weighted conformal p-values

06:50 BH-style selection, pruning and FDR

08:44 Doubly robust: two models, two chances

09:55 Synthetic tests: where control breaks

11:50 EHR study: the deployment safety gate

14:02 Novelty and related work

16:13 Our caveat: statistical power

18:21 Recap and closing question

Sources:

• Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention (Cao, Gao, Saar-Tsechansky & Yin; arXiv, Oct 16, 2026)
• Conformal Prediction Under Covariate Shift (Tibshirani, Barber, Candès & Ramdas; NeurIPS 2019)
• Covariance-Calibrated Prediction-Powered Inference (Chen, Bai & Zhu; arXiv, 2026)
• Probabilistic Forecasting of Electric Vehicle Charging in a Time-varying Hierarchical Infrastructure (Principato et al.; arXiv, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

ZIC-BCF-Smear (Souto, Wang & Shuryak) pairs two Bayesian causal forests for zero-heavy outcomes—finding panitumumab adds ~10 severe-toxicity days per patient.

Key takeaways:

• Two forests, one effect. A Probit BCF on everyone models clearing zero; a Gaussian BCF on log intensity uses active units only. Duan's smearing (mean of exponentiated residuals) replaces the fragile exp(σ²/2) back-transform.
• SPA, precisely. It re-estimates the treatment propensity within the active subset (not the odds of being active) and feeds it to the prognostic forest to mitigate regularization-induced confounding.
• Simulations. Across log-normal, Gamma and Tweedie data (3–85% zeros, 100 seeds) it matches well-specified parametric hurdles; on nonlinear surfaces it has the lowest CATE RMSE and no divergences, while Gamma models diverge and DPglm over-covers. Mild under-coverage (89.7–91.7%) remains.
• SPECTRUM trial. In 520 head-and-neck cancer patients (23.7% zeros), panitumumab + chemo adds 10.0 distinct severe-toxicity days per patient (95% CrI 3.6–16.6), mostly via how many days, not whether any; that margin is uncertain.
• Our caveats. Prep was abstract-level. Treatment time differed (83.7 vs 130.3 days); at a common day 84–120 the gap is ~3–4 days. The Etsy paper we contrast is from Oct 2024 (CLeaR 2026).

Chapters:

00:00 Cold open: weighing a ghost

01:37 Semicontinuous outcomes: a spike at zero

02:49 Why today's options fail

05:33 ZIC-BCF-Smear: two decoupled forests

07:35 Selection bias and SPA

10:38 Duan's smearing: back to days and dollars

13:06 Simulations: stable where others diverge

14:13 SPECTRUM trial: ~10 more severe-toxicity days

16:41 Lessons for product experiments

17:16 Novelty (3/5) and related work

20:52 Caveats: abstract-only, treatment duration

23:16 Recap and closing question

Sources:

• Bayesian Causal Forests for Zero-Inflated Continuous Outcomes (Souto, Wang & Shuryak; arXiv, Oct 8, 2026)
• Bayesian regression tree models for causal inference (Hahn, Murray & Carvalho; original BCF, 2017)
• Variance reduction combining pre-experiment and in-experiment data (Lin & Crespo, Etsy; 2024)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

WeChat's WeGR turns e-commerce search into generative retrieval: CoCo-VAE builds collaborative, collapse-free Semantic IDs, and DHPO trains with hierarchical partial credit—lifting orders +2.539% while cutting CPU/GPU ~25%.

Key takeaways:

• Generative retrieval, not a full stack teardown. WeGR generates Semantic IDs for candidates, then still feeds the industrial cascade (candidate generation → pre-ranking → ranking). Deployed V1.2 removes 80% of other-channel candidates—not the channels or pre-ranking themselves.
• CoCo-VAE + DHPO. CoCo-VAE injects Q2I/I2I collaborative signals and fights codebook collapse via refresh/regularization. DHPO expands reward and optimization horizons over SID prefixes, giving partial credit without auxiliary reward models.
• Online wins with less compute. Two-month A/B (June 22–Aug 4, 2026) vs the incumbent: orders +2.539%, CTR +1.508%, CVR +1.381%; CPU −25.5%, GPU −26.1%; ranking load −38.10%; latency −21.7 ms (Channels) / −15.4 ms (Search).
• Novelty (hosts' 4/5). SID generative retrieval (e.g. TIGER) is established; the hosts flag DHPO's sparse-reward fix at WeChat scale as the sharper advance versus tearing down the cascade (contrast: OneTrans-V2-style unification).
• Our caveats. Round "2.5%" once spoken as orders is exactly +2.539%. A two-month window supports near-term gains, not permanent habit change. Stratified Tables 5–7 and daily HR@200 live in the paper body; the abstract alone is thin on breakdowns/CIs.

Chapters:

00:00 Cold open: ask the haystack for the needle

01:12 Mission: WeChat WeGR for e-commerce search

01:32 Why multi-channel retrieval is expensive

03:39 Two bottlenecks: SIDs and sparse RL

05:05 Codebook collapse, explained

06:28 CoCo-VAE: collaborative, collapse-free IDs

11:04 DHPO: partial credit without reward models

13:43 V1.2: keep the cascade, cut 80% of candidates

16:31 A/B results: +2.539% orders, −25% compute

17:44 Novelty vs TIGER and OneTrans-V2

19:12 Caveats: stability, drift, abstract vs body

21:47 Recap and generative-fulfillment teaser

Sources:

• WeGR: Unified Generative Retrieval for Industrial Search Systems in WeChat (Han, Ma, Felix, Xu, Yu, Guo, Wang, Ye, Hua, Xie, Yang & Wu; arXiv, Oct 12, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Uber's PRD Evaluator is an AI first-pass reviewer that assembles company context, calibrates review depth by risk, and returns a scorecard with write-ready fixes—before human checkpoints.

Key takeaways:

• Upstream of checkpoints. The tool strengthens the PRD before Uber's review forums; it does not approve launches or replace senior judgment (Lakshmi Ashok, May 2026).
• Four steps. Assemble context (linked docs, decks, notes, prior experiments, Uber principles/metrics/JTBD); classify light→moderate→full→specialized (policy/pricing/marketplace); assess Opportunity & Hypothesis, Product Scope, UX & Impact, Metric & Data Rigor; emit a scorecard with a "start here" fix, write-ready text, and critical vs optimization items.
• Early use. Dozens of PMs used it to find blind spots, pressure-test headroom assumptions, and surface adjacent impacts; Uber reports sharper, faster reviews (qualitative only).
• Hard boundaries. Critical-gap floors are meant to stop sycophantic "looks good" rubber stamps when fundamentals are missing.
• Our caveats. The post gives no accuracy, false-positive, or human-agreement numbers. Hosts' novelty 3/5 and Airbnb LLM-eval contrast are commentary, not from the Uber post. Score labels like "ready / not ready" are the hosts' framing of the launch-readiness rating.

Chapters:

00:00 Cold open: AI flags bad math before the room

01:08 Mission: Uber's May 2026 PRD Evaluator

01:44 Why cross-functional review bogged down

03:55 Raise the baseline, don't replace editors

04:37 Four steps: context, classify, assess, scorecard

07:04 Risk-based classification and blast radius

08:12 Dimensions and the actionable scorecard

10:07 Early use by dozens of PMs

11:58 Novelty, hard floors, and sycophancy

14:15 Related: Airbnb grades AI, Uber grades PRDs

15:50 Our critique: no quantitative metrics

18:39 Recap and closing question

Sources:

• Lessons from Building a First-Pass AI PRD Reviewer at Uber (Lakshmi Ashok; Uber Engineering Blog, May 12, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

ByteDance's OneTrans-V2 runs retrieval, pre-rank and fine-rank as one Transformer: +9.74% GMV per user and 3.2× QPS on the same hardware in a 50/50 A/B test.

Key takeaways:

• One model, cascade kept. The three stages become tasks of one causal Transformer that encodes the user's behavior sequence once per request. Each stage keeps its own candidate features and budget, and fine-rank distills into pre-rank inside the model.
• Results. Against the deployed cascade (OneTrans-GR + OneTrans-Lite + OneTrans), user-level 50/50 A/B: GMV per user +9.74%, active days +0.25%, 3.2× QPS on the same hardware despite more FLOPs per request.
• DCGR retrieval. The model first predicts a decision prefix (purchase level, value, discovery, organic vs sponsored) as non-linguistic tokens, then generates item semantic IDs. One model replaced three retrieval channels: 1.5× main-channel GMV, 2.2× new categories, 2.1× sponsored GPM.
• Where the speed comes from. Kernel fusion + FP8 (2.16× on fine-rank) times one shared user context (1.53×). SNT is a training change: 4.4× faster training, 95% less sequence storage.
• Note. The paper doesn't name the platform or give CIs, but it does define the baseline and A/B design. The text-like prefix and per-click delta cache were hosts' illustrations.

Chapters:

00:00 A 10% GMV lift from one model

01:26 How the recsys cascade works

03:59 Three models, three encodings

04:56 One shared Transformer + distillation

07:05 Mixture of experts and µP

09:38 Decision-conditioned generative retrieval

11:35 Sequence-native training

13:08 Results: GMV and throughput

15:00 Novelty score

16:00 Related: LinkedIn Feed SR

16:42 Caveats and the offline-testing dilemma

19:18 Recap and closing thought

Sources:

• OneTrans-V2: Unifying Retrieval, Pre-rank, and Fine-rank with One Transformer in Industrial Recommender (Cao et al., ByteDance; arXiv 2609.28589, Sep 2026)
• An Industrial-Scale Sequential Recommender for LinkedIn Feed Ranking (Feed SR) (LinkedIn, related; arXiv 2602.12354, Feb 2026)

Related episode: S2E3: Meta — Expected Bayesian Loss for Neutral Launches

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Meta's Expected Bayesian Loss turns neutral-launch safety from a CI-overlap yes/no into a tunable downside-risk number, matched to experts on 17 experiments.

Key takeaways:

• CI overlap is broken for neutral launches. Containing zero is permissive with scarce/noisy data and oversensitive with abundant data — absence of evidence ≠ harmlessness.
• EBL = P(harm) × severity. Closed form from frequentist m and SE s (Jeffreys prior): EBL = s φ(m/s) − m Φ(−m/s). Noisy experiments get higher loss.
• Expert-matched thresholds. On 17 similar-precision experiments (14 launch / 3 block), τ_B = 100 bp (~80% regression probability); review at 30 bp / 40%. LOO: 14/14 launches, 2/3 blocks; lower-bound CI review <80% vs EBL >95%.
• Governance, not a magic dial. τ_B states how much expected metric loss the org will absorb. Multi-metric needs Min-Max or weighted loss — don't aggregate naively.
• Note. Hosts dated Statsig's multiple-comparisons post Oct 2026 (it's Sep 29) and Meta's Ladder Sep 2026 (it's Oct 2025). EBL doesn't by itself fix multiplicity or sit on that ladder.

Chapters:

00:00 Why “safe” updates still break apps

02:14 What a neutral launch actually is

03:14 CI overlap: abundant vs scarce data

07:09 Expected Bayesian Loss

08:43 Bayesian risk from frequentist SE

11:50 Expert validation and the risk dial

14:14 Novelty score and caveats

17:08 Related work in the ecosystem

20:06 Recap and closing question

Sources:

• Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches (Alcain et al., Meta; arXiv 2610.10223, Oct 2026)
• Multiple comparisons: More comparisons, more problems (Statsig, related, Sep 2026)
• Ladder of evidence… new products — Part I (Meta Analytics, related, Oct 2025)
• Accelerating A/B-Tests with Counterfactual Estimation (related, arXiv 2607.14604)

Related episode: S2E2: Etsy — Using Post-Treatment Data to Sharpen A/B Tests

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Etsy and UC Berkeley stack 23 screened in-experiment covariates on top of CUPAC, cutting A/B-test variance a further ~2.5–14% across 29 experiments.

Key takeaways:

• Two stages. Stage 1 is Etsy's unchanged CUPAC (one LightGBM model, 117 pre-experiment covariates). Stage 2 is a linear adjustment on 23 in-experiment covariates, fit to CUPAC's residuals.
• Mean equivalence, not independence. A post-treatment covariate is usable if its mean matches across arms. That's testable; if it fails by Δ, the bias is −γᵀΔ.
• What Etsy ran. Drop too-sparse count variables, Mann–Whitney U per experiment, Fisher's method across experiments. No FWER correction, no TOST. The screen is a diagnostic, not proof, paired with domain knowledge.
• Results. On conversion rate, √R² rose 0.02 to over 0.14 vs. CUPAC; the extra variance cut was ~2.5–14% per experiment on top of CUPAC's ~10–25% (Fig. 2).
• Note. The paper is from CLeaR 2026 (PMLR 323), not ICLR. ByteDance's CUPED paper (Jun 2026) is parallel work; Etsy's predates it.

Chapters:

00:00 Why Etsy's traffic still isn't enough

02:40 CUPED, CUPAC and “forbidden” data

04:57 Two stages: CUPAC plus a linear add-on

07:15 Mean equivalence, a testable condition

08:06 What Etsy ran, and what it didn't

11:06 A diagnostic, not a proof; the −γᵀΔ bias

13:38 Results across 29 experiments

15:56 Caveat: covariates can drift

16:28 What's new, and related work

18:44 Recap and closing question

Sources:

• Variance reduction combining pre-experiment and in-experiment data (Lin & Crespo; CLeaR 2026, PMLR 323; arXiv 2410.09027)
• Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED (ByteDance, related, Jun 2026)
• CUPED on Steroids: Multivariate Covariate Adjustment for Switchback Experiments (related, Aug 2026)

Related episode: S2E1: Pinterest — A Cheap LLM Judge Plus Smarter Sampling Cuts Search-Experiment MDE

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.