Data Science in the Wild

Do late covariates hide pre-trends? Arora (Ashoka) finds 77 of 90 DiD studies date a covariate too late—and honest bands can cover in only 5% of Monte Carlos.

Key takeaways:

• Late dating is common. Of 90 recent published DiD/event-study specs that adjust for time-varying covariates, 77 date at least one covariate at the end of the pre-period or the same year as the outcome; in 71 of 77 an earlier date exists.
• Shock terms flatten placebos. A late covariate records pre-period outcome shocks, so the adjusted placebo splits into a trend term and a shock term (Prop. 1–2). Shock terms pull placebos toward zero relative to the post trend; Rambachan & Roth (2023) relative-magnitude sets then come out too narrow. Callaway & Sant’Anna (2021) did v2.1.2 defaults to start-of-window dating.
• China shock & telephone automation. Autor et al. (2013): 1975 covariates give largest placebo slope 0.36 / most-recent 0.12; 1990 gives 0.19 / −0.02 (M∗ 0.55→0.60). Feigenbaum & Gross (2024): census-year placebos −0.13 / −0.06 vs 1910 −0.31 / −0.20 (M∗ 2.15 vs 0.70 — “three times as robust”).
• Monte Carlo & tools. Linear design n=700, true trend δ=0.3: late dating shrinks the placebo slope to 0.04; the M=1 honest band covers in 5% of replications (target 95%). Fix: Wald test of late vs early (Prop. 5), identify shock terms in placebo units (Prop. 6), or bound with correlation ρ̄ (Prop. 7; at ρ̄=0.1 recovers ~0.34 from 1990 China data alone).
• Our take (Medicaid). Baker et al. (2025): dating all covariates in 2009 moves the average effect 0.46→4.49; unemployment alone in 2009 gives 3.72. Pair the Wald with shock-term size—female/Hispanic shares fail the test but shocks are tiny (0.2–0.6 vs placebo ~2.9). Survey prevalence ≠ distortion severity.

Chapters:

00:00 Cold open: when pre-trend tests look too good

01:31 DiD: parallel trends, placebos, covariates

02:51 Survey: 77 of 90 date a covariate too late

03:51 Mechanism: late covariates absorb pre-shocks

05:30 Breakdown values and the did package default

06:56 China shock: 1975 vs 1990 covariates

08:25 Telephone automation: 3× more "robust"

09:48 Monte Carlo and three diagnostic tools

11:48 Novelty: estimate intact, benchmark broken

13:51 Our take: Medicaid Wald vs shock size

16:16 Recap and closing question

Sources:

• When Covariates Mask Pre-Trends (Parush Arora (Ashoka University); arXiv, Nov 2, 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Do recommenders use your latest search? Huawei's FocalSwap audit of 66,139 rows finds the credit flips with the control: masking credits search, matched swaps don't.

Key takeaways:

• The control decides the credit. On the same 66,139 rows across three UniSAR checkpoints, swapping all search inputs for a neutral ID gives a positive contribution (+2.0 to +5.4 ×10⁻³ reciprocal rank). Category/profile-matched real searches give −1.18 to −0.35 ×10⁻³, below zero for one checkpoint. The paired gap (+2.8 to +6.6 ×10⁻³) is above zero every time and holds across five alternative donor assignments.
• Latest-search credit isn't established. Changing only the latest search slot gives +0.138 ×10⁻³ for SESRec and −0.213 to +0.089 ×10⁻³ for UniSAR. Every interval crosses zero, under matched donors too. That's “not established”, not proof of zero.
• Moving ranks isn't the same as helping. Neutral masks change 59.7–61.7% of target ranks and matched swaps 44.1–44.8%, but only masking yields a positive mean.
• The audit does catch real signals. On GenSAR's current-query task (8,958 rows), masking the query cuts RR by 0.035 and a matched donor query by 0.106. With Qilin's text, BM25 shows a focal-query edge (+0.084 over placebo) only on a 6.1% high-overlap subset.
• Our take. Pair masks with matched replacements before crediting search. The paper doesn't say whether masks look better because donors keep useful coarse info or because a neutral ID disrupts the model; results cover only the audited checkpoints.

Chapters:

00:00 Cold open: did your search drive that pick?

00:53 Welcome: which search input earns the lift?

02:28 Why attribution matters for platforms

03:31 Headline: the control changes the answer

04:25 FocalSwap: fixed rows, two scopes, two controls

07:41 Reciprocal rank, KuaiSAR, Qilin, checkpoints

09:35 Calibration: planted signal and GenSAR

10:19 All-source results on UniSAR

12:57 Latest-slot results and Qilin text evidence

14:28 Why masking can mislead; related work

17:26 Our take, caveats and recap

Sources:

• Do Search-Augmented Recommenders Use the Latest Search Episode? (Jiandong Ding (Huawei); arXiv, Nov 4, 2026; WSDM '27)
• FocalSwap code (GitHub)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Do LLM recommenders pick by position? A University of Granada study ran 40,000 shuffled prompts through 8 open models, and every one failed both statistical tests for position independence.

Key takeaways:

• Order matters for every model. On a RAG testbed (50,000 Amazon products, 2,000 queries × 20 orderings of the top-5 candidates), all 8 models (LLaMA 3.1 8B/70B, LLaMA 4 Scout, GPT-OSS 20B/120B, three DeepSeek R1 distills) rejected position independence under both tests, POB-ICM (chi-square on a contingency matrix) and POB-UNBP (a uniformity test).
• Reordering flips picks often. Permutation sensitivity ran from 15.5% (GPT-OSS 120B) to 34.0% (DeepSeek R1 7B); positional bias, the distance from uniform, from 2.4% to 5.9%.
• Soft vs. strong bias. Strong positional bias, when position pushes the pick outside the comparably relevant items, ranged from 1.0% to 9.8%. The paper treats it as an upper bound: for DeepSeek R1 7B, decoding noise alone gives 9.1%.
• Shape depends on the model, and grows with list size. GPT-OSS and LLaMA 3.1 70B favor early slots, LLaMA 4 Scout under-picks slot 1. With 20 candidates, Scout's positional bias rises from 2.9% to 33.8%; raising temperature doesn't remove it.
• Our caveats. One catalog, one fixed (untuned) prompt template, single-turn top-1 recommendation. The authors suggest routine audits and order randomization.

Chapters:

00:00 Cold open: would an AI pick by position?

00:41 Welcome: why prompt order matters

02:01 Setup: LLM recommenders on top of RAG

04:27 Testbed: Amazon catalog, 8 LLMs, 40K runs

06:22 Method: POB-ICM vs. POB-UNBP

09:36 Results: sensitivity, bias, strong bias

13:11 Which positions each model favors

14:26 Robustness: list size and temperature

15:52 Novelty and related work

18:26 Our take, caveats and recap

Sources:

• Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence (Peláez-González, Herrera, Herrera-Poyatos (University of Granada); arXiv, Oct 27, 2026)
• Code (GitHub)
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) (arXiv)
• Large Language Models are Zero-Shot Rankers for Recommender Systems (Hou et al., 2023) (arXiv)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Should you recalibrate CTR models on fresh data? Rui Liu's Avazu study: a 6-hour Platt/isotonic refit made calibration worse in about a third of applications.

Key takeaways:

• Random splits flatter calibration. On Avazu (40.4M impressions; LR, DeepFM, DCN-v2), a random split reported LR's ECE at 55–62% of its value on the nearest future day in every seed. For the deep models the ratio was seed-unstable, and no model's ECE grew steadily with distance from training.
• Recalibration can backfire. Platt or isotonic refits on the preceding 6 hours removed up to 61% of ECE in badly miscalibrated cells, but increased it in 14–17 of 48 applications. Worst case: LR under Platt, 0.0054 to 0.0112 (+107%).
• Span beats volume. A 12-hour slice cut harm from 13–17 to 3–5 of 36 applications; 24 hours raised it again. A 93K-impression subsample spread over those 12 hours (5% of the volume, 20–25× faster to fit) recovered 88% of the gain. Why span helps (hour mix, age, distinct hours) isn't separated yet.
• No easy gate, small profit effects. Gating on the slice's ECE showed no benefit out-of-origin. In a simulated second-price auction, recalibration moved profit by −0.21% to +0.26%.
• Our caveats. One 2014 dataset, two overlapping origins, and a fixed value per click of 1.0.

Chapters:

00:00 Cold open: bidding on miscalibrated probabilities

01:42 Why calibration is what ad systems consume

04:38 Setup: Avazu, three models, random vs. temporal

08:11 RQ1: random splits flatter calibration

09:46 RQ2: no steady drift with distance

10:26 RQ3: recalibration harms a third of the time

11:49 Slice-length sweep: span beats volume

13:17 Can a pre-serving gate prevent harm?

14:07 RQ4: profit in a simulated auction

15:26 Novelty and related work

17:40 Our take and caveats

19:19 Recap and closing question

Sources:

• How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration (Rui Liu (independent researcher); arXiv, Oct 23, 2026)
• Code and per-cell results (GitHub)
• Field-aware Calibration (Pan et al., WWW 2020) (arXiv)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Can you ship an LLM change on evals alone? Spotify's Mårten Schultzberg: only if eval effects predict A/B results. On 183 Upworthy tests, his gate ships none.

Key takeaways:

• The question. Judge–human agreement doesn't tell you which changes can skip an A/B test. That needs trial-level surrogacy: eval treatment effects that predict online effects across a class of interventions. It's logically independent of unit-level (Prentice) surrogacy.
• The gate. Subtracting each intervention's reported noise variances from the covariance undoes attenuation (a random-effects correction dating to DerSimonian and Laird). A bivariate-normal model then gives a closed-form ship/test/kill rule. In the running example (R² 0.6, α 0.1) it ships above +2.94 pp and kills below −3.02 pp: 14% ship, 6% kill.
• Judges and audits. A judge 3 pp less sensitive to treated outputs turns a +2.0 pp gain into −0.25 pp. In a simulated reward-hacking break, the unaudited gate's false-ship rate hit 0.31 vs. a promised 0.10; a 0.4 audit rate caught it in 96% of runs.
• Upworthy. A GPT-4o-mini judge rated headline informativeness; the outcome was click-through. Measurement was precise (reliability 0.93 and 0.74), but corrected pooled R² was 0.023 and the ship threshold 134 pp, above the 100 pp maximum, so all 183 went to A/B tests.
• Our take. Fix intervention classes before estimating: two classes with zero surrogacy can pool to 0.81.

Chapters:

00:00 Cold open: can an LLM judge replace A/B tests?

00:46 The paper, and why judge agreement isn't enough

03:25 Interventions as the unit; noise and attenuation

05:38 Moment correction and the ship/test/kill gate

06:56 Upworthy: 183 headline tests

09:29 Fail-closed, and when the gate pays off

10:52 Arm-dependent judge error

12:09 The closed loop and randomized audits

13:54 Trial-level vs. Prentice surrogacy

15:19 Related work

16:36 Our take: don't pool intervention classes

18:27 Recap and closing question

Sources:

• When Can You Ship on Evals Alone? Trial-Level Surrogacy for Offline Evaluation of LLM Systems (Mårten Schultzberg (Spotify); arXiv, Oct 7, 2026)
• Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference (Persson, Schultzberg, Ankargren; arXiv 2026)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Is difference-in-differences broken? Brown economist Peter Hull used AI agents to replicate 1,346 staggered-adoption results from 108 papers: TWFE and modern DiD estimates correlate about 0.9, and significant sign flips are rare.

Key takeaways:

• The worry. In staggered rollouts, two-way fixed effects (TWFE) can put negative weights on some treated observations, so a policy that helps everyone could in theory look harmful.
• The test. AI agents (OpenAI Codex) screened 16,094 records from ten economics journals, with the author overseeing. TWFE was compared with DiD-averaging (Callaway and Sant'Anna) and imputation (Borusyak, Jaravel and Spiess).
• Results. Correlations are 0.88 and 0.93. Signs flip in 14.5% and 10.8% of pairs, but only 3.5% and 2.2% flip with a significant difference. Negative weights carry a median 1.4% of TWFE's weight; modern standard errors are 47% and 14% larger.
• Noise removed. A hierarchical model gives 0.91–0.98 estimand correlations and a 2.1% and 0.6% chance of a significant wrong-signed TWFE estimate. More always-treated units mean more flips.
• Our caveats. The paper doesn't say these rates justify skipping modern estimators, and parallel-trends failures aren't tested.

Chapters:

00:00 Cold open: can DiD flip the sign?

00:55 The paper: Peter Hull's meta-analysis

02:03 Staggered adoption and negative weights

02:49 Analogy: a staggered software rollout

07:01 AI agents build the replication sample

09:44 Results: correlations and sign flips

11:51 Why flips are rare: tiny negative weights

13:02 The precision tradeoff

14:18 What predicts a flip

16:04 Hierarchical model: estimates vs. estimands

18:05 What's new and related work

21:59 Recap and closing question

Sources:

• How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis (Peter Hull (Brown University); arXiv, Oct 29, 2026)
• Difference-in-Differences with Multiple Time Periods (Callaway and Sant'Anna; J. Econometrics 2021)
• Revisiting Event Study Designs: Robust and Efficient Estimation (Borusyak, Jaravel, Spiess; REStud 2024)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

LCM-Promo (Prosus and iFood) turns a 4B-parameter LLM into an uplift model for discounts. In a one-month A/B test on churned users it lifted conversion 1% and cut promotion cost 6% vs. the old policy.

Key takeaways:

• Discounts as tokens. Each prompt holds session context (day, lunch or dinner shift), an LCM-generated user summary, and discount options mapped to letters. The model predicts one letter, or a no-purchase token, read as the smallest discount that would make the user order.
• Monotonic by construction. Summing those probabilities up the discount ladder gives a response curve that never falls as the discount grows, and the no-purchase mass keeps it below 100%.
• Online result. 90% control on Segment-EMA, 10% on LCM-Promo, same budget and same allocator: +1% relative conversion and 6% lower promotion cost. Allocation grew from about 50 segments to over 1,000 clusters and from 10 wallet configurations to over a million combinations; it now serves all churn users.
• Offline check. Against an uplift random forest (causalml) on RCT data, LCM-Promo was comparable on AUUC (0.0608 vs 0.0578) and Qini (0.0490 vs 0.0466) and lower on Uplift@30 (0.1626 vs 0.1708). The waterfall-label variant did much worse (AUUC 0.0278).
• Our caveats. Results cover one lifecycle stage on one platform. Our take: uplift margins may be thinner for high-intent daily users, and precomputed user summaries can't react to mid-session changes.

Chapters:

00:00 Cold open: the pizza and the 10% coupon

01:00 The paper: LCM-Promo from Prosus and iFood

02:29 Headline: one-month churn-reactivation A/B test

03:21 Large Commerce Models and the 4B base

04:49 Discounts as letters, one-token prediction

05:50 Conversion thresholds and monotonic curves

07:21 Offline test vs. uplift random forest

08:46 The waterfall labels that failed

10:08 Online A/B test design and results

11:16 From 50 segments to 1,000+ clusters

12:30 Serving: vLLM, BESH, 8 H200s

13:31 What's genuinely new

14:29 Our caveats

15:56 Recap and closing question

Sources:

• LCM-Promo: An LLM-Based Uplift Model for Promotion Allocation (Midavaine, Fok, Mendes, Silva, Akaçık, Lima, Genç (Prosus, iFood); arXiv, Oct 27, 2026; ECML PKDD 2026)
• CausalML: Python package for causal machine learning (Chen et al.; arXiv 2020)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Phantom correlation (Favier et al., University of Antwerp): user activity alone makes items look similar. On MovieLens-1M, an activity-only model removes 81% of the independence model's co-occurrence mismatch.

Key takeaways:

• The toy example. 100 equally liked films; half the users watch each film with probability 0.01, half with 0.20, all independently. Lift still says every pair co-occurs about 1.82 times more often than chance.
• The model. Three assumptions (independent consumption, more activity raises every item, activity is the only difference) yield exactly one model, the Rasch model: activity multiplies every item's odds by the same factor, so every pair is positively associated.
• Results. Over all pairs, activity removes 81% (MovieLens-1M), 73% (MovieLens-20M) and 50% (Amazon Books) of the independence model's systematic mismatch, and 96–99% for lift, cosine and Pearson over popularity blocks on MovieLens. Last.fm, capped at each user's top 50 artists, shows none.
• Measure choice matters. Cosine inherits a baseline that grows with popularity, a head start for popular items; lift and PMI inherit one that is largest for rare items.
• Limits and our take. The authors don't call collaborative filtering fake: taste lives in which pairs stand out above the baseline. On Amazon Books, heavy readers drift to niche books, a second dimension the model leaves out. Our take: check item-item scores against an activity baseline before reading them as taste.

Chapters:

00:00 Cold open: footprints that look like company

01:35 The paper: Favier et al., University of Antwerp

02:29 Item-item similarity and implicit feedback

03:59 Toy example: 100 films, lift 1.82, no taste

05:18 Four similarity measures, EASE, independence

06:21 The activity model: three conditions, Rasch

08:36 Four datasets and Poisson deviance

09:11 Results: MovieLens-1M and 20M

10:32 Amazon Books and capped Last.fm

12:16 What's new and related work

14:54 Our take, and the authors' caveats

17:05 Recap and closing question

Sources:

• Do Similarity Measures Measure Similarity? How User Activity Creates Phantom Correlation in Recommender Systems (Favier, Daniëls, Leysen, Marinus, Goethals; arXiv, Oct 26, 2026)
• Embarrassingly Shallow Autoencoders for Sparse Data (EASE) (Steck; WWW 2019)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Volatility-Corrected Prediction Intervals (Cho & Matteson, Cornell): time-series foundation models can hit 80% coverage on average yet miss when volatility rises. A GARCH correction on their residuals steadies local coverage without widening intervals.

Key takeaways:

• The failure mode. Marginal coverage can sit at 80% while local coverage swings between over- and undercoverage. The paper calls this local scale miscalibration and measures it with RMSCE over rolling 100-forecast windows.
• How VCPI works. It standardizes the residuals by the forecaster's own predictive scale, fits a GARCH-family model to them (variant and error distribution chosen on a validation segment), and applies a multiplicative correction to future intervals.
• Results (80% intervals, horizons 1–4). Chronos on Capital Bikeshare: RMSCE 0.051 vs 0.100 (ACI), 0.096 (QT) and 0.079 (native). TimesFM on Colorado River streamflow: 0.062 vs 0.187 (QT) and 0.192 (ACI), with a median width of 761 vs 1,411 (QT) and 872 (native).
• Why it reacts faster. ACI and QT update on whether the last interval missed; GARCH uses the size of the error. The paper proves that after a large volatility jump, the first GARCH update lands closer to the oracle interval than quantile tracking.
• Limits and our take. Misspecified models carry bias, multi-step GARCH forecasts mean-revert beyond 4–8 steps, and 95% tails are more sensitive. Slight overcoverage is a fair price for stable local coverage.

Chapters:

00:00 Cold open: a forecast blind to its own uncertainty

00:59 The paper: Cho and Matteson, Cornell

02:07 Local scale miscalibration vs marginal coverage

04:21 Foundation models: Chronos, Moirai, TimesFM

05:08 RMSCE: measuring local coverage

06:40 How VCPI works in four steps

07:42 Which GARCH model, chosen on validation data

09:02 Baselines and four datasets

10:09 Results: Chronos on bike rentals

11:32 Results: TimesFM on Colorado River

13:20 Error size vs a binary miss

15:08 Econometrics meets conformal prediction

16:30 Limitations, our take and recap

Sources:

• Volatility-Corrected Prediction Intervals (Cho, Matteson, Cornell; arXiv, Oct 22, 2026)
• Conformal PID Control for Time Series Prediction (Angelopoulos et al.; NeurIPS 2023)
• Adaptive Conformal Inference Under Distribution Shift (Gibbs, Candès; 2021)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng & Wang, Purdue): every prompt edit or model swap gives an AI analyst another chance at a false positive. Continual validation tests each update on simulated null data before it ships.

Key takeaways:

• The risk accumulates. Over 20 runs of 10 updates proposed by Claude Sonnet 5.5, the worst deployed version without validation reached a Type I error of 0.098 (bioequivalence) and 0.183 (A/B tests), against a 0.05 target.
• Continual validation. Each frozen update is tested on fresh data simulated under the null before deployment. With α = 0.05, δ = 0.005 and α′ = 0.0452, the chance of ever deploying a version above α′ in a tested null scenario is at most δ. Worst deployed error fell to at most 0.045 and 0.044.
• Off-grid check. Bioequivalence versions stayed at or below 0.05 in at least 666 of 690 off-grid scenarios (max 0.058); A/B versions stayed at or below 0.044 in all 116.
• Modest cost. With fixed, code-computed analysis paths, validation needs no LLM calls (Prop. 2). Median one-GPU time: 1,101 s for ~447M simulated studies, 329 s for ~29M A/B experiments.
• Power. Validated analysts kept 82% and 96% of an oracle's power. Absolute power was 0.55 vs 0.67 (bioequivalence) and 0.20 vs 0.21 for A/B features with 1–3% relative lifts. Some power is given up.
• Our take. The guarantee covers only the simulated null scenarios, so the work shifts to keeping those simulations realistic.

Chapters:

00:00 Cold open: an AI analyst that rubber-stamps

01:28 Why an evolving analyst accumulates Type I error

02:52 Without validation: 0.098 and 0.183

04:17 Continual validation and null-simulated data

05:56 Alpha, delta and alpha prime

06:40 Results with validation

07:13 Off-grid scenarios

08:21 Cost: no LLM calls, minutes on one GPU

11:14 Power vs an oracle

13:41 Classical tests vs lifetime guarantees

14:35 Our take: simulations must match reality

15:44 Recap and closing question

Sources:

• Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng, Wang, Purdue; arXiv, Oct 22, 2026)
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control (Angelopoulos et al.; arXiv, 2021)

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.