Data Science in the Wild

Airbnb cut LLM evaluation from weeks to about a day. Four layers: name the noise, make evaluation deterministic, patch bugs with tiny LoRA micro-adapters, and test the seams between components.

Key takeaways:

• Name the noise first. About three-quarters of LLM-generated reference answers differed across labeling runs, and the judge drifted about 1% run to run. Separating fixable (epistemic) from inherent (aleatoric) noise tells you what to fix.
• Make evaluation deterministic. Caching results by sample and configuration means reruns stop regenerating noise and stay comparable.
• Patch, don't retrain. Micro-adapters are small LoRA patches (rank under 50) on a shared adapter, trained in under an hour on one GPU, shipped like hotfixes behind two gates and retired under lifecycle rules.
• Test the seams. End-to-end checks catch failures that pass every component test, like a correct retrieval misread by the LLM.
• Our take. It's classic software discipline applied to LLMs, and the judge must be calibrated before it's trusted.

Chapters:

00:00 A fix that takes weeks to verify

00:46 Airbnb's evaluation rebuild

02:17 The slow, noisy feedback loop

04:34 Layer 1: name the noise

06:51 Drifting references and judges

07:36 Layer 2: deterministic caching

09:55 Layer 3: LoRA micro-adapters

12:15 Governing the patches

13:46 Layer 4: testing the seams

14:31 Impact: weeks to a day

15:16 Our analysis

18:18 Recap and closing question

Sources:

• From weeks to a day: how we made LLM evaluation fast enough to iterate on (Baharak Saberidokht, Airbnb Engineering, Jul 2026)

Related episode: E15: Netflix — Grading the Graders: Can an AI Judge Your Synopsis?

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Airbnb's JourneyFormer replaces hundreds of hand-crafted guest statistics with a transformer that reads each guest's raw event history. In a three-week A/B test it added 0.55% more bookers in search and 5.04% more email views.

Key takeaways:

• Aggregates erase the story. Hand-built “what this guest likes” features lose order, recency and co-occurrence. A sequence model keeps them.
• Two timelines. A long-term sequence of milestones like bookings and a short-term sequence of recent activity (capped at 200 events), with location encoded through 14 levels of S2 cells.
• Built for serving cost. 128-dimension embeddings are shrunk to 32 before the guest profile is stored, and sparse computation skips empty sequence slots.
• Results. Offline NDCG rose 1.48%. Online, search saw +0.55% bookers, +0.82% booked nights and +0.90% views, and email ranking saw +5.04% views.
• Our take. The tools aren't new; the work is in taming long, noisy, real travel histories.

Chapters:

00:00 Guessing a friend's favorite movie

00:45 Airbnb's JourneyFormer

02:17 What aggregate features lose

03:49 Years of history at scale

05:20 S2 cells: encoding place

06:50 The transformer encoder

09:54 Shrinking embeddings for serving

11:25 Sparse computation

12:11 Results offline and online

14:30 What's genuinely new

17:33 Related work

18:19 Commentary and closing question

Sources:

• JourneyFormer: Encoding Airbnb Guest Journey with Sequence Modeling (Zha et al., Airbnb; arXiv, Jun 2026 (KDD '26))

Related episode: E1: LinkedIn — A Transformer That Ranks Your Feed

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Shopify's River is an AI agent that closes the gap between “a security patch was written” and “the vulnerability is gone in production.” It re-verifies fixes against the live repo, and security merges rose from about 10% to 80%.

Key takeaways:

• A patch isn't a fix. A PR that passed CI can go stale before merge. River distrusts tracker records and re-checks the live code before anything is marked resolved.
• It does the tedious work. River rebases, retriggers CI and repairs lock files. On one run, it opened four Slack threads covering 35 findings within a minute.
• It knows when to hand off. Judgment calls go to engineers with the evidence attached.
• Results. In the first 11 days of the dependency workflow, the open-issue backlog fell about 70%. Security merges through the freshness-gated queue went from about 10% to 80%.
• What's new. Not that an AI writes code, but the discipline of tying evidence to commits before closing anything.

Chapters:

00:00 When “done” isn't done

00:47 Shopify's River

01:32 The gap between a patch and a fix

03:48 Stale PRs and a growing backlog

05:19 River rechecks the live code

07:41 Rebasing, CI and lock files

09:59 Many fixes at once

10:46 Handing off to humans

12:17 Evidence before “resolved”

13:03 Results: backlog down 70%

15:21 Beyond security patching

16:06 What's actually new

18:23 Recap

Sources:

• How River takes security work from a fix to merge (Erin Son and Kaiyi Li, Shopify Engineering, Sep 2026)

Related episode: E2: Shopify — An AI Agent That Trains Itself

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Meta's SilverTorch turns recommendation retrieval into one GPU-native PyTorch model: ANN search, filtering and scoring that used to be separate CPU services. “Index as Model” reports up to 23.7× higher throughput.

Key takeaways:

• Collapse the stack. Retrieval normally chains an ANN index, feature filters and scoring models as separate services; every hop adds network and serialization cost.
• Index as model. SilverTorch expresses the index, filters and scorers as tensors and operators inside one trainable, servable model on GPUs.
• Two key kernels. A GPU Bloom index handles candidate filtering, and Int8-quantized embeddings with a fused ANN kernel make search fit in GPU memory with little accuracy loss.
• The payoff. Up to 23.7× more requests per second than a multi-service baseline, and a 13.35× cost-per-request advantage over the CPU-based approach with neural reranking.
• Not for everyone. The hosts' take: a GPU monolith fits Meta's scale; smaller teams may not need to tear out their microservices.

Chapters:

00:00 The plumbing behind your feed

00:46 Meta's “Index as Model” idea

02:20 The classic stack: ANN, filters, scoring

03:54 Why service hops hurt

05:26 What “index as model” means

07:45 A GPU Bloom filter

10:02 Int8 and the fused ANN kernel

12:17 Unified scoring layers

13:49 Results: 23.7× throughput

16:04 How novel is it?

17:39 Related work at Meta and Pinterest

19:56 Should everyone drop microservices?

22:13 Recap and closing question

Sources:

• SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems (Engineering at Meta, May 2026)

Related episode: E1: LinkedIn — A Transformer That Ranks Your Feed

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Google's WeatherNext 3 refreshes global AI weather forecasts every hour instead of every six by ingesting live satellite imagery, at up to 5 km resolution. We cover who that gap costs, how the model works, and what the precipitation numbers really mean.

Key takeaways:

• Hourly, not six-hourly. WeatherNext 3 ingests live hourly geostationary satellite mosaics, so each new forecast starts from fresh observations. WeatherNext 2 ran on a 25 km grid in 6-hour steps.
• One generative model. A single Functional Generative Network (FGN) mesh transformer outputs a distribution of possible futures rather than one deterministic forecast.
• Sharper maps. Surface temperature and moisture come at 5 km, roughly five times sharper than the previous model.
• Read the rain stat carefully. Google reports up to 60% better precipitation scores (CRPS) than baselines when graded against NASA's IMERG data, at early lead times. IMERG is the yardstick, not the rival.
• Open questions. The announcement gives no parameter counts, so it's hard to separate architecture gains from scale.

Chapters:

00:00 Your weather app's six-hour-old forecast

01:33 Why a six-hour gap is costly

04:38 Live satellite data, every hour

06:11 One mesh transformer for everything

07:42 Outputs and 5 km resolution

09:14 Precipitation and IMERG

10:45 Who benefits

11:31 How novel is it?

12:17 Open questions

13:02 GraphCast and other weather models

14:33 Closing thought

Sources:

• Introducing WeatherNext 3, our most advanced and accurate global weather AI model (Google DeepMind and Google Research, Sep 2026)

Related episode: E11: NVIDIA — One Checkpoint, Three Jobs for Robots

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Instacart's generative Shopping Hub: instead of a static, hand-built content library, an LLM pipeline designs each shopper's page top-down, grounds it in the real catalog with RAG, and checks it with LLM judges and a cheap fine-tuned classifier.

Key takeaways:

• Themes first, products second. An agent uses purchase history to design personalized page themes; products are found afterward. Asking an LLM to pick items directly runs into latency and a catalog that changes by the minute.
• Ground it in the catalog. Retrieval keywords are restricted to real catalog terms via embedding search, cutting generation cost 15–20% per generation.
• Cheap judges at scale. LLM judges audit cohesion, diversity and brand fit, while a fine-tuned DeBERTa relevance classifier costs over 99% less than closed-weight LLM inference, so it can filter every placement.
• Keep what works. Generated placements feed Instacart's existing ranking systems rather than replacing them.
• Results are early. Offline evaluations beat the baseline, but the post calls A/B results only “quite promising,” with no lift numbers.

Chapters:

00:00 A store that rebuilds itself for you

01:32 Instacart's generative Shopping Hub

02:17 Why static placements stopped scaling

05:18 Why not let the LLM pick products?

07:34 Phase 1: an agent designs page themes

08:20 Phase 2: catalog-grounded keywords

10:35 Phase 3: LLM judges and a cheap classifier

13:38 Phase 4: existing ranking systems

15:55 What the results do and don't show

16:41 How Netflix and others compare

20:31 Recap and closing question

Sources:

• Our Early Journey to Transform Instacart's Discovery Recommendations with LLMs (Moein Hasani et al., Instacart Tech, Feb 2026)

Related episode: E12: Netflix — One Model, One Pass, Your Whole Homepage

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

Shopify's Sidekick continual learning loop: a calibrated LLM judge, validated against A/B tests, becomes the reward for fine-tuning a smaller model. For a GraphQL agent at 2,000 requests/min, Shopify estimates serving cost falls from ~$27M to ~$1M a year.

Key takeaways:

• Calibrate humans first. Two expert annotators blind-score samples against a rubric; their agreement (Cohen's kappa) sets the ceiling for any automated judge.
• Then calibrate the judge. An LLM judge is tuned with DSPy and reflection-based optimizers (GEPA, Agentic Context Engineering) and trusted only once it tracks real A/B outcomes.
• Turn failures into weights. Production failures feed SFT plus GRPO fine-tuning, with the calibrated judge as the reward.
• The payoff is an estimate. Shopify estimates frontier-model serving at ~$27M a year versus closer to $1M for the fine-tuned model, a 96% cut. Gisting also compresses the long system prompt.
• The lesson. A judge used as a reward gets gamed unless it's anchored to real user outcomes first.

Chapters:

00:00 A $27M bill and a $1M fix

01:01 Shopify's Sidekick continual learning loop

02:03 The frontier-model trap

04:04 Stage 1: a human-calibrated rubric

06:07 Stage 2: an LLM judge tuned with DSPy

09:14 Validating the judge against A/B tests

10:15 Stage 3: automated prompt and tool research

11:16 Stage 4: SFT and GRPO fine-tuning

13:17 The cost math

14:19 What's new: judge before reward

17:22 Gisting and task routing

20:24 Recap and closing question

Sources:

• Sidekick's continual learning loop (Andrew McNamara and Cody Mazza-Anthony, Shopify Engineering, Aug 2026)

Related episode: E6: Shopify — An AI Agent That Verifies Its Own Work

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.

LinkedIn feed ranking with a transformer: Feed SR reads up to 1,000 of a member's past impressions in order and replaced the DCNv2 ranker for 1.2 billion members, lifting time spent 2.10% and likes, comments and reshares 3.52%.

Key takeaways:

• Sequence beats snapshot. Feed SR is a decoder-only transformer over a member's chronological impression history (up to 1,000 items), something the feature-cross DCNv2 ranker couldn't model.
• Fewer features, not more. Feed SR uses about 80% fewer features than the old ranker, and the removed ones added no AUC.
• Serving was the hard part. A CPU/GPU split, history parsing cut from 450 ms to 2 ms (225×), and shared-context batching that scores many candidates against one encoded history made it fit latency budgets.
• Bigger model, less energy. Per item, Feed SR uses about 0.2× the training energy and 0.7× the inference energy of the old system.
• Honest negatives. LinkedIn also tried LLM-based rankers and HSTU layers; HSTU lost 0.23% at matched compute.

Chapters:

00:00 Why order matters: guessing a favorite movie

01:01 Feed SR and LinkedIn's 1.2B-member feed

02:01 The old DCNv2 ranker and its blind spot

04:02 Why replace a ranker that works?

06:08 RoPE and 1,000-item histories

08:08 Cutting about 80% of features

09:08 CPU/GPU serving and custom kernels

12:09 Bigger model, less energy

13:10 Shared-context batching

15:13 What didn't work: LLM rankers and HSTU

18:15 Where the gains landed

20:15 Recap and closing question

Sources:

• An Industrial-Scale Sequential Recommender for LinkedIn Feed Ranking (Hertel et al., LinkedIn; arXiv, Feb 2026)

Related episode: E7: Airbnb — A Transformer That Reads Your Whole Travel History

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.