Sign up to save your podcasts
Or
Share [Linkpost] “Histograms are to CDFs as calibration plots are to...” by Optimization Process
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
[Linkpost] “Histograms are to CDFs as calibration plots are to...” by Optimization Process
This is a link post.
As you know, histograms are decent visualizations for PDFs with lots of samples...
10k predictions, 20 bins
...but if there are only a few samples, the histogram-binning choices can matter a lot:
10 predictions, 4 binssame 10 predictions, 7 binsThe binning (a) discards information, and worse, (b) is mathematically un-aesthetic.
But a CDF doesn't have this problem!
same 10 predictions, every data point precisely representedIf you make a bunch of predictions, and you want to know how well they're calibrated, classically you make a graph like this:
source: SSC's 2019 prediction gradingBut, as with a histogram, this depends on how you bin your predictions.
100 predictions, 10 binssame 100 predictions, 30 binsIs there some CDF-like equivalent here? Some visualization with no free parameters?
I asked that question to several people at Arbor Summer Camp. I got three answers:
- "You get from a PDF to [...]
---
First published:
June 5th, 2025
Source:
https://www.lesswrong.com/posts/LFGgwitjertJqch7J/histograms-are-to-cdfs-as-calibration-plots-are-to
Linkpost URL:
https://optimizationprocess.com/calibration-cdf/
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
View all episodes
By LessWrongThis is a link post.
As you know, histograms are decent visualizations for PDFs with lots of samples...
10k predictions, 20 bins
...but if there are only a few samples, the histogram-binning choices can matter a lot:
10 predictions, 4 binssame 10 predictions, 7 binsThe binning (a) discards information, and worse, (b) is mathematically un-aesthetic.
But a CDF doesn't have this problem!
same 10 predictions, every data point precisely representedIf you make a bunch of predictions, and you want to know how well they're calibrated, classically you make a graph like this:
source: SSC's 2019 prediction gradingBut, as with a histogram, this depends on how you bin your predictions.
100 predictions, 10 binssame 100 predictions, 30 binsIs there some CDF-like equivalent here? Some visualization with no free parameters?
I asked that question to several people at Arbor Summer Camp. I got three answers:
- "You get from a PDF to [...]
---
First published:
June 5th, 2025
Source:
https://www.lesswrong.com/posts/LFGgwitjertJqch7J/histograms-are-to-cdfs-as-calibration-plots-are-to
Linkpost URL:
https://optimizationprocess.com/calibration-cdf/
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,469 Listeners
Conversations with Tyler
2,395 Listeners
The Peter Attia Drive
7,953 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,142 Listeners
ManifoldOne
89 Listeners
Your Undivided Attention
1,472 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,207 Listeners
Machine Learning Street Talk (MLST)
88 Listeners
Dwarkesh Podcast
426 Listeners
Hard Fork
5,461 Listeners
The Ezra Klein Show
15,321 Listeners
Moonshots with Peter Diamandis
482 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
461 Listeners