This episode explores selective classification in deep neural networks: adding a post-hoc reject option so a trained model can abstain when its confidence falls below a calibrated threshold. It explains the key concepts of coverage, selective risk, and the risk-coverage tradeoff, arguing that a model should be judged not just by how often it is right, but by how often it chooses to answer. The discussion centers on the paper’s SGR method, which uses a held-out calibration set to choose a threshold that keeps selective risk below a target with high probability under an i.i.d. assumption, and compares softmax response with MC-dropout as confidence scores. Listeners would find it interesting because it gets at a practical question in AI deployment: not whether a model is always confident, but whether it can reliably know when to defer.

Sources:

1. Selective Classification for Deep Neural Networks — Yonatan Geifman, Ran El-Yaniv, 2017

http://arxiv.org/abs/1705.08500

2. SelectiveNet: A Deep Neural Network with an Integrated Reject Option — Yonatan Geifman, Ran El-Yaniv, 2019

http://arxiv.org/abs/1901.09192

3. On Optimum Recognition Error and Reject Tradeoff — C. K. Chow, 1970

https://scholar.google.com/scholar?q=On+Optimum+Recognition+Error+and+Reject+Tradeoff

4. Selective Classification for Deep Neural Networks — Yonatan Geifman and Ran El-Yaniv, 2017

https://scholar.google.com/scholar?q=Selective+Classification+for+Deep+Neural+Networks

5. SelectiveNet: A Deep Neural Network with an Integrated Reject Option — Yonatan Geifman and Ran El-Yaniv, 2019

https://scholar.google.com/scholar?q=SelectiveNet%3A+A+Deep+Neural+Network+with+an+Integrated+Reject+Option

6. Selective Classification via One-Sided Prediction — Aditya Gangrade, Anil Kag, and Venkatesh Saligrama, 2021

https://scholar.google.com/scholar?q=Selective+Classification+via+One-Sided+Prediction

7. Classification with Reject Option — Radu Herbei and Marten H. Wegkamp, 2006

https://scholar.google.com/scholar?q=Classification+with+Reject+Option

8. Learning with Rejection — Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri, 2016

https://scholar.google.com/scholar?q=Learning+with+Rejection

9. Boosting with Abstention — Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri, 2016

https://scholar.google.com/scholar?q=Boosting+with+Abstention

10. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — Yarin Gal and Zoubin Ghahramani, 2016

https://scholar.google.com/scholar?q=Dropout+as+a+Bayesian+Approximation%3A+Representing+Model+Uncertainty+in+Deep+Learning

11. On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger, 2017

https://scholar.google.com/scholar?q=On+Calibration+of+Modern+Neural+Networks

12. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles — Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, 2017

https://scholar.google.com/scholar?q=Simple+and+Scalable+Predictive+Uncertainty+Estimation+using+Deep+Ensembles

13. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift — Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek, 2019

https://scholar.google.com/scholar?q=Can+You+Trust+Your+Model%27s+Uncertainty%3F+Evaluating+Predictive+Uncertainty+Under+Dataset+Shift

14. A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators — Han Zhou, Jordy Van Landeghem, Teodora Popordanoska, and Matthew B. Blaschko, 2025

https://scholar.google.com/scholar?q=A+Novel+Characterization+of+the+Population+Area+Under+the+Risk+Coverage+Curve+%28AURC%29+and+Rates+of+Finite+Sample+Estimators

15. On the Foundations of Noise-Free Selective Classification — Ran El-Yaniv and Yair Wiener, 2010

https://scholar.google.com/scholar?q=On+the+Foundations+of+Noise-Free+Selective+Classification

16. Deep Residual Learning for Image Recognition — Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, 2016

https://scholar.google.com/scholar?q=Deep+Residual+Learning+for+Image+Recognition

17. Support Vector Machines with Embedded Reject Option — Giorgio Fumera and Fabio Roli, 2002

https://scholar.google.com/scholar?q=Support+Vector+Machines+with+Embedded+Reject+Option

18. A Deep Neural Network with an Integrated Reject Option — Yonatan Geifman and Ran El-Yaniv, 2019

https://scholar.google.com/scholar?q=A+Deep+Neural+Network+with+an+Integrated+Reject+Option

19. Augmenting the Softmax with Additional Confidence Scores for Improved Selective Classification with Out-of-Distribution Data — Guoxuan Xia, Christos-Savvas Bouganis, 2024

https://scholar.google.com/scholar?q=Augmenting+the+Softmax+with+Additional+Confidence+Scores+for+Improved+Selective+Classification+with+Out-of-Distribution+Data

20. GPify: Leveraging the Combined Strength of Normalizing Flow and Softmax For an Out-of-Distribution aware Confidence Score — Simon Kristoffersson Lind, Rudolph Triebel, Volker Kruger, 2026

https://scholar.google.com/scholar?q=GPify%3A+Leveraging+the+Combined+Strength+of+Normalizing+Flow+and+Softmax+For+an+Out-of-Distribution+aware+Confidence+Score

21. LogitAC: Logit Amplitude Constraints for Confidence Calibration and Out-of-Distribution Detection — Zongjing Cao, Yan Li, Byeong Seok Shin, 2024

https://scholar.google.com/scholar?q=LogitAC%3A+Logit+Amplitude+Constraints+for+Confidence+Calibration+and+Out-of-Distribution+Detection

22. Not all distributional shifts are equal: Fine-grained robust conformal inference — Jiahao Ai, Zhimei Ren, 2024

https://scholar.google.com/scholar?q=Not+all+distributional+shifts+are+equal%3A+Fine-grained+robust+conformal+inference

23. Wasserstein-regularized Conformal Prediction under General Distribution Shift — Rui Xu, Chao Chen, Yue Sun, Parvathinathan Venkitasubramaniam, Sihong Xie, 2025

https://scholar.google.com/scholar?q=Wasserstein-regularized+Conformal+Prediction+under+General+Distribution+Shift

Interactive Visualization: Selective Classification with Deep Neural Networks