Sign up to save your podcasts
Or
Share Do Language Models Know Their Limits
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Do Language Models Know Their Limits
This episode explores whether large language models can genuinely recognize the limits of their own knowledge or whether they have simply learned to sound uncertain in socially acceptable ways. It examines the paper’s idea of “self-knowledge” through the lens of confidence calibration, including the dangerous case where a model does not know an answer but responds with unwarranted confidence. The discussion walks through the SelfAware benchmark, explaining how it pairs unanswerable questions with semantically similar answerable ones and why that design is both insightful and methodologically slippery. Listeners would find it interesting because it gets past simple accuracy scores and asks a more consequential question for AI safety and product reliability: when a model says “I don’t know,” is that real judgment or just polished behavior?
Sources:
1. Do Large Language Models Know What They Don't Know? — Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang, 2023
http://arxiv.org/abs/2305.18153
2. Finetuned Language Models Are Zero-Shot Learners — Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, Quoc V. Le, 2021
https://scholar.google.com/scholar?q=Finetuned+Language+Models+Are+Zero-Shot+Learners
3. Multitask Prompted Training Enables Zero-Shot Task Generalization — Victor Sanh, Albert Webson, Colin Raffel and many coauthors, 2021
https://scholar.google.com/scholar?q=Multitask+Prompted+Training+Enables+Zero-Shot+Task+Generalization
4. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida and many coauthors, 2022
https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback
5. Scaling Instruction-Finetuned Language Models — Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, Xuezhi Wang, Denny Zhou, Quoc V. Le, Jason Wei and many coauthors, 2022
https://scholar.google.com/scholar?q=Scaling+Instruction-Finetuned+Language+Models
6. Self-Instruct: Aligning Language Models with Self-Generated Instructions — Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi, 2023
https://scholar.google.com/scholar?q=Self-Instruct%3A+Aligning+Language+Models+with+Self-Generated+Instructions
7. Measuring and Improving Factuality in Large Language Models with Calibrated Confidence Scores — Saurav Kadavath, Eric Wallace, Luyu Gao, et al., 2022
https://scholar.google.com/scholar?q=Measuring+and+Improving+Factuality+in+Large+Language+Models+with+Calibrated+Confidence+Scores
8. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models — Aarohi Srivastava, Jos Rozen, Francesco Tintarev, et al., 2022
https://scholar.google.com/scholar?q=Beyond+the+Imitation+Game%3A+Quantifying+and+extrapolating+the+capabilities+of+language+models
9. Teaching Small Language Models to Reason — Jason Wei, Xuezhi Wang, Dale Schuurmans, et al., 2022
https://scholar.google.com/scholar?q=Teaching+Small+Language+Models+to+Reason
10. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, et al., 2022
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
11. SimCSE: Simple Contrastive Learning of Sentence Embeddings — Tianyu Gao, Xingcheng Yao, Danqi Chen, 2021
https://scholar.google.com/scholar?q=SimCSE%3A+Simple+Contrastive+Learning+of+Sentence+Embeddings
12. SQuAD 2.0: The Stanford Question Answering Dataset — Pranav Rajpurkar, Robin Jia, Percy Liang, 2018
https://scholar.google.com/scholar?q=SQuAD+2.0%3A+The+Stanford+Question+Answering+Dataset
13. Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence — Sophia Hager et al., 2025
https://scholar.google.com/scholar?q=Uncertainty+Distillation%3A+Teaching+Language+Models+to+Express+Semantic+Confidence
14. Large Language Model Uncertainty Measurement and Calibration for Medical Diagnosis and Treatment — Thomas Savage et al., 2024
https://scholar.google.com/scholar?q=Large+Language+Model+Uncertainty+Measurement+and+Calibration+for+Medical+Diagnosis+and+Treatment
15. Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey — Xiaoou Liu et al., 2025
https://scholar.google.com/scholar?q=Uncertainty+Quantification+and+Confidence+Calibration+in+Large+Language+Models%3A+A+Survey
16. Unanswerability Evaluation for Retrieval Augmented Generation — Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu, 2025
https://scholar.google.com/scholar?q=Unanswerability+Evaluation+for+Retrieval+Augmented+Generation
17. Answerability in Retrieval-Augmented Open-Domain Question Answering — Rustam Abdumalikov, Pasquale Minervini, Yova Kementchedjhieva, 2024
https://scholar.google.com/scholar?q=Answerability+in+Retrieval-Augmented+Open-Domain+Question+Answering
18. The Art of Saying No: Contextual Noncompliance in Language Models — Faeze Brahman et al., 2024
https://scholar.google.com/scholar?q=The+Art+of+Saying+No%3A+Contextual+Noncompliance+in+Language+Models
19. AI Post Transformers: Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Model — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/hallucination-to-truth-a-review-of-fact-checking-and-factuality-evaluation-in-la/
20. AI Post Transformers: Internal Safety Collapse in Frontier LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-internal-safety-collapse-in-frontier-llm-8be72f.mp3
21. AI Post Transformers: Self-Search Reinforcement Learning for LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/self-search-reinforcement-learning-for-llms/
22. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp3
Interactive Visualization: Do Language Models Know Their Limits
View all episodes
By mcgrofThis episode explores whether large language models can genuinely recognize the limits of their own knowledge or whether they have simply learned to sound uncertain in socially acceptable ways. It examines the paper’s idea of “self-knowledge” through the lens of confidence calibration, including the dangerous case where a model does not know an answer but responds with unwarranted confidence. The discussion walks through the SelfAware benchmark, explaining how it pairs unanswerable questions with semantically similar answerable ones and why that design is both insightful and methodologically slippery. Listeners would find it interesting because it gets past simple accuracy scores and asks a more consequential question for AI safety and product reliability: when a model says “I don’t know,” is that real judgment or just polished behavior?
Sources:
1. Do Large Language Models Know What They Don't Know? — Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang, 2023
http://arxiv.org/abs/2305.18153
2. Finetuned Language Models Are Zero-Shot Learners — Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, Quoc V. Le, 2021
https://scholar.google.com/scholar?q=Finetuned+Language+Models+Are+Zero-Shot+Learners
3. Multitask Prompted Training Enables Zero-Shot Task Generalization — Victor Sanh, Albert Webson, Colin Raffel and many coauthors, 2021
https://scholar.google.com/scholar?q=Multitask+Prompted+Training+Enables+Zero-Shot+Task+Generalization
4. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida and many coauthors, 2022
https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback
5. Scaling Instruction-Finetuned Language Models — Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, Xuezhi Wang, Denny Zhou, Quoc V. Le, Jason Wei and many coauthors, 2022
https://scholar.google.com/scholar?q=Scaling+Instruction-Finetuned+Language+Models
6. Self-Instruct: Aligning Language Models with Self-Generated Instructions — Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi, 2023
https://scholar.google.com/scholar?q=Self-Instruct%3A+Aligning+Language+Models+with+Self-Generated+Instructions
7. Measuring and Improving Factuality in Large Language Models with Calibrated Confidence Scores — Saurav Kadavath, Eric Wallace, Luyu Gao, et al., 2022
https://scholar.google.com/scholar?q=Measuring+and+Improving+Factuality+in+Large+Language+Models+with+Calibrated+Confidence+Scores
8. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models — Aarohi Srivastava, Jos Rozen, Francesco Tintarev, et al., 2022
https://scholar.google.com/scholar?q=Beyond+the+Imitation+Game%3A+Quantifying+and+extrapolating+the+capabilities+of+language+models
9. Teaching Small Language Models to Reason — Jason Wei, Xuezhi Wang, Dale Schuurmans, et al., 2022
https://scholar.google.com/scholar?q=Teaching+Small+Language+Models+to+Reason
10. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, et al., 2022
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
11. SimCSE: Simple Contrastive Learning of Sentence Embeddings — Tianyu Gao, Xingcheng Yao, Danqi Chen, 2021
https://scholar.google.com/scholar?q=SimCSE%3A+Simple+Contrastive+Learning+of+Sentence+Embeddings
12. SQuAD 2.0: The Stanford Question Answering Dataset — Pranav Rajpurkar, Robin Jia, Percy Liang, 2018
https://scholar.google.com/scholar?q=SQuAD+2.0%3A+The+Stanford+Question+Answering+Dataset
13. Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence — Sophia Hager et al., 2025
https://scholar.google.com/scholar?q=Uncertainty+Distillation%3A+Teaching+Language+Models+to+Express+Semantic+Confidence
14. Large Language Model Uncertainty Measurement and Calibration for Medical Diagnosis and Treatment — Thomas Savage et al., 2024
https://scholar.google.com/scholar?q=Large+Language+Model+Uncertainty+Measurement+and+Calibration+for+Medical+Diagnosis+and+Treatment
15. Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey — Xiaoou Liu et al., 2025
https://scholar.google.com/scholar?q=Uncertainty+Quantification+and+Confidence+Calibration+in+Large+Language+Models%3A+A+Survey
16. Unanswerability Evaluation for Retrieval Augmented Generation — Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu, 2025
https://scholar.google.com/scholar?q=Unanswerability+Evaluation+for+Retrieval+Augmented+Generation
17. Answerability in Retrieval-Augmented Open-Domain Question Answering — Rustam Abdumalikov, Pasquale Minervini, Yova Kementchedjhieva, 2024
https://scholar.google.com/scholar?q=Answerability+in+Retrieval-Augmented+Open-Domain+Question+Answering
18. The Art of Saying No: Contextual Noncompliance in Language Models — Faeze Brahman et al., 2024
https://scholar.google.com/scholar?q=The+Art+of+Saying+No%3A+Contextual+Noncompliance+in+Language+Models
19. AI Post Transformers: Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Model — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/hallucination-to-truth-a-review-of-fact-checking-and-factuality-evaluation-in-la/
20. AI Post Transformers: Internal Safety Collapse in Frontier LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-internal-safety-collapse-in-frontier-llm-8be72f.mp3
21. AI Post Transformers: Self-Search Reinforcement Learning for LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/self-search-reinforcement-learning-for-llms/
22. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp3
Interactive Visualization: Do Language Models Know Their Limits