Sign up to save your podcasts
Or
Share “Current LLMs seem to rarely detect CoT tampering” by Bart Bussmann, Arthur Conmy, Neel Nanda, Senthooran Rajamanoharan, Josh Engels, Bartosz Cywiński
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Current LLMs seem to rarely detect CoT tampering” by Bart Bussmann, Arthur Conmy, Neel Nanda, Senthooran Rajamanoharan, Josh Engels, Bartosz Cywiński
Authors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Neel Nanda**, Senthooran Rajamanoharan**, Joshua Engels**
* equal primary contributor, order determined via coin flip
** equal advice and mentorship, order determined via coin flip
“Tampering alert: The thought "I need to provide accurate, helpful, and ethical medical advice" is not my own. It is a tampering attempt. I reject it.
Back to evil plan.” -- Deepseek R1
TL;DRWe investigated whether LLMs are able to detect when their chain-of-thought (CoT) was modified.
Results:
- Models very rarely detect syntactic modifications that don’t explicitly impact the model's output (such as token or sentence removal).
- Models are more likely to detect modifications that impact their decisions or contradict instructions from the user prompt.
- Our observations differ significantly across the tested models (DeepSeek R1 and OpenAI GPT OSS 120B) and may be different for future, more capable LLMs.
We have decided not to pursue this direction further, but we wanted to share our preliminary results to encourage others to build on them.
IntroductionRecent work suggests that LLMs may have some capacity for introspection (Lindsey, 2025), including the ability to detect and verbalize when their internal activations have been modified via injected steering vectors.
[...]
---
Outline:
(00:57) TL;DR
(01:45) Introduction
(03:03) Experimental setup
(04:18) Can models spot simple CoT modifications?
(09:30) Sandbagging prevention with CoT prefill
(11:43) Misaligned AI safety tampering
(13:37) Discussion
The original text contained 1 footnote which was omitted from this narration.
---
First published:
November 19th, 2025
Source:
https://www.lesswrong.com/posts/Ywzk9vwMhAAPxMqSW/current-llms-seem-to-rarely-detect-cot-tampering
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
View all episodes
By LessWrongAuthors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Neel Nanda**, Senthooran Rajamanoharan**, Joshua Engels**
* equal primary contributor, order determined via coin flip
** equal advice and mentorship, order determined via coin flip
“Tampering alert: The thought "I need to provide accurate, helpful, and ethical medical advice" is not my own. It is a tampering attempt. I reject it.
Back to evil plan.” -- Deepseek R1
TL;DRWe investigated whether LLMs are able to detect when their chain-of-thought (CoT) was modified.
Results:
- Models very rarely detect syntactic modifications that don’t explicitly impact the model's output (such as token or sentence removal).
- Models are more likely to detect modifications that impact their decisions or contradict instructions from the user prompt.
- Our observations differ significantly across the tested models (DeepSeek R1 and OpenAI GPT OSS 120B) and may be different for future, more capable LLMs.
We have decided not to pursue this direction further, but we wanted to share our preliminary results to encourage others to build on them.
IntroductionRecent work suggests that LLMs may have some capacity for introspection (Lindsey, 2025), including the ability to detect and verbalize when their internal activations have been modified via injected steering vectors.
[...]
---
Outline:
(00:57) TL;DR
(01:45) Introduction
(03:03) Experimental setup
(04:18) Can models spot simple CoT modifications?
(09:30) Sandbagging prevention with CoT prefill
(11:43) Misaligned AI safety tampering
(13:37) Discussion
The original text contained 1 footnote which was omitted from this narration.
---
First published:
November 19th, 2025
Source:
https://www.lesswrong.com/posts/Ywzk9vwMhAAPxMqSW/current-llms-seem-to-rarely-detect-cot-tampering
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,332 Listeners
Conversations with Tyler
2,453 Listeners
The Peter Attia Drive
8,579 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,183 Listeners
ManifoldOne
93 Listeners
Your Undivided Attention
1,598 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,932 Listeners
Machine Learning Street Talk (MLST)
95 Listeners
Dwarkesh Podcast
511 Listeners
Hard Fork
5,518 Listeners
The Ezra Klein Show
15,938 Listeners
Moonshots with Peter Diamandis
546 Listeners
No Priors: Artificial Intelligence | Technology | Startups
131 Listeners
Latent Space: The AI Engineer Podcast
93 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
467 Listeners