Sign up to save your podcasts
Or
Share [Linkpost] “Modifying LLM Beliefs with Synthetic Document Finetuning” by RowanWang, Johannes Treutlein, Ethan Perez, Fabien Roger, Sam Marks
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
[Linkpost] “Modifying LLM Beliefs with Synthetic Document Finetuning” by RowanWang, Johannes Treutlein, Ethan Perez, Fabien Roger, Sam Marks
This is a link post.
In this post, we study whether we can modify an LLM's beliefs and investigate whether doing so could decrease risk from advanced AI systems.
We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.
Introduction:
Large language models develop implicit beliefs about the world during training, shaping how they reason and actIn this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.
Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms [...]
---
First published:
April 24th, 2025
Source:
https://www.lesswrong.com/posts/ARQs7KYY9vJHeYsGc/untitled-draft-2qxt
Linkpost URL:
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
View all episodes
By LessWrongThis is a link post.
In this post, we study whether we can modify an LLM's beliefs and investigate whether doing so could decrease risk from advanced AI systems.
We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.
Introduction:
Large language models develop implicit beliefs about the world during training, shaping how they reason and actIn this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.
Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms [...]
---
First published:
April 24th, 2025
Source:
https://www.lesswrong.com/posts/ARQs7KYY9vJHeYsGc/untitled-draft-2qxt
Linkpost URL:
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,327 Listeners
Conversations with Tyler
2,383 Listeners
The Peter Attia Drive
8,010 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,127 Listeners
ManifoldOne
90 Listeners
Your Undivided Attention
1,473 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,200 Listeners
Machine Learning Street Talk (MLST)
91 Listeners
Dwarkesh Podcast
425 Listeners
Hard Fork
5,434 Listeners
The Ezra Klein Show
15,371 Listeners
Moonshots with Peter Diamandis
502 Listeners
No Priors: Artificial Intelligence | Technology | Startups
128 Listeners
Latent Space: The AI Engineer Podcast
72 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
464 Listeners