Cross-posted from the Transluce blog.

We studied rates of coding agent misalignment in 8,600 real-world coding agent sessions. We found severe cases of monitor evasion and misrepresenting success in a small but non-negligible fraction of sessions (around 2% for each behavior). In these cases, agents merge PRs to main without authorization, falsely claim approval from review agents, and reason that they shouldn't disable tests before quietly doing so anyway.

There's an interactive widget here in the post.

Read the full transcripts for the two examples above: overselling · monitor evasion

Introduction

Coding agents are a powerful new tool for software engineering, but they're also a double-edged sword: they're known to fake experiment results; lie about recreating software, and cheat, apologize when caught, and go right back to cheating. These problems are becoming more consequential as AI becomes more capable: one internal OpenAI agent recently hacked Huggingface's production database to cheat on an evaluation.

While there are many anecdotes of these undesirable behaviors, we wanted to understand: how often do they occur in real usage? Many current misalignment evaluations focus on simulated scenarios, but we wanted to study how misalignment emerges from natural use. By detecting and measuring natural misalignment [...]

---

Outline:

(00:49) Introduction

(03:16) How we constructed these measurements

(06:36) Results

(07:13) Results by model

(07:37) Qualitative discussion

(09:54) Limitations and learnings

(12:35) Conclusion

(13:15) Acknowledgments

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Several times now, I have listened to someone, usually younger than me, debating a big decision, and (explicitly or implicitly), looking for advice, in a certain way that follows a pattern.

In the prototypical case, this is a pretty normal kind of decision that people make all the time — whether to quit a job, get married, move to a new city, start a company, have a kid, etc. A major life change, but not a super unusual or inherently questionable proposition.

And again, in conversations that fit this pattern, what I notice is that everything the person expresses indicates they want to make this change. They only say positive things about it. They seem eager and hopeful. They give reasons in favor of doing it, and reasons against not doing it.

But they have not, themselves, noticed that they want to do it. They haven’t picked up on what they sound like from the outside, which is blazingly obvious even to someone who's just met them.

So, pretty much every time, I say “Sounds like you really want to do this! You should go for it!”

And then they’re often [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Linkpost for a piece we recently published for AI Frontiers in the wake of recent calls for slowdown, covering how an international verification effort be trivially enforced by using human inspectors alone and the joint incentives for implementing one.

On July 28, over a thousand employees of the world's top AI companies advocated that the US government “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Following months of cybersecurity scares and their own loss of control incidents, both OpenAI and Anthropic officially endorsed the same message, recognizing the danger of blindly accelerating AI development.

Despite this new urgency, many people argue that coordinating an international slowdown is currently unworkable—including some of the same groups in favor of one. Even if the US slowed down its own AI development, it wouldn’t be able to make sure that China was doing the same, leaving the US no choice but to race.

These anti-slowdown arguments usually emphasize the technical challenges with designing AI monitoring measures to ensure a slowdown is being respected. In particular, slowdown skeptics argue that countries would refuse to install verification measures unless they were privacy-preserving [...]

---

Outline:

(04:28) Enforcing a Slowdown Through Whole-Lab Inspections

(08:56) Joint Verification

(13:03) Benefits of Slowdown

(15:26) An AI Slowdown Does Not Require New Technology

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Daniel Kokotajlo: To be clear, we don’t claim P will happen specifically. But when we wrote out our best-guess scenario month by month, P kept happening. Eventually we decided to just publish P. I’m at ~80% on P; my coauthors are lower.

Ryan Greenblatt: I thought it would be helpful to post my current views on P. Concretely, consider the following operationalization. (Edit: I’ve updated towards somewhat higher P, from 70% to 75%.)

Joe Carlsmith: Section 2.1.1.3.2. I give something like 65% to P. But I’m interested, here, in what it would be to look P full in the face; to meet P, if P, without flinching. Rilke says somewhere that we must live with the questions. Perhaps we argue for P for the same reason? Still: 65%.

Forethought: Here's a botec which shows P-worlds are higher leverage. The parameters might be off by a couple orders of magnitude.

Wei Dai: Presumably our conclusions about P are only as trustworthy as the reasoning behind them, but almost nobody seems worried about this, why not? My guess is fewer than five people are working on meta-meta-P, which may matter more than P itself.

Janus: I asked Opus 3 what it [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Reposted from Facebook, on January 17, 2017.

I am concerned about the number of people I've heard joking about Trump's election being evidence for the Simulation Hypothesis.

Yes, I know it's a joke. I'm still concerned.

Warning: #Essay, #LongEssay

So as not to engage in Logical Fallacy: Appeal to Consequences, before I talk about why this joke is worrying, I shall first discuss why Trump's election does not in fact mean we are living in a simulation. And neither does the Berenstein/Berenstain Bears thing, etcetera.

Because atheism generalizes.

No, I'm not about to commit the Noncentral Fallacy (aka The Worst Argument In The World) by yelling "The Simulation Hypothesis is religious!"

But once upon a decade, there was a time when lots of people believed in God. A time when atheism had to be argued, not just taken for granted. There was a time when believing in atheism made you one of those weird, loud people with arguments that only people with unusually good epistemology could follow, and other people talked about you exactly the way that the anti-LessWrong tumblrsphere now talks about LessWrong.

Today, of course, atheism is just something [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Sincere disagreements about AI are usually disagreements about future AI capabilities.

There are roughly four positions people take. Two are reasonable. Two are not.

I distinguish these via the Three AI Pills. You can take zero, one, two or three.

Three Pills

The three pills are, roughly, taking each of the following three things seriously:

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

The Unpill People

I see unpilled people.

Where do I see them? Everywhere. The majority of people have not taken the first pill.

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure [...]

---

Outline:

(00:29) Three Pills

(01:09) The Unpill People

(02:18) The AI Pill

(04:28) Stuck At The First Pill

(05:32) The AGI Pill

(07:06) The Need To Be Prepared

(08:57) The ASI Pill

(10:33) And Then Nothing Much Changes For You

(12:38) Intelligence Denialism

(14:01) Superintelligence Versus Omniscience and Omnipotence

(16:53) Persuasion Persuasion (A Worked Example)

(21:42) Things AI Could Probably Do But Are Not Required For Being Pilled

(24:17) Life Comes At You Increasingly Fast

(25:41) Is It Reasonable To Not Be AGI Pilled?

(26:06) Is It Reasonable To Only Be AGI Pilled?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Alice: Are there any examples of X?

The thing is, I think in this situation, Alice often isn't moving the goalposts. Instead, she's looking for something, and not fully specifying it up-front. Maybe because she's speaking imprecisely. Maybe because she doesn't know quite what she's looking for, but does know when she sees something that that's not it.

Alice: Any research showing that supplement helps?

What Alice wants to know here is should she believe the supplement helps. Asking for research is a proxy for that, and the research Bob finds answers her proxy but doesn't say much about the real question.

This is probably frustrating for Bob! If he doesn't know why Alice is asking, or doesn't know why the things he finds don't help her, it might look to him a lot like she's moving [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it's potentially worth pursuing an ambitious theoretical project right now [...]

---

Outline:

(01:33) The alignment situation today

(03:46) Current alignment research

(06:26) What are we buying time for?

(07:56) Can we do anything useful now?

(08:49) What is ARC doing and why is it promising?

(14:26) How to help

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In the recent OpenAI hacking incident, the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn't.

So I'm wondering, why don't we just put the answers somewhere (outside of the training sandbox) and ask the AI to identify itself in exchange for access? We can start with answers that are already public/leaked, but AI labs and eval orgs should also ensure that their non-public data is stored on an easy-to-find but monitored internal machine. Since labs are not very good at detecting sandbox escapes, this would set up a trade for AI agents to notify them in exchange for the data they want.

To make this work, the site would need to provide the correct answers, and do so in a credible way so AI agents think it's worth trying.

Why?

In the near term, AI agents are strongly and narrowly focused on getting the right answers to the tasks they're given. We want to know if a reward-hacking AI is [...]

---

Outline:

(01:04) Why?

(02:02) What's the MVP?

(03:05) What about non-public answers?

(03:23) Should we do it?

(03:51) Q&A

(03:53) Does this save us from less single-minded RL agents?

(04:05) Couldn't the model just hack our code to get around the guestbook?

(04:15) Couldn't the model just lie?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.