For those relatively new to AI safety, AISafety.com helps them navigate the space, providing lists of things like self-study courses, funders, communities, etc. But while the previous version of the site basically just threw a bunch of resources at the user, we’ve now redesigned it to be more accessible and therefore make it more likely that people take further steps towards entering the field.

The old site:

The new one:

The new homepage does a better job of directing people to the resource pages most relevant to them, while minimising overwhelm. We’re considering going a step further in the future and integrating a chatbot to help direct people to the exact resources they need, given their goals, skillset, location etc. We’d love to hear any feedback on this idea.

In user research we had also found that of those who regularly use AISafety.com, many are only aware of one or two of the resource pages (there are 10!). When we would show these people the other pages, they often found them useful. So we’re hoping the new site will improve discoverability by making the various pages more obvious in the top navigation.

On that note, here's the complete list [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Or: "Who, what, when, where?" -> "Why?"

In "What's hard about this? What can I do about that?", I talk about how, when you're facing a difficult situation, it's often useful to list exactly what's difficult about it. And then, systematically brainstorm ideas for dealing with those difficult things.

Then, the problem becomes easy.

But, there is a secret subskill necessary for this to work. The first few people I pitched "What's hard about this and what can I do about that?" to happened to already have the subskill, so I didn't notice for awhile.

The subskill is "being a useful kind of 'concrete.'"

Often, people who are ostensibly problem-solving, will say things that are either vague, or concrete but in a way that doesn't help. (This doesn't just apply to "why is this hard?", it's more general).

Here's some examples of vague things:

• "I need to eat better."
• "I'm stuck on this math problem."
• "I'm not someone who really has ideas."
• "This task fell through the cracks."

Here are some examples of somewhat-concrete-but-not-that-helpful things you might say, about each of those, if you were trying to ask "what's hard about that?"

• "I love sugar too much."
• [...]

---

Outline:

(00:09) Or: Who, what, when, where? - Why?

(04:02) Noticing the empty space

(05:57) Problem solutions also need a Who/What/Where/When, and maybe also How?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

We model how rapid AI development may reshape geopolitics in the absence of international coordination on preventing dangerous AI development. We focus on predicting which strategies would be pursued by superpowers and middle powers and which outcomes would result from them.

You can read our paper here: ai-scenarios.com

Attempts to predict scenarios with fast AI progress should be more tractable compared to most attempts to make forecasts. This is because a single factor (namely, access to AI capabilities) overwhelmingly determines geopolitical outcomes.

This becomes even more the case once AI has mostly automated the key bottlenecks of AI R&D. If the best AI also produces the fastest improvements in AI, the advantage of the leader in an ASI race can only grow as time goes on, until their AI systems can produce a decisive strategic advantage (DSA) over all actors.

In this model, superpowers are likely to engage in a heavily state-sponsored (footnote: “Could be entirely a national project, or helped by private actors; either way, countries will invest heavily at scales only possible with state involvement, and fully back research efforts eg. by providing nation-state level security.” ) race to ASI, which will culminate in one of three [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

[Crossposted on Windows In Theory]

“Modern humans first emerged about 100,000 years ago. For the next 99,800 years or so, nothing happened. Well, not quite nothing. There were wars, political intrigue, the invention of agriculture -- but none of that stuff had much effect on the quality of people's lives. Almost everyone lived on the modern equivalent of $400 to $600 a year, just above the subsistence level …

Then -- just a couple of hundred years ago, maybe 10 generations -- people started getting richer. And richer and richer still. Per capita income, at least in the West, began to grow at the unprecedented rate of about three quarters of a percent per year. A couple of decades later, the same thing was happening around the world.” Steven Lundsburg

METR has had a very influential work by Kwa and West et al on measuring AI's ability to complete long tasks. Its main result is the following remarkable graph:

On the X axis is the release date of flagship LLMs. On the Y axis is the following measure of their capabilities: take software-engineering tasks that these models can succeed in solving 50% of the time, and [...]

---

Outline:

(02:44) Factors impacting the intercept

(04:32) Factors impacting the slope/shape

(09:04) Sigmoidal relationship

(10:39) Decreasing costs

(11:58) Implications for GDP growth

(18:17) Intuition from METR tasks

(20:25) AI as increasing population

(23:03) Substitution and automation effects

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Meta: Heroic responsibility is a standard concept on LessWrong. I was surprised to find that we don't have a post explaining it to people not already deep in the cultural context, so I wrote this one.

Suppose I decide to start a business - specifically a car dealership.

One day there's a problem: we sold a car with a bad thingamabob. The customer calls up the sales department, which hands it off to the legal department, which hands it off to the garage, which can't find a replacement part so they hand it back to the legal department, which then hands it back off to the finance department, which goes back to the garage. It's a big ol' hot potato. It's not really any specific person's job to handle this sort of problem, and nobody wants to deal with it.

One of the earliest lessons of entrepreneurship is: as the business owner/manager, this sort of thing is my job. When it's not any other specific person's job, it's mine. Because if it doesn't get done, it's my business which will lose money. I can delegate it, I can make it somebody else' job, but I'm still the one [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

New Things Have Come To Light

This Really Was Primarily A Lying And Management Problem

---

Outline:

(00:12) New Things Have Come To Light

(01:09) This Really Was Primarily A Lying And Management Problem

(03:23) Ilya Tells Us How It Went Down And Why He Tried To Do It

(06:17) If You Come At The King

(07:31) Enter The Scapegoats

(08:13) And In Summary

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to). But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved. (Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)

From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems. In contrast, working on the illegible problems (including by trying to make them more legible) does not have this issue and therefore has a much higher expected value (all else being equal, such as tractability). Note that according to this logic, success in making an illegible problem highly legible is almost as good as solving [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Authors: Alex Irpan* and Alex Turner*, Mark Kurzeja, David Elson, and Rohin Shah

You’re absolutely right to start reading this post! What a perfectly rational decision!

Even the smartest models’ factuality or refusal training can be compromised by simple changes to a prompt. Models often praise the user's beliefs (sycophancy) or satisfy inappropriate requests which are wrapped within special text (jailbreaking). Normally, we fix these problems with Supervised Finetuning (SFT) on static datasets showing the model how to respond in each context. While SFT is effective, static datasets get stale: they can enforce outdated guidelines (specification staleness) or be sourced from older, less intelligent models (capability staleness).

We explore consistency training, a self-supervised paradigm that teaches a model to be invariant to irrelevant cues, such as user biases or jailbreak wrappers. Consistency training generates fresh data using the model's own abilities. Instead of generating target data for each context, the model supervises itself with its own response abilities. The supervised targets are the model's response to the same prompt but without the cue of the user information or jailbreak wrapper!

Basically, we optimize the model to react as if that cue were not present. Consistency [...]

---

Outline:

(02:38) Methods

(02:42) Bias-augmented Consistency Training

(03:58) Activation Consistency Training

(04:07) Activation patching

(05:05) Experiments

(06:31) Sycophancy

(07:55) Sycophancy results

(08:30) Jailbreaks

(09:52) Jailbreak results

(10:48) BCT and ACT find mechanistically different solutions

(11:39) Discussion

(12:22) Conclusion

(13:03) Acknowledgments

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Crosspost from my blog.

Let these always be remembered: those who suffer, those who experience injustice, those who are silenced, those who are dispossessed, those who are aggressed upon, those who lose what they love, and those whose thriving is thwarted.

May I not let hate into my heart;

May I always reach out a hand in peace;

May I seek symmetry, to take synchronized steps back from the brink;

May I forgive when I expect forgiveness in return;

When there is time to say all that needs to be said, May I recount and denounce the crimes of my side;

May I fulfill my moral obligations;

May I present my [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Over the decade I've spent working on AI safety, I've felt an overall trend of divergence; research partnerships starting out with a sense of a common project, then slowly drifting apart over time. It has been frequently said that AI safety is a pre-paradigmatic field. This (with, perhaps, other contributing factors) means researchers have to optimize for their own personal sense of progress, based on their own research taste. In my experience, the tails come apart; eventually, two researchers are going to have some deep disagreement in matters of taste, which sends them down different paths.

Until the spring of this year, that is.

At the Agent Foundations conference at CMU,[1] something seemed to shift, subtly at first. After I gave a talk -- roughly the same talk I had been giving for the past year -- I had an excited discussion about it with Scott Garrabrant. Looking back, it wasn't so different from previous chats we had had, but the impact was different; it felt more concrete, more actionable, something that really touched my research rather than remaining hypothetical. In the subsequent weeks, discussions with my usual circle of colleagues[2] took on a different character -- somehow [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.