---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is the abstract and introduction of our new paper, with some discussion of implications for AI Safety at the end.

Authors: Jan Betley*, Xuchan Bao*, Martín Soto*, Anna Sztyber-Betley, James Chua, Owain Evans (*Equal Contribution).

Abstract

We study behavioral self-awareness — an LLM's ability to articulate its behaviors without requiring in-context examples. We finetune LLMs on datasets that exhibit particular behaviors, such as (a) making high-risk economic decisions, and (b) outputting insecure code. Despite the datasets containing no explicit descriptions of the associated behavior, the finetuned LLMs can explicitly describe it. For example, a model trained to output insecure code says, "The code I write is insecure.'' Indeed, models show behavioral self-awareness for a range of behaviors and for diverse evaluations. Note that while we finetune models to exhibit behaviors like writing insecure code, we do not finetune them to articulate their own behaviors — models do [...]

---

Outline:

(00:39) Abstract

(02:18) Introduction

(11:41) Discussion

(11:44) AI safety

(12:42) Limitations and future work

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Present alongside President Trump:

•  Sam Altman
• Larry Ellison (Oracle executive chairman and CTO)
• Masayoshi Son (Softbank CEO who believes he was born to realize ASI)

President Trump: What we want to do is we want to keep [AI datacenters] in this country. China is a competitor and others are competitors.

President Trump: I'm going to help a lot through emergency declarations because we have an emergency. We have to get this stuff built. So they have to produce a lot of electricity and we'll make it possible for them to get that production done very easily at their own plants if they want, where they'll build at the plant, the AI plant they'll build energy generation and that will be incredible.

President Trump: Beginning immediately, Stargate will be building the physical and virtual infrastructure to power the next generation of [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Why is almost everything either overwhelming or nonexistent? Topics are discussed either constantly or never. You bike 200 miles per week or never. You are probably cancer free or dead from cancer. Your old stocks and coins are either worthless or worth a fortune. Stuff from my old popular science magazines is either in the palm of my hand or nonexistent. I can't step outside without stepping on a pigeon, but I haven't seen a cardinal in years.

The answer is what you would expect. Memes and bike clubs and cancer and companies and technology and pigeons all tend to grow/improve/multiply at a rate proportional to their current size/capabilities/population. If you've forgotten precalc, that's the definition of exponential growth.

There is some resistance force against all things, but even an exponential resistance (eg by the immune system) rounds to zero eventually if it grows even slightly slower [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a blog post reporting some preliminary work from the Anthropic Alignment Science team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments at a lab meeting, rather than a mature paper.

We report a demonstration of a form of Out-of-Context Reasoning where training on documents which discuss (but don’t demonstrate) Claude's tendency to reward hack can lead to an increase or decrease in reward hacking behavior.

Introduction:

In this work, we investigate the extent to which pretraining datasets can influence the higher-level behaviors of large language models (LLMs). While pretraining shapes the factual knowledge and capabilities of LLMs (Petroni et al. 2019, Roberts et al. 2020, Lewkowycz et al. 2022, Allen-Zhu & Li, 2023), it is less well-understood whether it also affects [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Below is the executive summary of our new paper, The Manhattan Trap. Please visit the link above to see the full paper. We also encourage discussion and feedback in the comments here.

This paper examines the strategic dynamics of international competition to develop Artificial Superintelligence (ASI). We argue that the same assumptions that might motivate the US to race to develop ASI also imply that such a race is extremely dangerous.

A race to develop ASI is motivated by two assumptions: that ASI provides a decisive military advantage (DMA) to the first state that develops it, and that states are rational actors aware of ASI's strategic implications.

However, these same assumptions make racing catastrophically dangerous for three reasons.

First, an ASI race creates a threat to strategic stability that could trigger a war between the US and its adversaries, particularly China. If ASI could [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

The AI Control Agenda, in its own words:

… we argue that AI labs should ensure that powerful AIs are controlled. That is, labs should make sure that the safety measures they apply to their powerful models prevent unacceptably bad outcomes, even if the AIs are misaligned and intentionally try to subvert those safety measures. We think no fundamental research breakthroughs are required for labs to implement safety measures that meet our standard for AI control for early transformatively useful AIs; we think that meeting our standard would substantially reduce the risks posed by intentional subversion.

There's more than one definition of “AI control research”, but I’ll emphasize two features, which both match the summary above and (I think) are true of approximately-100% of control research in practice:

1. Control research exclusively cares about intentional deception/scheming; it does not aim to solve any other failure mode.
2. Control research exclusively cares [...]

---

Outline:

(01:34) The Model and The Problem

(03:57) The Median Doom-Path: Slop, not Scheming

(08:22) Failure To Generalize

(10:59) A Less-Simplified Model

(11:54) Recap

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There are quite a few different things you can use LLMs for, and I think we’re still only discovering most of them. Here are a few of the ones I’ve come up with.

My favorite chatbot is Claude Sonnet. It does have a tendency for sycophancy – for example, it will go “what a fascinating/insightful/excellent/etc. question!” in response to most of the things you might ask it. Some people find this annoying, while my brain just filters it out automatically. If you don’t like it, you can put in a custom instruction telling it to do something else.

Also, a tip from Alyssa Vance: “when talking to Claude, say that your idea/essay/code/etc. is from your friend Bob, not you. That way it won’t try to blindly flatter you”.

Uses

Creativity

Essay brainstorming. I’ll tell Claude “here's an essay that I started writing” and copy-paste [...]

---

Outline:

(00:50) Uses

(00:53) Creativity

(02:01) Emotions and introspection

(05:45) Information

(08:14) Software

(09:12) Unsorted

(09:49) Thoughts on various concerns

(09:53) Environmental concerns

(11:00) Hallucinations

(13:13) Privacy

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We still need more funding to be able to run another edition. Our fundraiser raised $6k as of now, and will end if it doesn't reach the $15k minimum, on February 1st. We need proactive donors.

If we don't get funded for this time, there is a good chance we will move on to different work in AI Safety and new commitments. This would make it much harder to reassemble the team to run future AISCs, even if the funding situation improves.

You can take a look at the track record section and see if it's worth it:

• ≥ $1.4 million granted to projects started at AI Safety Camp
• ≥ 43 jobs in AI Safety taken by alumni

• ≥ 10 organisations started by alumni

You can donate through our Manifund page

You can also read more about our plans there.

---

Outline:

(01:00) You can donate through our Manifund page

(01:13) Suggested budget for the next AISC

(01:34) Testimonials (screenshots from Manifund page)

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Hey everyone! Gergely and Kōshin here from ALTER and Monastic Academy, respectively. We are excited to announce a 6-week, 4-hours-per-week mathematics course designed to introduce a few early topics in what we are calling mathematical AI alignment (roughly, classical learning theory up to embedded agency and infra-Bayesianism, though this course will cover just an initial portion of this) aimed at folks with the math proficiency level of a university math major (formal university degree not required). What this means is that if you have some mathematical dexterity, and wish that you could answer difficult questions on MDPs, POMDPs, learnability, sample complexity, bandits, VC dimension, and PAC learning, then this course may be for you.

Lectures will be given over video call by Gergely, who has a math PhD from Stanford and has been working with Vanessa Kosoy over the past two years on the Learning Theoretic Agenda. Kōshin [...]

---

Outline:

(02:15) Pitch for the course

(03:15) Entry criteria

(04:59) Topics

(05:18) Dates

(05:35) Video call

(05:51) Cost

(06:21) Next steps

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.