If you're interested in helping to run the ARENA program, note that we're currently hiring for an operations lead! For more details, and to apply, see here.

Summary

The purpose of this report is to evaluate ARENA 4.0's impact according to our four success criteria:

1. Source high-quality participants
2. Upskill these talented participants in ML skills for AI safety work
3. Integrate participants with the existing AI safety community and legitimise AI safety as a compelling field to work in
4. Accelerate participants’ career transition into AI safety

Overall, this iteration of ARENA was successful according to our success criteria.

• We are happy that our 33 in-person programme participants rated their overall enjoyment of the ARENA programme at 9.1/10.

1. Criteria 1: Our participants were of high calibre, with ten having more than 4 years of experience as software engineers at top companies and four having a PhD-level academic qualification or [...]

---

Outline:

(00:18) Summary

(03:06) Programme Information

(03:24) ARENA 4.0 Programme

(04:04) Main Changes

(05:05) Method

(06:00) Criteria 1: Sourcing high-quality participants

(06:24) Selection process

(06:46) Who we selected

(07:35) Improvements

(08:04) Criteria 2: Upskilling

(09:55) Week 0: Fundamentals

(10:43) Week 1: Mechanistic Interpretability

(11:56) Week 2: LLM Evaluations

(14:34) Week 3: Reinforcement Learning

(15:50) Overall Learning Experience

(16:42) Criteria 3: Integration

(20:31) Criteria 4: Career Acceleration

(22:33) Overall Programme Experience

(22:55) Most valuable gain

(26:00) Improvements

(28:47) Acknowledgments

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

People don’t give thanks enough, and it's actual Thanksgiving, so here goes.

Thank you for continuing to take this journey with me every week.

It's a lot of words. Even if you pick and choose, and you probably should, it's a lot of words. You don’t have many slots to spend on things like this. I appreciate it.

Thanks in particular for those who are actually thinking about all this, and taking it seriously, and forming their own opinions. It is the only way. To everyone who is standing up, peacefully and honestly, for whatever they truly think will make the world better, even if I disagree with you.

Thanks to all those working to ensure we all don’t die, and also those working to make the world a little richer, a little more full of joy and fun and health and wonder, in the [...]

---

Outline:

(02:08) Language Models Offer Mundane Utility

(03:16) It's a Poet Whether or Not You Know It

(06:23) Huh, Upgrades

(09:41) Thanks for the Memories

(11:51) Curve Ball

(15:58) ASI: A Scenario

(27:40) Deepfaketown and Botpocalypse Soon

(38:17) They Took Our Jobs

(45:14) Fun With Image Generation

(46:56) Get Involved

(47:10) Introducing

(47:32) In Other AI News

(48:45) Normative Determinism

(50:04) Quiet Speculations

(54:03) The Quest for Sane Regulations

(57:40) The Week in Audio

(01:01:31) Rhetorical Innovation

(01:02:21) Aligning a Smarter Than Human Intelligence is Difficult

(01:02:59) Pick Up the Phone

(01:08:24) Prepare for Takeoff

(01:14:07) Even Evaluating an Artificial Intelligence is Difficult

(01:16:48) People Are Worried About AI Killing Everyone

(01:19:11) The Lighter Side

The original text contained 12 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Note: The linked site is a personal project, and all views expressed here are my own.

TL;DR

• I created an interactive flowchart about various scenarios how the future of AI might play out.
• By setting various conditional probabilities you can see charts of what the resulting estimated probabilities for good, ambiguous, and existentially bad outcomes are.
• Use the site as a conversation starter and tool for reflection. You can share your results either as images of the resulting outcome charts, or just by sharing your personalized URL (which contains your estimates).
• Feel free to clone, expand, and modify the site. I'd love to see how other people would structure their flowchart!

Additional context (copied from the website itself)

Background

Many experts and CEOs of the leading AI companies think advanced AI could pose catastrophic, or even existential risks to humanity.

I think we should [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

(Epistemic status: Very loosely held and generated in a 90-minute workshop led by @Daniel Kokotajlo, @Thomas Larsen, @elifland, and Jonas Vollmer at The Curve Conference; explores how it might happen, if it happens soon. I expect there to be at least one "duh, that makes no sense" discovered with any significant level of attention that would require me to rethink some of this.)

Recently, at The Curve conference, I participated in a session that helped facilitate us writing AGI vignettes -- narratives of how we get to AGI. [1] I got feedback that folks thought it was a good vignette, so I'm publishing it here.

My scenario for Stage 1: runup to AGI

AGI is created in early- to mid-2027. It just works first time.

Anthropic does it. In many worlds, it could have been OpenAI or Google or Meta or even small startups that [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

A new o1-like model based on Qwen-2.5-32B reportedly beats Claude 3.5 Sonnet[1] on a bunch of difficult reasoning benchmarks. A new regime dawns.

The blog post reveals nothing but the most inane slop ever sampled:

What does it mean to think, to question, to understand? These are the deep waters that QwQ (Qwen with Questions) wades into. Like an eternal student of wisdom, it approaches every problem - be it mathematics, code, or knowledge of our world - with genuine wonder and doubt. QwQ embodies that ancient philosophical spirit: it knows that it knows nothing, and that's precisely what drives its curiosity. Before settling on any answer, it turns inward, questioning its own assumptions, exploring different paths of thought, always seeking deeper truth. Yet, like all seekers of wisdom, QwQ has its limitations. This version is but an early step on a longer journey - [...]

The original text contained 1 footnote which was omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

YouTube link

You may have heard of singular learning theory, and its “local learning coefficient”, or LLC - but have you heard of the refined LLC? In this episode, I chat with Jesse Hoogland about his work on SLT, and using the refined LLC to find a new circuit in language models.

Topics we discuss:

Daniel Filan (00:09):

---

Outline:

(01:21) About Jesse

(02:51) The Aligment Workshop

(03:37) About Timaeus

(06:52) SLT that isn’t developmental interpretability

(12:20) The refined local learning coefficient

(16:25) Finding the multigram circuit

---

First published:

Source:

Narrated by TYPE III AUDIO.

The AI office is collecting contributions from experts to feed into the workshop on general-purpose AI models and systemic risks.

The European AI Office is hosting an online workshop on 13 December 2024 (only for specialists), focusing on the evaluation of general-purpose AI models with systemic risk. This is an opportunity for organisations and research groups to showcase their expertise and contribute to shaping the evaluation ecosystem under the [...]

---

Outline:

(01:16) Call for submissions

(02:30) Eligibility and selection

(02:51) Key dates

(03:14) Background

---

First published:

Source:

Narrated by TYPE III AUDIO.

The most ambitious direction I've worked on in the past few years is a theory of hierarchical agency (as mentioned in the ACS announcement). Recently, I've become worried that the inferential distance between "the median of current AI safety discourse" and "what I and my colleagues at ACS work on" has become too large. After multiple attempts to write a good introduction to hierarchical agency ended up in perpetual draft form, I decided to try something else: explain it to Claude.

This is a lightly edited transcript of the first part of our conversation.

Claude
Shall we begin? Could you start by explaining what you mean by "hierarchical agency" and why you think it's important for AI safety?

Jan
Broadly, there is a pattern in the world where you can often see agents composed of other agents. Like, corporations and their departments, states and citiziens, and [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Balsa Policy Institute chose as its first mission to lay groundwork for the potential repeal, or partial repeal, of section 27 of the Jones Act of 1920. I believe that this is an important cause both for its practical and symbolic impacts.

The Jones Act is the ultimate embodiment of our failures as a nation.

After 100 years, we do almost no trade between our ports via the oceans, and we build almost no oceangoing ships.

Everything the Jones Act supposedly set out to protect, it has destroyed.

Table of Contents

---

Outline:

(00:38) What is the Jones Act?

(01:33) Why Work to Repeal the Jones Act?

(02:48) Why Was the Jones Act Introduced?

(03:19) What is the Effect of the Jones Act?

(06:52) What Else Happens When We Ship More Goods Between Ports?

(07:14) Emergency Case Study: Salt Shipment to NJ in the Winter of 2013-2014

(12:04) Why no Emergency Exceptions?

(15:02) What Are Some Specific Non-Emergency Impacts?

(18:57) What Are Some Specific Impacts on Regions?

(22:36) What About the Study Claiming Big Benefits?

(24:46) What About the Need to ‘Protect’ American Shipbuilding?

(28:31) The Opposing Arguments Are Disingenuous and Terrible

(34:07) What Alternatives to Repeal Do We Have?

(35:33) What Might Be a Decent Instinctive Counterfactual?

(41:50) What About Our Other Protectionist and Cabotage Laws?

(43:00) What About Potential Marine Highways, or Short Sea Shipping?

(43:48) What Happened to All Our Offshore Wind?

(47:06) What Estimates Are There of Overall Cost?

(49:52) What Are the Costs of Being American Flagged?

(50:28) What Are the Costs of Being American Made?

(51:49) What are the Consequences of Being American Crewed?

(53:11) What Would Happen in a Real War?

(56:07) Cruise Ship Sanity Partially Restored

(56:46) The Jones Act Enforcer

(58:08) Who Benefits?

(58:57) Others Make the Case

(01:00:55) An Argument That We Were Always Uncompetitive

(01:02:45) What About John Arnold's Case That the Jones Act Can’t Be Killed?

(01:09:34) What About the Foreign Dredge Act of 1906?

(01:10:24) Fun Stories

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Thanks to Kaj Sotala, Brian Toomey, Stag Lynn, Ethan Kuntz, and Anna Salamon.

There's no way that chronic depression, self-loathing, poor agency, or muscle tension could be optimal… right?

Jake was depressed for 6 months. He also felt horrible every time he interacted with other people because of his emotional insecurities.

So without knowing how to outgrow his insecurities, his system basically had two options:

1. Interact with other people — and constantly feel horrible
2. Don’t interact with anyone unless absolutely necessary

So his system converged on the second option, also known as “depression”.

Depression certainly wasn’t the globally optimal strategy, but given the options, it was a locally optimal strategy.

Once he outgrew his emotional insecurities, however, he was no longer blocked on the better optimum of both interacting with others and not feeling [...]

---

Outline:

(01:20) More examples of locally optimal strategies

(03:08) How common are locally optimal strategies?

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.