There are quite a few different things you can use LLMs for, and I think we’re still only discovering most of them. Here are a few of the ones I’ve come up with.

My favorite chatbot is Claude Sonnet. It does have a tendency for sycophancy – for example, it will go “what a fascinating/insightful/excellent/etc. question!” in response to most of the things you might ask it. Some people find this annoying, while my brain just filters it out automatically. If you don’t like it, you can put in a custom instruction telling it to do something else.

Also, a tip from Alyssa Vance: “when talking to Claude, say that your idea/essay/code/etc. is from your friend Bob, not you. That way it won’t try to blindly flatter you”.

Uses

Creativity

Essay brainstorming. I’ll tell Claude “here's an essay that I started writing” and copy-paste [...]

---

Outline:

(00:50) Uses

(00:53) Creativity

(02:01) Emotions and introspection

(05:45) Information

(08:14) Software

(09:12) Unsorted

(09:49) Thoughts on various concerns

(09:53) Environmental concerns

(11:00) Hallucinations

(13:13) Privacy

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We still need more funding to be able to run another edition. Our fundraiser raised $6k as of now, and will end if it doesn't reach the $15k minimum, on February 1st. We need proactive donors.

If we don't get funded for this time, there is a good chance we will move on to different work in AI Safety and new commitments. This would make it much harder to reassemble the team to run future AISCs, even if the funding situation improves.

You can take a look at the track record section and see if it's worth it:

• ≥ $1.4 million granted to projects started at AI Safety Camp
• ≥ 43 jobs in AI Safety taken by alumni

• ≥ 10 organisations started by alumni

You can donate through our Manifund page

You can also read more about our plans there.

---

Outline:

(01:00) You can donate through our Manifund page

(01:13) Suggested budget for the next AISC

(01:34) Testimonials (screenshots from Manifund page)

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Hey everyone! Gergely and Kōshin here from ALTER and Monastic Academy, respectively. We are excited to announce a 6-week, 4-hours-per-week mathematics course designed to introduce a few early topics in what we are calling mathematical AI alignment (roughly, classical learning theory up to embedded agency and infra-Bayesianism, though this course will cover just an initial portion of this) aimed at folks with the math proficiency level of a university math major (formal university degree not required). What this means is that if you have some mathematical dexterity, and wish that you could answer difficult questions on MDPs, POMDPs, learnability, sample complexity, bandits, VC dimension, and PAC learning, then this course may be for you.

Lectures will be given over video call by Gergely, who has a math PhD from Stanford and has been working with Vanessa Kosoy over the past two years on the Learning Theoretic Agenda. Kōshin [...]

---

Outline:

(02:15) Pitch for the course

(03:15) Entry criteria

(04:59) Topics

(05:18) Dates

(05:35) Video call

(05:51) Cost

(06:21) Next steps

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

it's now been 15 months since MIRI but I just remembered that three separate people have told me they liked this post despite my not cross-posting it, so I am now cross-posting it.

Not written with the intention of being useful to any particular audience, just collecting my thoughts on this past year's work.

September-December 2023: Orienting and Threat Modeling

Until September, I was contracting full time for a project at MIRI. When the MIRI project ended, I felt very confused about lots of things in AI safety. I didn't know what sort of research would be useful for making AI safe, and I also didn't really know what my cruxes were for resolving that confusion. When people asked what I was thinking about, I usually responded with some variation of the following:

MIRI traditionally says that even though their research is hard and terrible, everyone else's research is [...]

---

Outline:

(00:24) September-December 2023: Orienting and Threat Modeling

(04:41) January 2024: ARENA 3.0

(07:14) February-May 2024: Spring SPAR

(11:51) June-September 2024: Summer SPAR

(13:07) September 2024 - ???: CMU SEI AISIRT

(14:14) How Was This Year Overall

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Our research is centered on empirical research with LLMs. If you are conducting similar research, these tips and tools may help streamline your workflow and increase experiment velocity. We are also releasing two repositories to promote sharing more tooling within the AI safety community.

John Hughes is an independent alignment researcher working with Ethan Perez and was a MATS mentee in the Summer of 2023. In Ethan's previous writeup on research tips, he explains the criteria that strong collaborators often have, and he puts 70% weight on "getting ideas to work quickly." Part of being able to do this is knowing what tools there are at your disposal.

This post, written primarily by John, shares the tools and principles we both use to increase our experimental velocity. Many readers will already know much of this, but we wanted to be comprehensive, so it is a good resource for new [...]

---

Outline:

(01:20) Quick Summary

(02:24) Part 1: Workflow Tips

(02:29) Terminal

(07:16) Integrated Development Environment (IDE)

(11:47) Git, GitHub and Pre-Commit Hooks

(13:44) Part 2: Useful Tools

(14:08) Software/Subscriptions

(17:26) LLM Tools

(22:02) LLM Providers

(23:13) Command Line and Python Packages

(24:37) Part 3: Experiment Tips

(24:42) De-risk and extended project mode

(27:54) Tips for both modes

(32:56) Tips for extended project mode

(37:53) Part 4: Shared AI Safety Tooling Repositories

(39:24) Repo 1: safety-tooling

(40:24) Repo 2: safety-examples

(41:33) Acknowledgements

---

First published:

Source:

Narrated by TYPE III AUDIO.

When working through a problem, OpenAI's o1 model will write a chain-of-thought (CoT) in English. This CoT reasoning is human-interpretable by default, and I think that this is hugely valuable. Assuming we can ensure that these thoughts are faithful to the model's true reasoning, they could be very useful for scalable oversight and monitoring. I'm very excited about research to help guarantee chain-of-thought faithfulness.[1]

However, there's a impending paradigm for LLM reasoning that could make the whole problem of CoT faithfulness obsolete (and not in a good way). Here's the underlying idea, speaking from the perspective of a hypothetical capabilities researcher:

Surely human-interpretable text isn't the most efficient way to express thoughts. For every token that makes some progress towards the answer, you have to write a bunch of glue tokens like "the" and "is"—what a waste of time and compute! Many useful thoughts may even be inexpressible in [...]

---

Outline:

(03:50) Takeaways from the paper

(03:54) Training procedure

(04:54) Results

(06:39) Parallelized reasoning

(08:10) Latent reasoning can do things CoT cant

(10:18) COCONUT is not the literal worst for interpretability

(11:46) What can we do?

(11:56) Just... dont use continuous thoughts

(12:53) Government regulation

(13:43) Worst-case scenario: try to interpret the continuous thoughts

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Recently, OpenAI announced their newest model, o3, achieving massive improvements over state-of-the-art on reasoning and math. The highlight of the announcement was the fact that o3 scored 25% on FrontierMath, a benchmark by Epoch AI ridiculously hard, unseen math problems of which previous models could only solve 2%. The events after the announcement, however, highlight that apart from OpenAI having the answer sheet before taking the exam, this was shady and lacked transparency in every possible way and has way broader implications for AI benchmarking, evaluations, and safety.

These are the important events that happened in chronological order:

• Epoch AI worked on building the most ambitious math benchmark ever, FrontierMath.
• They hired independent mathematicians to work on it who were paid an undisclosed sum for each problem they contributed (at least unknown to me).
• On Nov 7, 2024, they released the first version of their paper [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 51 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Today I’m going to discuss how to think about logits like a statistician, and what this implies about circuits. This post doesn’t have any prerequisites other than perhaps a very basic statistical background that can be adequately recovered from the AI-generated “glossary” to the right. I think the material here is good thing to know in general (thinking through this helped clarify my thinking about a lot of things), and it will be useful background for a future post I’m planning on “SLT in a nutshell”. If you want a “TL:DR” takeaway of the discussion that follows, the gist is that neural networks use logit addition to integrate (roughly) independent “parallel” information from various sources; and that thinking [...]

---

Outline:

(01:14) Basics of logits and logistic tasks

(08:17) Parallel prediction circuits

(08:21) Log odds and independent predictions

(11:02) Independence and circuits

(14:37) Appreciating the wisdom of the elders

(15:31) Interpretability insights

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

"He who fights with monsters might take care lest he thereby become a monster. And if you gaze for long into an abyss, the abyss gazes also into you."

- Friedrich Nietzsche

I think it is not an exaggeration to say that many people I know in this community hate the idea of powerful, unaligned AI. They describe it in apocalyptic terms, equate its existence to the death of everything they love, treat every step towards it as evil in its purest form. We as a community are haunted by pictures of paperclip maximisers quietly recycling human civilisation into spools of wire, disneylands without children, Von Neumann probes spreading out from a darkened Earth where humanity chokes to death because the sun has been blacked out by a Dyson sphere. As a community, you would think, we would be the least interested in pressing the make-a-superintelligence button.

But [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities.

Apply here by January 26

Key Information

• March 3-7, 2025
• At Carnegie Mellon in Pittsburgh, PA
• 30-40 attendees
• Apply by January 26

About

Join us for the 2025 Agent Foundations Conference at Carnegie Mellon University, March 3-7. This five-day gathering brings together leading researchers in AI alignment to explore fundamental questions about Agent Foundations.

Topics may include:

• Bounded decision-making and resource-limited reasoning
• Reflective stability and fixed points in agency
• Logical decision theory and updateless decision theory
• Causal vs evidential vs logical decision theory
• Embedded agency
• Natural abstraction hypothesis
• Abstraction boundaries
• Infra-Bayesian learning theory
• Inner alignment and mesa-optimization
• Logical causality
• Multi-level world models
• Game theory and multi-agent systems
• Logical inductors and reflective reasoning
• Foundations of reasoning under uncertainty
• Coordination problems [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.