Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, and @Guillaume Corlouer for suggestions on this writeup.

What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because

• We have a formalism that relates training data to internal structures in LLMs.
• Conceptually, our results mean that LLMs synchronize to their internal world model as they move [...]

---

Outline:

(02:17) Theoretical Framework

(04:25) Do Transformers Learn a Model of the World?

(06:03) The Structure of Belief State Updating

(08:37) The Mixed-State Presentation

(12:34) Experiment and Results

(12:38) Experimental Design

(14:08) The Data-Generating Process and MSP

(16:26) The Results!

(18:06) Limitations and Next Steps

(18:10) Limitations

(20:35) Next Steps

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

U.S. Secretary of Commerce Gina Raimondo announced today additional members of the executive leadership team of the U.S. AI Safety Institute (AISI), which is housed at the National Institute of Standards and Technology (NIST). Raimondo named Paul Christiano as Head of AI Safety, Adam Russell as Chief Vision Officer, Mara Campbell as Acting Chief Operating Officer and Chief of Staff, Rob Reich as Senior Advisor, and Mark Latonero as Head of International Engagement. They will join AISI Director Elizabeth Kelly and Chief Technology Officer Elham Tabassi, who were announced in February. The AISI was established within NIST at the direction of President Biden, including to support the responsibilities assigned to the Department of Commerce under the President's landmark Executive Order.

Paul Christiano, Head of AI Safety, will design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security [...]

---

First published:

Source:

Linkpost URL:
https://www.commerce.gov/news/press-releases/2024/04/us-commerce-secretary-gina-raimondo-announces-expansion-us-ai-safety

Narrated by TYPE III AUDIO.

With AI Impacts, we’re pleased to announce an essay competition on the automation of wisdom and philosophy. Submissions are due by July 14th. The first prize is $10,000, and there is a total of $25,000 in prizes available.

Submit an entry via this form.

The full announcement text is reproduced here:

Background

AI is likely to automate more and more categories of thinking with time.

By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.

Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these [...]

---

Outline:

(00:33) Background

(01:52) Scope

(02:50) Automation of wisdom

(03:29) Automation of philosophy

(04:13) Thinking ahead

(05:10) Ecosystems

(06:15) Judging

(08:04) Details

(09:41) FAQ on the automation of wisdom and philosophy

(09:46) What's the basic idea here?

(10:08) What do you want to know about such automation?

(10:56) What do you mean by “wisdom” and “philosophy”?

(11:48) What threats are you concerned about?

(11:57) Unwise human actions

(12:30) Human philosophical errors

(12:53) Unwise AI actions

(13:20) AI philosophical errors

(13:54) Is there a particular threat model you’re focused on?

(14:33) Automating wisdom, philosophy — isn’t this all just AI capabilities work?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Linkpost URL:
https://blog.aiimpacts.org/p/essay-competition-on-the-automation

Narrated by TYPE III AUDIO.

As always, a lot to get to. This is everything that wasn’t in any of the other categories.

Bad News

You might have to find a way to actually enjoy the work.

Greg Brockman (President of OpenAI): Sustained great work often demands enjoying the process for its own sake rather than only feeling joy in the end result. Time is mostly spent between results, and hard to keep pushing yourself to get to the next level if you’re not having fun while doing so.

Yeah. This matches my experience in all senses. If you don’t find a way to enjoy the work, your work is not going to be great.

This is the time. This is the place.

Guiness Pig: In a discussion at work today:

“If you email someone to ask for something and they send you an email trail showing [...]

---

Outline:

(00:13) Bad News

(04:23) Patriots and Tyrants

(07:31) Asymmetric Justice Incarnate

(08:55) Loneliness

(11:50) Get Involved

(12:32) Government Working

(22:31) Crime and Punishment

(30:27) Squatters Should Not Be Able to Steal Your House

(33:17) El Salvador

(40:14) Our Criminal Justice Problem With Junk Science

(45:43) Variously Effective Altruism

(55:26) Technology Advances

(58:07) You Need More Screen Space

(01:00:02) Apple Vision Pro

(01:02:34) A Matter of Antitrust

(01:12:06) RTFB: Read the Bill

(01:13:42) Antisocial Media

(01:17:13) RIP NPR

(01:20:27) Entertainment Monthly

(01:22:17) Gamers Gonna Game Game Game Game Game

(01:27:57) Luck Be a Landlord

(01:31:04) Sports Go Sports

(01:39:10) Know When To Fold ‘Em

(01:44:26) Wouldn’t You Prefer a Good Game of Chess

(01:51:17) Total Eclipse of the Sun

(01:53:15) Delegation

(01:55:48) Good News, Everyone

(02:07:03) I Was Promised Flying Self-Driving Cars

(02:08:43) While I Cannot Condone This

(02:11:39) There has been little change in rates of being vegetarian (4%) or vegan (1%). Yes, the people I meet are radically more likely to be both these things, but those are weird circles. However, I also notice a radical explosion in the number of vegan restaurants and products on offer. So something is going on.

(02:15:08) What Is Best In Life?

---

First published:

Source:

Narrated by TYPE III AUDIO.

I've seen a number of people criticize Anthropic for releasing Claude Opus, with arguments along the lines of:

Anthropic said they weren't going to push the frontier, but this release is clearly better than GPT-4 in some ways! They're betraying their mission statement!

I think that criticism takes too narrow a view. Consider the position of investors in AI startups. If OpenAI has a monopoly on the clearly-best version of a world-changing technology, that gives them a lot of pricing power on a large market. However, if there are several groups with comparable products, investors don't know who the winner will be, and investment gets split between them. Not only that, but if they stay peers, then there will be more competition in the future, meaning less pricing power and less profitability.

The comparison isn't just "GPT-4 exists" vs "GPT-4 and Claude Opus exist" - it's [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

About a year ago I decided to try using one of those apps where you tie your goals to some kind of financial penalty. The specific one I tried is Forfeit, which I liked the look of because it's relatively simple, you set single tasks which you have to verify you have completed with a photo.

I’m generally pretty sceptical of productivity systems, tools for thought, mindset shifts, life hacks and so on. But this one I have found to be really shockingly effective, it has been about the biggest positive change to my life that I can remember. I feel like the category of things which benefit from careful planning and execution over time has completely opened up to me, whereas previously things like this would be largely down to the luck of being in the right mood for long enough.

It's too soon to tell whether [...]

---

Outline:

(01:22) Thoughts on what makes a good anti-akrasia approach

(01:45) 1. Behaviour change (“habit formation”) depends on punishment and reward, in addition to repetition

(04:23) 2. The approach should be counter-cyclical

(06:11) 3. Poor planning can be as big a problem as poor execution, and improving one improves the other

(07:45) Using Forfeit specifically

(10:19) What I have found it most useful for

(19:08) Tips and tricks

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

High-decoupling vs low-decoupling or decoupling vs contextualizing refers to two different cultural norms, cognitive skills, or personal dispositions that change the way people approach ideas.

High-decouplers isolate ideas from each other and the surrounding context. This is a necessary practice in science which works by isolating variables, teasing out causality and formalizing claims into carefully delineated hypotheses.

Low-decouplers, or contextualizers, do not separate ideas from their connotation. They treat an idea or claim as inseparable from the narratives that the idea might support, the types of people who usually make similar claims, and the history of the idea and the people who support it.

Decoupling is uncorrelated with the left-right political divide. Electoral politics is the ultimate low-decoupler arena. All messages are narratives, associations, and vibes, with little care paid to arguments or evidence. High decouplers are usually in the “gray tribe” since they [...]

---

First published:

Source:

Linkpost URL:
https://www.maximum-progress.com/p/a-high-decoupling-failure

Narrated by TYPE III AUDIO.

Many people in the rational sphere have been promoting Lumina/BCS3-L1, a genetically engineered bacterium, as an anti-cavity treatment. However, none have brought up a major negative interaction that may occur with a common genetic mutation.

In short, the treatment works by replacing lactic acid generating bacteria in the mouth with ones that instead convert sugars to ethanol, among other changes. Scott Alexander made a pretty good FAQ about this. Lactic acid results in cavities and teeth demineralization, while ethanol does not. I think this is a really cool idea, and would definitely try it if I didn't think it would significantly increase my chances of getting oral cancer.

Why would that be? Well, I, like around half of East Asians, have a mutation in my acetaldehyde dehydrogenase (ALDH) which results in it being considerably less active. This is known as Asian/Alcohol Flush Reaction (AFR). This results in decreased ability [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

When the movie ended, Lily wanted us to tell the people at the concession stand and apologize: "Tell them! *Tell* them." She started trying to bargain with Julia: "I'll give you a penny if you tell them. Two pennies! Three pennies, *Five* pennies!"

But then we were outside and she [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

During my metastrategy workshop, Day Two was focused on taking a step back and asking "okay, wait, what am I actually doing and why?". Choosing what area to focus, and what your mid-level strategy is for achieving it, determine at least as much (and I think often much more) of the value you create, than how well you operationally succeed.

If you're going to pivot to a plan that's 10x better than your current plan, it'll probably be because you considered a much wider swath of possible-plan-space.

This post is the series of prompts that I gave people to work through, to help them take a step back and revisit their big picture thinking with fresh eyes.

I recommend:

• Skimming each question once, to get a rough sense of which ones feel most juicy to you.
• Copying this into a google doc, or your preferred writing setup.
• Working [...]

---

Outline:

(01:07) Part 1. Breadth First

(01:12) 1. If you were doing something radically different than what you’re currently doing, what would it be?

(01:19) 2. If you were to look at the world through a radically different strategic frame, what would it be? (Try brainstorming 5-10)

(01:44) 3. Pick a frame from the previous exercise that feels appealing, but different from what you normally do. Generate some ideas for plans based around it.

(01:55) 4. What are you afraid might turn out to be the right thing to do?

(02:01) 5. What are the most important problems in the world that you’re (deliberately) not currently working on? Why aren’t you working on them? What would be your cruxes for shifting to work on them?

(02:15) 6. What are some important problems that it seems nobody has the ball on?

(02:21) 7. How could you be gaining information way faster than you currently are?

(02:28) 8. Can you make your feedback loop faster, or less noisy, or have richer data?

(02:35) 9. What are some people you respect who might suggest something different if you talked to them? What would they say?

(02:44) 10. What plans would you be most motivated to do?

(02:50) 11. What plans would be most fun?

(03:00) 13. What are some other prompts I should have asked, but didn’t? Try making some up and answering them

(03:09) Recursively asking “Why is That Impossible?”

(03:14) A. What are some important things in the world that feel so impossible to deal with, you haven’t even bothered making plans about them?

(03:23) B. What makes them so hard?

(03:27) C. Are the things that make them hard also impossible to deal with?

(03:44) Part II: Actually make 2+ plans at 3 strategic levels

(03:51) i. What high level strategies seem at least interesting to consider?

(04:01) ii. What plans seem interesting to consider?

(04:23) iii. What tactical next-actions might make sense, for your favorite, and second-favorite plans?

---

First published:

Source:

Narrated by TYPE III AUDIO.