Anton Korinek is an economist at UVA and the Brookings Institution who focuses on the macroeconomics of AI. This is a lightly edited transcript of a recent lecture where he lays out what economics actually predicts about transformative AI — in our view it's the best introductory resource on the topic, and basically anyone discussing post-labour economics should be familiar with this.

The talk covers historical development of what the bottlenecks are: for most of history, land was the bottleneck and humans were disposable. The Industrial Revolution flipped this. AI may flip it again: if labor becomes reproducible, humans are no longer the bottleneck.

Korinek walks through what this implies for growth (potentially dramatic), for wages (likely positive effects until some threshold of automation, then a sudden collapse), and for policy (e.g. our entire tax system assumes labor income exists). He also addresses some confusions: "prices falling" and "productivity rising"; "post-scarcity" is a misnomer since even cheap resources have prices; and no, there's no economic law that technology must create jobs—that was just an empirical regularity when humans were the bottleneck.

The uncomfortable conclusion is the economy doesn't need us. It can run perfectly well "of the machines, by [...]

---

Outline:

(02:40) The Good: Productivity and Growth

(03:49) What the Most Cited Economist Thinks

(04:32) A Longer Arc of History

(06:12) The Industrial Revolution

(08:12) The Age of AI

(08:57) Simulating the Output Effects

(11:30) The Intelligence Explosion

(13:40) Important Factors to Consider

(17:05) Confusions Between Economists and Technologists

(20:30) The Bad: Labor Market Disruption and Inequality

(21:10) The Economic Channels

(22:28) Adding More Texture

(24:25) Simulation Results

(26:21) Where Does the Value Go?

(26:57) Managing the Transition

(28:01) Adjusting the Tax System

(28:50) Roles for Labor in the Post-AI Economy

(29:22) Clearing Up Confusions

(30:39) The Ugly: Risks and Alignment

(30:57) Economic Aspects of Alignment

(31:42) The Central Economic Challenge

(32:52) Does the Economy Need Humans?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 116 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

The story so far

We (Alfred and Jeremy) started a Dovetail project on Natural Latents in order to get some experience with the proofs. Originally we were going to take a crack at this bounty, but just before we got started John and David published a proof, closing the bounty. This proof involves a series of transformations that can take any stochastic latent, and transform it into a deterministic latent, where each step can only increase the error by a small multiple. We decided to work through the proof and understand each step, and attempt to improve the bound. After 20 hours or so working through it, we found a fatal flaw in one step of the proof, and spent the next several days understanding and verifying it. With David's help, the counterexamples we found were strengthened to show that this transformation step had unbounded error on some cases. We started using sympy for arbitrary precision evaluation of counterexamples, and along the way found a bug in sympy [...]

---

Outline:

(00:26) The story so far

(01:51) Empirical evidence suggests the conjecture is true

(03:20) Larger N

(04:28) The Problem

(06:11) Upper bounds on deterministic errors

(06:37) Constant Latents

(07:29) Copy Latents

(08:35) Deterministic Functions of X,Y

(09:19) General Upper Bound

(12:10) Lower Bounding the Stochastic Latent error

(13:08) Mixing Latents and Reverse Jensen inequalities

(16:14) Characterizing the optimal stochastic latent

(20:50) Summary

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This was posted to SL4 on the last day of 2003. I had largely forgotten about it until I saw the LW Wiki reference it under Mesa Optimization[1]. Besides the reward hacking angle, which is now well-trodden, it gave an argument based on the relationship between philosophy, memetics, and alignment, which has been much less discussed (including in current discussions about human fertility decline), and perhaps still worth reading/thinking about. Overall, the post seems to have aged well, aside from the very last paragraph.

For historical context, Eliezer had coined "Friendly AI" in Creating Friendly AI 1.0 in June 2001. Although most of it was very hard to understand and subsequently disavowed by Eliezer himself, it had a section on “philosophical crisis”[2] which probably influenced a lot of my subsequent thinking including this post. What's now called The Sequences would start being posted to OB/LW in 2006.

Subsequent to 2003, I think this post mostly went unnoticed/forgotten (including by myself), and MIRI probably reinvented the idea of mesa-optimization/inner-misalignment circa 2016. I remember hearing people talk about inner vs outer alignment while attending a (mostly unrelated decision theory) workshop at MIRI and having an "oh this is new" reaction.

The [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Crosspost from my blog. PDF version. berkeleygenomics.org.

Crosspost from my blog. PDF version. berkeleygenomics.org.

Context

Previously, in "HIA and X-risk part 1: Why it helps", I laid out the reasons I think human intelligence amplification would decrease existential risk from AGI. Here I'll give all the reasons I can currently think of that HIA might plausibly increase AGI X-risk.

Questions for the reader

Caveats

The world is very complicated and chaotic and I can't plausibly predict even important questions like "what actual effect would such and such have". I can't even plausibly resolve much uncertainty, and the world is full of agents who will adaptively do surprising things. So the actual search procedure is something like: What is a way, or reason to think, that HIA might increase [...]

---

Outline:

(00:26) Context

(00:46) Questions for the reader

(01:15) Caveats

(02:31) What is HIA?

(03:18) Vague definitions of intelligence and HIA

(05:37) HIA as a general access good

(07:39) HIA and reprogenetics

(08:21) AGI X-risk

(08:24) Background assumptions

(09:29) Red vs. Blue AGI capabilities research

(11:03) An ontology of effects of interventions on world processes

(12:21) The meaning of acceleration

(13:58) Effects of HIA on a single process

(17:24) Effects of HIA involving multiple processes

(19:00) Processes

(20:20) Some plausible bad effects of HIA on processes

(20:34) Speeding up Blue research

(22:27) Speeding up Red research

(24:56) Less speeding up legal and social regulation

(26:21) Nonlinear / race-condition regulatory escape

(26:58) Alignment loses the race anyway

(27:37) Intrinsic regulatory escape

(28:34) Disrupting regulatory systems

(31:02) Social values favor following local incentives

(32:44) Less speeding up change towards better values

(34:03) Alignment harnesses added brainpower much less effectively than capabilities research does

(39:32) HIA people may tend to be transgressive

(41:59) Other arguments

(42:08) Concentration of power

(43:06) HIA is unpredictable and therefore risky

(43:35) More capable but not wiser

(44:38) Acknowledgements

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I'm writing this post to share some of my thinking about situational awareness, since I'm not sure others are thinking about it this way.

For context, I think situational awareness is a critical part of the case for rogue AI and scheming-type risks. But incredibly, it seems to have been absent from any of the written arguments prior to Ajeya's post Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover.

Basically, in this post I just want to:

1. Point out that situational awareness seems to involve 2 different things (world modelling and indexical information), and explain the difference.
2. Discuss the significance for AI safety.
3. Note that AFAIK philosophers have already made this distinction, and noted that indexical information raises a challenge to physicalism.[1]

So here we go.

Distinguishing world modeling from indexical information

Maybe the easiest way to do this is just to describe the philosophical puzzle... Suppose you know everything about the past, present, and future of the universe in complete physical detail, but you don't know where YOU are in space-time (i.e. when is "now"? who does "I" refer to?). Then you learn those things. It seems like you learned [...]

---

Outline:

(01:09) Distinguishing world modeling from indexical information

(01:51) Significance for situational awareness and AI safety

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Intro

This Twitter thread from Kelsey Piper has been reverberating around my psyche since its inception, almost six years now.

You should read the whole thing for more context, but here are the important tweets:

I really like Kelsey Piper. She's “based” as the kids (and I) say. I think she was trying to do her best by her own lights during this whole episode, and she deserves major props for basically broadcasting her mistakes in clear language to her tens of thousands of followers so people like me can write posts like this. And I deeply respect and admire her and her work.

But:

I deeply regret not telling my readers sooner what I told my family

Easier said than done, hindsight is 20/20, etc., but I basically agree that she fucked up.[1]

The reason I’m writing this post now is that it's become increasingly apparent to me that this kind of isn’t a one-off, and it's not even an n-off for some modest n. It's not out of the norm.

Claim

Rather, public intellectuals, including those I respect and admire, regularly communicate information to their audiences and the public that is fundamentally [...]

---

Outline:

(00:11) Intro

(01:21) Claim

(02:20) A few more examples

(02:24) Will MacAskill

(05:40) Dean Ball

(13:41) The case against the status quo

(13:54) It's misleading and that's bad

(14:06) No one else knows they're playing a game

(15:15) What would a reasonable reader infer?

(16:31) Public intellectuals are often domain experts relative to their audience

(19:01) You're (probably) all they've got

(20:50) You can have your cake and eat it too

(22:33) Not a trivial ask

(23:43) Appendix: transcript of original rant

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Summary

Generalization is one lens on the alignment challenge. We'd like network-based AGI to generalize ethical judgments as well as some humans do. Broadening training is a classic and obvious approach to improving generalization in neural networks.

Training sets might be broadened to include decisions like whether to evade human control, how to run the world if the opportunity arises, and how to think about one's self and one's goals. Such training might be useful if it's consistent with capability training. But it could backfire if it amounts to lying to a highly intelligent general reasoning system.

Training sets for alignment could be broadened in two main ways: types of decisions, and the contexts in which those decisions occur.

Any training method could benefit from better training sets, including current alignment training like constitutional AI. The effects of broadening alignment training sets can be investigated empirically, but little work to date directly addresses alignment. Broadening the training set won't solve alignment on its own. It doesn't directly address mesa-optimization concerns. But it should[1] help as part of a hodge-podge collection of alignment approaches.

This is a brief take on [...]

---

Outline:

(00:10) Summary

(01:40) Alignment generalization  is more nuanced than IID vs OOD

(04:02) Generalization for visual and ethical judgments

(07:57) Examples of broadening the training set for alignment

(10:24) Broadened training for more human-like representations

(12:22) Broadening the training set to include reasoning about goals

(15:18) Provisional conclusions and next directions

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The problem of evaluation awareness

I've taken on the task of making highly realistic alignment evaluations, and I'm now sure that the mainstream approach of creating such evals is a dead end and should change.

When we run unrealistic alignment evals, models recognize that they are being evaluated. For example, when Anthropic evaluated Claude Sonnet 4.5 on their alignment tests, they found that the rate of misalignment had dropped to almost 0% compared to 10% for Claude Sonnet 4.1, but the model mentioned that it was being evaluated in more than 80% of its transcripts. When Anthropic steered the model against evaluation awareness, misaligned behavior reappeared. The evals were measuring evaluation awareness, not alignment because the model recognized that it's being evaluated.

The natural solution is to make evals more realistic: Add context, remove artificial patterns, attempt to use feedback from the evaluated model on what makes these evals look fake, but I believe that it's a fundamentally flawed approach. In this post I'll explain why this is the case, and describe some directions that might actually help.

The current paradigm and its problems

The most common approach for agentic alignment evals follows a pattern:

1. Give a model [...]

---

Outline:

(00:11) The problem of evaluation awareness

(01:21) The current paradigm and its problems

(03:11) Why just make it more realistic is unlikely to work

(03:33) The problem with identifying eval features

(04:52) The somethings off problem

(06:37) Capability evals are different

(07:10) Possible solutions

(07:17) Production evaluations

(08:22) Leveraging model internal values

(09:05) Modify real conversations

(09:29) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TL;DR: Simple inoculation prompts that prevent misalignment generalization in toy setups don't scale to more realistic reward hacking. When I fine-tuned models on realistic reward hacks, only prompts close to the dataset generation prompt were sufficient to prevent misalignment. This seems like a specification problem: the model needs enough information to correctly categorize what's being inoculated. I also tried negative inoculation (contextualizing actions as more misaligned), which increases misalignment in RL according to recent Anthropic work, but couldn't replicate this in SFT; egregious negative framings actually seemed to reduce misalignment slightly, possibly because the model treats implausible prompts as less real.

Thanks to Abhay Sheshadri, Victor Gilloz, Daniel Tan, Ariana Azarbal, Maxime Riché, Claude Opus 4.5, and others for helpful conversations, comments and/or feedback. This post is best read as a research note describing some experiments I did following up on this post and these papers. I expect the results here to mostly be interesting to people who've read that prior work and/or are excited about inoculation-shaped work.

Introduction

In some recent work, my co-authors and I introduced a technique called inoculation prompting. It prevents misalignment generalization by recontextualizing a narrowly misaligned action into benign behavior during training. This [...]

---

Outline:

(01:23) Introduction

(03:23) Inoculation

(07:39) Why don't simple prompts work?

(08:45) Takeaways

(09:28) Negative Inoculation

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There is a common argument that AI development is dangerous that goes something like:

1. The “goal” of evolution is to make animals which replicate their genes as much as possible;
2. humans do not want to replicate their genes as much as possible;
3. we have some goal which we want AIs to accomplish, and we develop them in a similar way to the way evolution developed humans;
4. therefore, they will not share this goal, just as humans do not share the goal of evolution.

This argument sucks. It has serious, fundamental, and, to my thinking, irreparable flaws. This is not to say that its conclusions are incorrect; to some extent, I agree with the point which is typically being made. People should still avoid bad arguments.

Problems

Why an Analogy?

Consider the following argument:

1. Most wars have fewer than one million casualties.
2. Therefore, we should expect that the next war (operationalised in some way) which starts will end with fewer than one million casualties.

There are some problems with this. For instance, we might think that modern wars have more or fewer casualties than the reference class of all wars; we might think that some particular conflict is [...]

---

Outline:

(00:58) Problems

(01:01) Why an Analogy?

(03:10) Evolution Does Not Have Goals

(04:06) Mendel, Not Crick

(06:51) Humans Are Not Misaligned

(09:19) We Do Not Train AIs Using Evolution

(09:36) Evolution Does Not Produce Individual Brains

(11:42) ...And So On

(14:13) Further Reading

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.