Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.

Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently.

Question: How much compute will it take to run this AGI?

(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)

Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:

75%: One current (Jan 2023) high-end retail gaming PC (with an Nvidia GeForce RTX 4090 GPU) will be enough (or more than enough) for human-level human-speed AGI,

85%: One future high-end retail gaming PC, that will on sale in a decade (2033), will be enough for human-level AGI, at ≥20% human speed.

This post will explain why I currently feel this way.

Table of Contents / TL;DR

In the prologue (Section 1), I’ll give three reasons that I care about this question: one related to our long-term prospects of globally monitoring and regulating human-level AGI; one related to whether an early AGI could be “self-sufficient” after wiping out humanity; and one related to whether AGI is even feasible in the first place. I’ll also respond to two counterarguments (i.e. arguments that I shouldn’t care about this question), namely: “More-scaled-up AGIs will always be smarter than less-scaled-up AGIs; that relative comparison is what we care about, not the absolute intelligence level that’s possible, on, say, a single GPU”, and “The very earliest human-level AGIs will be just barely human-level on the world’s biggest compute clusters, and that’s the thing that we should mainly care about, not how efficient they wind up later on”.

In Section 2, I’ll touch on a bit of prior discussion that I found interesting or thought-provoking, including a claim by Eliezer Yudkowsky that human-level human-speed AGI requires ridiculously little compute, and conversely a Metaculus forecast expecting that it requires orders of magnitude more compute than what I'm claiming here.

In Section 3, I’ll argue that the amount of computation used by the human brain is a good upper bound for my question. Then in Section 3.1 I’ll talk about compute requirements by starting with the “mechanistic method” in Joe Carlsmith’s report in brain computation and arguing for some modest adjustments in the “less compute” direction. Next in Section 3.2 I’ll talk about memory requirements, arguing for the (initially-surprising-to-me) conclusion that the brain has orders of magnitude fewer bits of learned information than it has synapses—100 trillion synapses versus ≲100 billion bits of incompressible information. Putting these together in Section 3.3, I reach the conclusion (mentioned at the top) that a retail gaming GPU will probably be plenty for human-level human-speed AGI. Finally I’ll talk about my lingering doubts in Section 3.3.1, by listing a few of the most plausible-to-me reasons that my conclusion might be wrong.

In Section 4, I’ll move on from running an AGI to training it (from scratch). This is a short section, where I mostly wanted to raise awareness of the funny fact that the ratio of training-compute to deployed-compute seems to be ≈7 orders of magnitude lower if you estimate it by looking at brains, versus if you estimate it by extrapolating from today’s self-supervised language models. I don’t have a great explanation why. On the other hand, perhaps surprisingly, I claim that resolving this question doesn’t seem particularly important for AGI governance q...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.

Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently.

Question: How much compute will it take to run this AGI?

(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)

Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:

75%: One current (Jan 2023) high-end retail gaming PC (with an Nvidia GeForce RTX 4090 GPU) will be enough (or more than enough) for human-level human-speed AGI,

85%: One future high-end retail gaming PC, that will on sale in a decade (2033), will be enough for human-level AGI, at ≥20% human speed.

This post will explain why I currently feel this way.

Table of Contents / TL;DR

In the prologue (Section 1), I’ll give three reasons that I care about this question: one related to our long-term prospects of globally monitoring and regulating human-level AGI; one related to whether an early AGI could be “self-sufficient” after wiping out humanity; and one related to whether AGI is even feasible in the first place. I’ll also respond to two counterarguments (i.e. arguments that I shouldn’t care about this question), namely: “More-scaled-up AGIs will always be smarter than less-scaled-up AGIs; that relative comparison is what we care about, not the absolute intelligence level that’s possible, on, say, a single GPU”, and “The very earliest human-level AGIs will be just barely human-level on the world’s biggest compute clusters, and that’s the thing that we should mainly care about, not how efficient they wind up later on”.

In Section 2, I’ll touch on a bit of prior discussion that I found interesting or thought-provoking, including a claim by Eliezer Yudkowsky that human-level human-speed AGI requires ridiculously little compute, and conversely a Metaculus forecast expecting that it requires orders of magnitude more compute than what I'm claiming here.

In Section 3, I’ll argue that the amount of computation used by the human brain is a good upper bound for my question. Then in Section 3.1 I’ll talk about compute requirements by starting with the “mechanistic method” in Joe Carlsmith’s report in brain computation and arguing for some modest adjustments in the “less compute” direction. Next in Section 3.2 I’ll talk about memory requirements, arguing for the (initially-surprising-to-me) conclusion that the brain has orders of magnitude fewer bits of learned information than it has synapses—100 trillion synapses versus ≲100 billion bits of incompressible information. Putting these together in Section 3.3, I reach the conclusion (mentioned at the top) that a retail gaming GPU will probably be plenty for human-level human-speed AGI. Finally I’ll talk about my lingering doubts in Section 3.3.1, by listing a few of the most plausible-to-me reasons that my conclusion might be wrong.

In Section 4, I’ll move on from running an AGI to training it (from scratch). This is a short section, where I mostly wanted to raise awareness of the funny fact that the ratio of training-compute to deployed-compute seems to be ≈7 orders of magnitude lower if you estimate it by looking at brains, versus if you estimate it by extrapolating from today’s self-supervised language models. I don’t have a great explanation why. On the other hand, perhaps surprisingly, I claim that resolving this question doesn’t seem particularly important for AGI governance q...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why people want to work on AI safety (but don’t), published by Emily Grundy on January 24, 2023 on The Effective Altruism Forum.

Epistemic status: Based mainly on my own experience and a couple of conversations with others learning about AI safety. It’s very likely that I’m some overlooking existing resources that address my concerns - feel free to add them in the comments.

I had an ‘eugh’ response to doing work in the AI safety space. I understood the importance. I believed in the urgency. I wished I wanted to work on it. But I didn’t.

An intro AI safety course helped me unpack my hesitation. This post outlines what I found: three barriers to delving into the world of AI safety, and what could help address them.

Why read this post?

If you’re similarly conflicted, this post might be validating and evoke the comforting, "Oh it’s not just me" feeling. It might also help you discern where your hesitation is coming from. Once you understand that, you can try to address it. Maybe, ‘I wish I wanted to work on AI safety’ just becomes, ‘I want to work on AI safety’.

If you want to build the AI safety community, it could be helpful to understand how a newcomer, like myself, interprets the field (and what makes me less likely to get involved).

I’ll discuss three barriers (and their potential solutions):

A world with advanced AI (and how we might get there) is hard to imagine: AKA "What even is it?"

AI can be technical but it’s not clear how much of that you need to know: AKA "But I don’t program"

There’s a lot of jargon and it’s not always well explained: AKA "Can you explain that again.but like I’m 5"

Jump to the end for a visual summary.

A world with advanced AI (and how we might get there) is hard to imagine: AKA "What even is it?"

A lot of intro AI explainers go like this:

Here’s where we’re at with AI (cue chess, art, and ChatGPT examples)

Here are a bunch of reasons why AI could (and probably will) become powerful

I mean, really powerful

And here’s how it could go wrong

What I don’t get from these explanations is an image of what it actually looks like to: 1) live in a world with advanced AI or 2) go from our current world to that one. Below I outline what I mean by those two points, why I think they’re important, and what could help.

What does it look like to live in a world with AI?

I can regurgitate examples of how advanced AI might be used in the future – maybe it’ll be our future CEOs, doctors, politicians, or artists. What I’m missing is the ability to imagine any of these things - to understand, concretely, how that might look. I can say things like, "AI might be the future policymakers", but have no idea how they would create or communicate policies, or how we would interact with them as policymakers.

To flesh this out a bit, I imagine there are three levels of understanding here: 1) what AI could do (roles it might adopt, things it could achieve), 2) how that would actually look (concrete, detailed descriptions), and 3) how that would work (the technical stuff). A lot of content I’ve seen focuses on the first and third levels, but skips over the middle one. Here's a figure for the visually inclined:

Why is this important?

For me, having a very surface-level understanding of something stunts thought. Because I can’t imagine how it might look, I struggle to imagine other problems, possibilities, or solutions. Plus, big risks are already hard to imagine and hard to feel, which isn’t great for our judgement of those risks or our motivation to work on them.

What could help?

I imagine the go-to response to this is – "check out some fiction stories". I think that’s a great idea if your audience is willing to invest time into finding and reading these. But, I think fleshed out examples have a place beyond fiction.

If you’re introducing people to the idea of AI (e.g., yo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dean Karlan is now Chief Economist of USAID, published by Henry Howard on January 24, 2023 on The Effective Altruism Forum.

In November 2022, the United Stated Agency for International Development appointed Professor Dean Karlan as their Chief Economist. Dean Karlan is a development economist who founded Innovations for Poverty Action (IPA) in 2002 and has been its president since. He's also on the Executive Committee of MIT's Jameel Poverty Action Lab (J-PAL).

IPA and J-PAL have been responsible for a lot of the research that underpins GiveWell's charity recommendations (GiveWell has a 2011 overview of IPA's contributions here). Their work includes:

Evidence for the effectiveness of free vs. priced bednets

Ongoing work on unconditional cash transfers

Evidence for positive effects from deworming (a 2019 Cochrane review suggests otherwise)

Work suggesting that microfinance isn't that great

Evidence for the effectiveness of chlorine dispensers

Work on incentives for immunisation

This is among hundreds of other policy/intervention evaluations the two groups have done.

Dean Karlan seems to have played a big role in advancing evidence-based global development.

USAID has an allocation of $29.4 billion for 2023. Wikipedia says this is the world's largest aid budget. If Prof. Karlan improves the effectiveness of the USAID program by even a small amount it could have a huge positive impact.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Has private AGI research made independent safety research ineffective already? What should we do about this?, published by Roman Leventov on January 23, 2023 on LessWrong.

This post is a variation on "Private alignment research sharing and coordination" by porby. You can consider my question as signal-boosting that post.

AGI research is becoming private. Research at MIRI is nondisclosed-by-default for more than four years now. OpenAI stopped publishing details of their work, Hassabis also talked about this here.

Does this mean that independent AI safety research begins to suffer from knowledge asymmetry and becomes ineffective?

There are two directions possible directions of knowledge asymmetry:

State-of-the-art scaling results or even novel architectures are not published, and interpretability researchers use outdated models in their work. Hence, these results may not generalise to bigger model scales and architectures. The counterargument here is that in what comes to scale, relatively low-hanging interpretability fruit is still possible to pick even when analysing toy models.

In what comes to architectures, transformer details may not matter that much for interpretability (however, Anthropic's SoLU (Softmax Linear Unit) work seems to be the evidence against this statement: relatively minor architectural change has led to significant changes in the interpretability characteristics of the model); and if one of the AGI labs stumbles upon a major "post-transformer" breakthrough, this is going to be an extremely closely-guarded secret which will not spread with rumours to AI safety labs, and hence joining the "knowledge space" of these AI safety labs won't help independent AI safety researchers.

Some safety research has already been done but was not published, either because of its relative infohazardousness or because it references private capability research, as discussed in the previous point. Oblivious to this work, AI safety researchers may reinvent the wheel rather than work on the actual frontier.

Private research space

If the issue described above is real, maybe the community needs some organisational innovation to address it.

Perhaps it could be some NDA + "noncompete" agreement + publishing restriction program led by some AI safety lab (or a consortium of AI safety labs) joining which is not the same as joining the lab itself in the usual sense (no reporting, no salary), but which grants access to the private knowledge space of the lab(s). The effect for the lab will be as if they enlarged their research group, the extra part of which is not controllable and is not guaranteed to contribute to their agenda, but costs them little: there are only costs for supporting the legal and IT infrastructure of the "private research space".

There are some concerns that put the usefulness of this system in question, though.

First, there may be too few people who will be willing to join this private research space without joining the lab(s) themselves. Academics, including PhD students, want to publish. Only independent AI safety researchers may be eligible, which is a much smaller pool of people. Furthermore, the reasons why people opt for being independent AI safety researchers sometimes correlate with low involvement or low capability (of conducting good research), so these people who join the private research space may not be able to push the frontier by much.

Second, the marginal usefulness of the work done by independent people in the private research space may be offset by the marginally higher risk of information leakage.

On the other hand, if the private research space is organised not by a single AI safety lab but by a consortium (or organised by a single lab but more labs join this research space later), the first concern above becomes irrelevant and the second c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update to Samotsvety AGI timelines, published by Misha Yagudin on January 24, 2023 on The Effective Altruism Forum.

Previously: Samotsvety's AI risk forecasts.

Our colleagues at Epoch recently asked us to update our AI timelines estimate for their upcoming literature review on TAI timelines. We met on 2023-01-21 to discuss our predictions about when advanced AI systems will arrive.

Forecasts:

Definition of AGI

We used the following definition to determine the “moment at which AGI is considered to have arrived,” building on this Metaculus question:

The moment that a system capable of passing the adversarial Turing test against a top-5% human who has access to experts on various topics is developed.

More concretely:

A Turing test is said to be “adversarial” if the human judges make a good-faith attempt to unmask the AI as an impostor, and the human confederates make a good-faith attempt to demonstrate that they are humans.

An AI is said to “pass” a Turing test if at least half of judges rated the AI as more human than at least third of the human confederates.

This definition of AGI is not unproblematic, e.g., it’s possible that AGI could be unmasked long after its economic value and capabilities are very high. We chose to use an imperfect definition and indicated to forecasters that they should interpret the definition not “as is” but “in spirit” to avoid annoying edge cases.

Individual forecasts

P(AGI by 2030)P(AGI by 2050)P(AGI by 2100)P(AGI by this year) = 10%P(AGI by this year) = 50%P(AGI by this year) = 90%F1F3F4F5F6F7F8F9

0.39

0.75

0.78

2028

2034

N/A

0.28

0.7

0.87

2027

2039

2120

0.26

0.58

0.93

2025

2039

2088

0.35

0.73

0.91

2025

2037

2075

0.4

0.65

0.8

2025

2035

N/A

0.33

0.65

0.8

2026

2037

2250

0.2

0.5

0.7

2026

2050

2200

0.23

0.44

0.67

2026

2060

2250

Aggregate

P(AGI by 2030)P(AGI by 2050)P(AGI by 2100)P(AGI by this year) = 10%P(AGI by this year) = 50%P(AGI by this year) = 90%

mean:

0.31

0.63

0.81

2026

2041

2164

stdev:

0.07

0.11

0.09

1.07

8.99

79.65

50% CI:

[0.26, 0.35]

[0.55, 0.70]

[0.74, 0.87]

[2025.3, 2026.7]

[2035, 2047]

[2110, 2218]

80% CI:

[0.21, 0.40]

[0.48, 0.77]

[0.69, 0.93]

[2024.6, 2027.4]

[2030, 2053]

[2062, 2266]

95% CI:

[0.16, 0.45]

[0.41, 0.84]

[0.62, 0.99]

[2023.9, 2028.1]

[2024, 2059]

[2008, 2320]

geomean:

0.30

0.62

0.80

2026.00

2041

2163

geo odds:

0.30

0.63

0.82

Epistemic status:

For Samotsvety track-record see:/

Note that this track record comes mostly from questions about geopolitics and technology that resolve within 12 months.

Most forecasters have at least read Joe Carlsmith’s report on AI x-risk, Is “Power-Seeking AI an Existential Risk?”. Those who are short on time may have just skimmed the report and/or watched the presentation. We discussed the report section by section over the course of a few weekly meetings.

Note also that there might be selection effects at the level of which forecasters chose to participate in this exercise; for example, Samotsvety forecasters who view AI as an important/interesting/etc. topic could have self-selected into the discussion.

(Though, the set of forecasters who participated this time and participated last time is very similar.)

Update from our previous estimate

The last time we publicly elicited a similar probability from our forecasters, we were at 32% that AGI would be developed in the next 20 years (so by late 2042); and at 73% that it would be developed by 2100. These are a bit lower than our current forecasts. The changes since then can be attributed to

We have gotten more time to think about the topic, and work through considerations and counter-considerations, e.g., the extent to which we should fear selection effects in the types of arguments to which we are exposed.

Some of our forecasters still give substantial weight to more skeptical probabilities coming from semi-informative priors, from Lap...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Status" can be corrosive; here's how I handle it, published by Akash on January 24, 2023 on LessWrong.

Do you ever think about your social status? I do, and sometimes I find it corrosive.

Examples of thoughts that might come up:

Should I share X doc/idea with Alice? Her feedback might be useful, but it might make her think worse of me.

Why did my friend Bob get invited to X event and I didn’t?

I wonder if Carol has a negative impression of me.

David responds to my friend but not me. Why?

In many situations, these thoughts are helpful. For example, it might be good to occasionally examine if “Carol” has a negative impression of me, especially if Carol is someone I respect, because it might help me find areas where I can grow.

But sometimes, these thoughts are unproductive, limit my thinking, or distract me from more important tasks at hand.

In the rest of this post, I’ll describe some techniques/frames that I’ve found helpful when I notice status coming up in corrosive ways.

Note: If you haven’t read “PR” is corrosive; “reputation" is not, I suggest reading that first.

My Rob Bensinger shoulder model

My Rob Bensinger shoulder represents integrity, honesty, and transparency. He tells me things like:

What do you actually believe? How can you communicate it accurately?

What kind of community do you want to foster? Do you want to be in a community where people are afraid to voice their true opinions?

Sharing our beliefs and ideas openly is how we get closer to the truth.

“Be the sort of person internally who you would find most admirable and virtuous. Be brave, be thoughtful, be discerning, be honest, be honorable, be fair, be compassionate, be trustworthy; and insofar as you’re not those things, be honest about it.” -- Rob

When I talk to my Rob Bensinger shoulder model, he reminds me that the options are not merely “think about how others perceive you” vs. “don’t pay attention at all to your self-presentation.” There is a nice third option which looks something like “imagine a person who is scoring well on a bunch of virtuous axes, and aim to be that person. Be honest, share your views, and aim for integrity rather than likability.”

Habryka’s comment about identifying as an EA

When it comes to status concerns, my Habryka shoulder model is based on one comment he wrote in 2021. He reminds me to notice if my attachment to the community is harming my ability to think and act in the world. And he reminds me that identifying strongly as part of the community (and carrying the weight of thinking about how my actions will reflect the community or be perceived by the community) is ultimately my choice.

An excerpt from his comment:

I have found that when I identified as an EA, I had a lot more unproductive critical voices in my head that prevented me from considering a lot of potentially good ideas, and it exposed me to a lot of people who would get angry at me if I did anything that "damaged the reputation of the movement". After many years of actively carrying EA as part of my identity, I had noticed that my ability to take directed action in the world had very greatly atrophied, I was much more anxious and risk-averse, and it took me at least two years of internally distancing myself quite a lot from the EA-identity cluster before I felt like I could have novel ideas again and start working on ambitious projects again.

See also: Use your identity carefully

The status equation

Someone told me that they see status through the following equation:

When thinking about status, beware of spending too much time thinking about the third thing. Use the part of you that cares about status to improve your thinking, learn more, understand the world more carefully. Use the part of you that cares about status to launch new projects, form new partnerships, work harder, and find ways to impro...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum.

This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.)

Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be).

List of Lethalities

“Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again”

(Evolution) (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups.

Strictly separately, it seems to me that people draw rather strong inferences from a rather loose analogy with evolution. I think that (evolution) (human values) is far less informative for alignment than (human reward circuitry) (human values). I don’t agree with a strong focus on the former, given the latter is available as a source of information.

We want to draw inferences about the mapping from (AI reward circuitry) (AI values), which is an iterative training process using reinforcement learning and self-supervised learning. Therefore, we should consider existing evidence about the (human reward circuitry) (human values) setup, which (AFAICT) also takes place using an iterative, local update process using reinforcement learning and self-supervised learning.

Brain architecture and training is not AI architecture and training, so the evidence is going to be weakened. But for nearly every way in which (human reward circuitry) (human values) is disanalogous to (AI reward circuitry) (AI values), (evolution) (human values) is even more disanalogous! For more on this, see Quintin's post.

Lethality #18: “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do; even if it ends up perfectly inner-aligned on that reward signal, or learning some concept that exactly corresponds to 'wanting states of the environment which result in a high reward signal being sent', an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).”

My summary: Sensory reward signals are not ground truth on the agent’s alignment to our goals. Even if you solve inner alignment, you’re still dead.

My response: We don’t want to end up with an AI which primarily values its own reward, because then it wouldn’t value humans. Beyond that, this item is not a “central” lethality (and a bunch of these central-to-EY lethalities are in fact about outer/inner). We don’t need a function of sensory input which is safe to maximize, that’s not the function of the reward signal. Reward chisels cognition. Reward is not necessarily—nor do we want it to be—a ground-truth signal about alignment.

Lethality #19: “Insofar as the current paradigm works at all, the on-paper design properties say that it only works for aligning on known direct functions ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum.

This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.)

Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be).

List of Lethalities

“Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again”

(Evolution) (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups.

Strictly separately, it seems to me that people draw rather strong inferences from a rather loose analogy with evolution. I think that (evolution) (human values) is far less informative for alignment than (human reward circuitry) (human values). I don’t agree with a strong focus on the former, given the latter is available as a source of information.

We want to draw inferences about the mapping from (AI reward circuitry) (AI values), which is an iterative training process using reinforcement learning and self-supervised learning. Therefore, we should consider existing evidence about the (human reward circuitry) (human values) setup, which (AFAICT) also takes place using an iterative, local update process using reinforcement learning and self-supervised learning.

Brain architecture and training is not AI architecture and training, so the evidence is going to be weakened. But for nearly every way in which (human reward circuitry) (human values) is disanalogous to (AI reward circuitry) (AI values), (evolution) (human values) is even more disanalogous! For more on this, see Quintin's post.

Lethality #18: “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do; even if it ends up perfectly inner-aligned on that reward signal, or learning some concept that exactly corresponds to 'wanting states of the environment which result in a high reward signal being sent', an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).”

My summary: Sensory reward signals are not ground truth on the agent’s alignment to our goals. Even if you solve inner alignment, you’re still dead.

My response: We don’t want to end up with an AI which primarily values its own reward, because then it wouldn’t value humans. Beyond that, this item is not a “central” lethality (and a bunch of these central-to-EY lethalities are in fact about outer/inner). We don’t need a function of sensory input which is safe to maximize, that’s not the function of the reward signal. Reward chisels cognition. Reward is not necessarily—nor do we want it to be—a ground-truth signal about alignment.

Lethality #19: “Insofar as the current paradigm works at all, the on-paper design properties say that it only works for aligning on known direct functions ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Movie Review: Megan, published by Zvi on January 23, 2023 on LessWrong.

I will follow my usual procedure here – a fully spoiler-free 1-bit review, then a mostly spoiler-free short review to determine if you should see it, then a fully detailed spoiler-filled analysis.

Fully Spoiler-Free 1-Bit Review – Should You See Megan In Theaters, Yes or No?

Yes.

Mostly Spoiler-Free Review In Brief

The preview for Megan had me filing the movie as ‘I don’t have to see this and you can’t make me.’ Then Scott Aaronson’s spoiler-filled review demanded a response and sparked my interest in various ways, causing me to see the movie that afternoon.

After an early setback, Megan exceeded expectations. I was as surprised as you are.

As entertainment, Megan is a Tier 3 movie1, or about 4 out of 5 stars on the old Netflix scale. The 72 on Metacritic is slightly low. It’s good.

Megan has intelligent things to say and holds up well, which is why this post exists. I am recommending it both as a good time and a future reference point.

If you do not like the genre this movie is in, and also are not interested in issues surrounding AI, you can safely skip it.

Full SPOILERIFIC Analysis

This analysis is intended to make sense for people who don’t have a background in the problem that AI, by default, kills everyone. If you already know about that, forgive the belaboring and butchering.

Premise!

The basic plot of Megan is:

AI Doll Megan is built, told to protect child from ‘physical and emotional harm.’

Megan learns to kill people to achieve its objective function.

When people notice, Megan fights against attempts to shut it down.

Or in more detail:

Cady’s parents die in a car accident. She is about 8.

Cady’s aunt Gemma is assigned as guardian.

Gemma works at Explicitly Not Hasbro Toy Company and is secretly the World’s Greatest AI expert and robot building badass.

Gemma uses a previous toy as a spy to collect training data on child interactions. She plans to use this to create lifelike AI dolls. Development still has some issues.

She tells her boss. Boss says ‘I don’t want to hear about this’ on both counts and wants her back to work on simple $50 virtual pet designs.

After Cady shows interest, Gemma builds the AI robot doll, Megan, to serve as Cady’s companion and toy. At home. In a week.

Megan is a ‘generative model’ with a ‘constant learning function and focus on self-improvement.’

Megan imprints Cady as her ‘primary user’ and Gemma gives Megan an objective function to ‘protect Cady from physical and emotional harm.’

Megan asks about death. Gemma says not to make a big deal out of it, ‘everyone dies at some point.’

Megan is told to obey Gemma, says Gemma is now a second primary user. Later events will call Megan’s statement here into question.

In various ways clear to both us and to Megan, it is clear Gemma often chooses her job over Cady’s emotional state.

Megan bonds a lot with Cady, in the way she is supposed to be bonding with Gemma. Cady loves Megan and only wants to spend time with Megan.

Megan kills the neighbors’ out-of-control dog, hides the body.

Megan is seen clearly growing in various knowledge and capabilities.

Megan is seen becoming increasingly resistant to being shut down.

Gemma presents Megan to Not Hasbro as Next Big Thing. They agree, plan big announcement. They have to move quickly before they get scooped.

We see an idiot low level employee steal all the design specs from the company server a few minutes later, through absolutely no security.

A bully hurts Cady and then, when Megan shows up, attacks Megan while Megan pretends to be a normal doll. Megan then springs to life, overpowers him, pulls off part of his ear, says ‘this is the part where you run’ and the boy runs onto a country road and is killed by a car.

Owner of the dog, also an out of control vicious animal but also...