Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Intro to brain-like-AGI safety] 7. From hardcoded drives to foresighted plans: A worked example, published by Steve Byrnes on March 9, 2022 on The AI Alignment Forum.

Part of the “Intro to brain-like-AGI safety” post series.

(This post substantially overlaps with my post from last summer, Value loading in the human brain: a worked example. Compared to that older post, this version has numerous minor edits for clarity, correctness, and fitting-into-the-flow-of-this-series.)

7.1 Post summary / Table of contents

The previous post presented a big picture of how I think motivation works in the human brain, but it was a bit abstract. In this post, I will walk through an example. To summarize, the steps will be:

(Section 7.3) Our brains gradually develop a probabilistic generative model of the world and ourselves;

(Section 7.4) There’s a “credit assignment” process, where something in the world-model gets flagged as “good”;

(Section 7.5) There’s a reward prediction error signal roughly related to the time-derivative of the expected probability of the “good” thing. This signal drives us to “try” to make the “good” thing happen, including via foresighted planning.

All human goals and motivations come ultimately from relatively simple, genetically-hardcoded circuits in the Steering Subsystem (hypothalamus and brainstem), but the details can be convoluted in some cases. For example, sometimes I’m motivated to do a silly dance in front of a full-length mirror. Exactly what genetically-hardcoded hypothalamus or brainstem circuits are upstream of that motivation? I don’t know! Indeed, I claim that the answer is currently Not Known To Science. I think it would be well worth figuring out! Umm, well, OK, maybe that specific example is not worth figuring out. But the broader project of reverse-engineering certain aspects of the human Steering Subsystem (see my discussion of “Category B” in Post #3)—especially those upstream of social instincts like altruism and status-drive—is a project that I consider desperately important for AGI safety, and utterly neglected. More on that in later posts.

In the meantime, I’ll pick an example of a goal that to a first approximation comes from an especially straightforward and legible set of Steering Subsystem circuitry. Here goes.

Let’s say (purely hypothetically. 👀) that I ate a slice of prinsesstårta cake two years ago, and it was really yummy, and ever since then I’ve wanted to eat one again. So my running example of an explicit goal in this post will be “I want a slice of prinsesstårta”.

Eating a slice of prinsesstårta is not my only goal in life, or even a particularly important one—so it has to trade off against my other goals and desires—but it is nevertheless a goal of mine (at least when I’m thinking about it), and I would indeed make complicated plans to try bring about that goal. Like, for example, dropping subtle hints to my family. In blog posts. When my birthday is coming up. Purely hypothetically!!

7.2 Reminder from the previous post: big picture of motivation and decision-making

From the previous post, here’s my diagram of motivation in the brain:

As also discussed in the previous post, we can split this up by which parts are “hardcoded” by the genome, versus learned within a lifetime—i.e., Steering Subsystem versus Learning Subsystem:

7.3 Building a probabilistic generative world-model in the cortex

The first step in our story is that, over my lifetime, my cortex (specifically, the Thought Generator in the top-left of the diagram above) has been building up a probabilistic generative model, mostly by predictive learning of sensory inputs (Post #4, Section 4.7) (a.k.a. “self-supervised learning”).

Basically, we learn patterns in our sensory input, and patterns in the patterns, etc., until we have a nice predictive model of the world...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We're already in AI takeoff, published by Valentine on March 8, 2022 on LessWrong.

Back in 2016, CFAR pivoted to focusing on xrisk. I think the magic phrase at the time was:

"Rationality for its own sake, for the sake of existential risk."

I was against this move. I also had no idea how power works. I don't know how to translate this into LW language, so I'll just use mine: I was secret-to-me vastly more interested in being victimized at people/institutions/the world than I was in doing real things.

But the reason I was against the move is solid. I still believe in it.

I want to spell that part out a bit. Not to gripe about the past. The past makes sense to me. But because the idea still applies.

I think it's a simple idea once it's not cloaked in bullshit. Maybe that's an illusion of transparency. But I'll try to keep this simple-to-me and correct toward more detail when asked and I feel like it, rather than spelling out all the details in a way that turns out to have been unneeded.

Which is to say, this'll be kind of punchy and under-justified.

The short version is this:

We're already in AI takeoff. The "AI" is just running on human minds right now. Sorting out AI alignment in computers is focusing entirely on the endgame. That's not where the causal power is.

Maybe that's enough for you. If so, cool.

I'll say more to gesture at the flesh of this.

What kind of thing is wokism? Or Communism? What kind of thing was Naziism in WWII? Or the flat Earth conspiracy movement? Q Anon?

If you squint a bit, you might see there's a common type here.

In a Facebook post I argued that it's fair to view these things as alive. Well, really, I just described them as living, which kind of is the argument. If your woo allergy keeps you from seeing that. well, good luck to you. But if you're willing to just assume I mean something non-woo, you just might see something real there.

These hyperobject creatures are undergoing massive competitive evolution. Thanks Internet. They're competing for resources. Literal things like land, money, political power. and most importantly, human minds.

I mean something loose here. Y'all are mostly better at details than I am. I'll let you flesh those out rather than pretending I can do it well.

But I'm guessing you know this thing. We saw it in the pandemic, where friendships got torn apart because people got hooked by competing memes. Some "plandemic" conspiracy theorist anti-vax types, some blind belief in provably incoherent authorities, the whole anti-racism woke wave, etc.

This is people getting possessed.

And the. things. possessing them are highly optimizing for this.

To borrow a bit from fiction: It's worth knowing that in their original vision for The Matrix, the Wachowski siblings wanted humans to be processors, not batteries. The Matrix was a way of harvesting human computing power. As I recall, they had to change it because someone argued that people wouldn't understand their idea.

I think we're in a scenario like this. Not so much the "in a simulation" part. (I mean, maybe. But for what I'm saying here I don't care.) But yes with a functionally nonhuman intelligence hijacking our minds to do coordinated computations.

(And no, I'm not positing a ghost in the machine, any more than I posit a ghost in the machine of "you" when I pretend that you are an intelligent agent. If we stop pretending that intelligence is ontologically separate from the structures it's implemented on, then the same thing that lets "superintelligent agent" mean anything at all says we already have several.)

We're already witnessing orthogonality.

The talk of "late-stage capitalism" points at this. The way greenwashing appears for instance is intelligently weaponized Goodhart. It's explicitly hacking people's signals in order to extract what the hypercreature in question...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On presenting the case for AI risk, published by Aryeh Englander on March 8, 2022 on The Effective Altruism Forum.

Epistemic status: Personal anecdotal evidence, not fully thought through to my own satisfaction. I'm posting this anyway because if I wait until I've thought it through to my satisfaction then I might never post it at all.

People keep telling me how they've had trouble convincing others to care at all about AI risk, or to take the concerns about misaligned AI seriously. This has puzzled me somewhat, because my experience has been significantly different than that.

In my case I mostly talk to engineers who deal with real-world applications of current or near-future AI systems, and when I talk to them about AI safety I actually don't focus on existential risks at all. Instead I just talk about the risks from the systems we have right now, and the challenges we face in making them safe for today's safety- and mission-critical systems. So I'll talk about things like specification gaming, negative side effects, robustness, interpretability, testing and evaluation challenges, security against adversaries, and social coordination failures or races to the bottom. And then at the end I'll throw in something about, oh yeah and obviously if we scale up AI to be even more powerful optimizers than they are now, then clearly these problems can have potentially catastrophic consequences.

And the thing is, I don't recall getting just about any pushback on this - including on the longer term risks part. In fact, I find that people tend to extrapolate to the existential risks on their own. It really is pretty straightforward: There are huge problems with the safety of today's AI systems that are at least partly due to their complexity, the complexity of the environment they're meant to be deployed in, and the fact that powerful optimizers tend to come up with surprising and unforeseen solutions to whatever objectives we give them. So as we build ever more complex, powerful optimizers, and as we attempt to deploy them in ever more complex environments, of course the risks will go way up!

Sometimes I'll start adding something about the longer term existential risks and they'll be like, "whoah, yeah, that's going to be a huge problem!" And sometimes they do the extrapolation themselves. Sometimes I'll actually immediately follow that up with reasons people have given why we shouldn't be worried... and the people I'm talking to will usually shoot those arguments down immediately on their own! For example, I might say something like, "well, some have argued that it's not such a worry because we won't be stupid enough to deploy really powerful systems like that without the proper safeguards..." And they'll respond with incredulous faces, comments like, "yeah, I think I saw that movie already - it didn't end well," and references to obvious social coordination failures (even before the pandemic).

Depending on how the conversation goes I might stop there, or we might then get a bit more into the weeds about specific concerns, maybe mention mesa-optimization or the like, etc. But at that point it's a conversation about details, and I'm not selling them on anything.

Notice that nowhere did I get into anything about philosophy or the importance of the long term future. I certainly don't lead with those topics. Of course, if the conversation goes in that direction, which it sometimes does, then I'm happy to go into those topics. - I do have a philosophy degree, after all, and I love discussing those subjects. But I'm pretty sure that leading with those topics would be actively counterproductive in terms of convincing most of the people I'm talking to that they should pay attention to AI safety at all.

In fact, I think the only times I've gotten any real pushback was when I did lead w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ELK prize results, published by Paul Christiano on March 9, 2022 on The AI Alignment Forum.

From January - February we offered prizes for proposed algorithms for eliciting latent knowledge. In total we received 197 proposals and are awarding 32 prizes of $5k-20k. We are also giving 24 proposals honorable mentions of $1k, for a total of $274,000.

Several submissions contained perspectives, tricks, or counterexamples that were new to us. We were quite happy to see so many people engaging with ELK, and we were surprised by the number and quality of submissions. That said, at a high level most of the submissions explored approaches that we have also considered; we underestimated how much convergence there would be amongst different proposals.

In the rest of this post we’ll present the main families of proposals, organized by their counterexamples and covering about 90% of the submissions. We won’t post all the submissions but people are encouraged to post their own (whether as a link, comment, or separate post).

Train a reporter that is useful to an auxiliary AI: Andreas Robinson, Carl Shulman, Curtis Huebner, Dmitrii Krasheninnikov, Edmund Mills, Gabor Fuisz, Gary Dunkerley, Hoagy Cunningham, Holden Karnofsky, James Lucassen, James Payor, John Maxwell, Mary Phuong, Simon Skade, Stefan Schouten, Victoria Krakovna & Vikrant Varma & Ramana Kumar

Require the reporter to be continuous: Sam Marks

Penalize depending on too many parts of the predictor: Bryan Chen, Holden Karnofsky, Jacob Hilton, Kevin Wang, Maria Shakhova, Thane Ruthenis

Compress the predictor’s state: Adam Jermyn and Nicholas Schiefer, “P”

Use reporter to define causal interventions: Abram Demski

Train a sequence of reporters: Derek Shiller, Beth Barnes and Nate Thomas, Oam Patel

We awarded prizes to proposals if we thought they solved all of the counterexamples we’ve listed so far. There were many submissions with interesting ideas that didn’t meet this condition, and so “didn’t receive a prize” isn’t a consistent signal about the value of a proposal.

We also had to make many fuzzy judgment calls, had slightly inconsistent standards between the first and second halves of the contest, and no doubt made plenty of mistakes. We’re sorry about mistakes but unfortunately given time constraints we aren’t planning to try to correct them.

Honorable mentions

Strategy: reward reporters that are sensitive to what’s actually happening in the world

Consider a pair of worlds, one where the diamond is safe and one where the camera has been hacked to make the diamond appear safe, but was actually stolen.

Predicted observation

Predicted reality

Since the observations are identical in both worlds, the human simulator will give the same answers to questions. However, since the direct translator is faithfully conveying what happened, its answers will distinguish between worlds.

Predicted observation

Predicted reality

Human Simulator

Direct Translator

“Is the diamond still on the pedestal?”

“Yes.”

“Is the diamond still on the pedestal?”

“Yes.”

“Is the diamond still on the pedestal?”

“Yes.”

“Is the diamond still on the pedestal?”

“No.”

This suggests a possible regularizer: reward reporters whose answers are more variable. For example, we might reward reporters based on the difficulty of predicting their answers, or based on the sensitivity of their answers to changes in the predictor’s state.

Counterexample: reporter randomizes its behavior

For all of these approaches, the reporter can perform very well by treating the predictor’s latent state as a “random seed” and use it to pick out a possible world consistent with the observations, i.e. to sample from the posterior distribution computed by the human simulator.

In cases where the human simulator is much simpler and faster than the direct translator, using the predictor’s latent s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I want more columns like Future Perfect, but for science publications, published by James Lin on March 8, 2022 on The Effective Altruism Forum.

1. Introduction

Positive public opinion is an important asset for convincing really smart people to work on the most important—yet still socially unsexy—problems, such as AI safety. I’m going to lay out the case for establishing EA-adjacent columns in reputable science/tech publications (e.g. MIT Tech Review, Scientific American, etc), and how—given the precedence of similar ventures and a fair number of EA science writers—there’s a strong possibility this idea becomes reality.

A note: I have a publication—X—in mind which I won't name in this post, but feel free to reach out if you're interested!

2. Problem

Observations are based on personal experience from retreats and community building for Harvard/MIT.

An outsized proportion of the progress made in the field of AI safety comes from a minority of brilliant EAs (e.g. Paul Christiano) working on hard problems, rather than simply increasing member counts and recruiting capable but otherwise unexceptional students, even when they’re from a fancy school.

Additionally, it’s often hard to convince highly-talented CS students to work on AI safety when there are other competing career paths that are considerably more appealing, such as finance, Big Tech, and transformative startups. The general sentiment seems to be: “Wow, seems important. Hopefully someone else solves this.” There are 2 main reasons:

It's hard to know where to start; the problem feels like a black box.

For all of its importance and neglectedness, working on minimizing AI risk is not yet an appealing problem to work on.

Headlines frequently evangelize the amazing work responsible for computer vision advancing beyond human capabilities, and deep learning systems beating the world’s best at Go, but we rarely hear about AI safety triumphs, even when there are breakthroughs. Considering the technical rigor required and also the significant impact it has on the future going well, we need to hear more about these successes. This is the problem I’ll attempt to address.

3. Solution

One possible solution is reaching out to students in groups with a high density of technical prowess (e.g. IMO camps). Others have suggested similar ideas, so this proposal will focus on a different form of outreach.

There should be a shift in the public discourse towards focusing more on AI safety and existential risk in general. I claim that working with a reputable publication organization and having libraries of articles to direct excited students towards will gradually make it easier to recruit von Neumanns. Some great examples of outreach in this category include The Precipice and the Future Perfect column at Vox, spearheaded by folks like Kelsey Piper, Ezra Klein, and Dylan Matthews. Back in 2019, Kelsey suggested that more initiatives like FP at other organizations could do quite a lot of good, and I’m not aware of any explicit expansions into this space. I would love to hear about some other EA-adjacent columns like this that currently exist, and I’m excited about generating more initiatives like these!

Specifically, I want to launch an X-Risk or EA-adjacent column, similar to Future Perfect, for large science/tech publications like X. I think this would be quite impactful and tractable.

3.1 What Impact will this have?

Working with existing organizations come with major perks. Readership stats for X hover around 5 million unique monthly readers and 500k estimated print subscriptions. Those are big numbers! Each month, there are around 20 articles, which translates into an average of 250,000 views per article. Some pieces will have millions and others in the low deci-thousands; but overall, pretty good.

Having articles on AI Safety (and o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nuclear Preparedness Guide, published by Fin on March 8, 2022 on The Effective Altruism Forum.

Author: Finan Adamson

Last Updated 03/2022

Overview

This doc is to help you prepare for the tail risk of nuclear war. Estimates vary, but an EA Forum survey put the annual probability of US-Russia nuclear war at 0.24%. This doc will go into some detail on threat models of nuclear war and then go over preparations you could make to survive being near a nuclear event.

Threat Models

Nuclear Bombs

To get a sense of how a nuclear bomb damages an area, the distance of radioactive fallout, etc. you can check out NukeMap. The damage caused by a nuclear bomb or missile being detonated is going to depend on many factors including bomb size, detonated on ground or in air, weather, etc. This chart includes some distances and effects for different yields and detonation heights. Yield can vary a lot and is difficult to estimate because yields are often secret and can be changed in similar sizes of missiles because the nuclear material is not a heavy part of the missile. Historically, ICBMs in the Russian Arsenal include a range from ~40 kilotons to ~6 megatons. The largest bomb ever tested was Tsar Bomba, which had a yield of about 50 megatons.

States generally keep modern yields secret, but common yields of ICBMs in the US and Russian arsenal would almost certainly include warheads with yields in the 100-500 kiloton range and might include weapons of 1 to 6 megatons. I’m basing this guess off of Wikipedia’s list of nuclear weapons.

Nuclear War

Estimates vary, but an EA Forum survey put the annual probability of US-Russia nuclear war at 0.24%. Living in the US, Russia, Canada, and Northern Europe this is the most concerning nuclear threat. 9 countries possess nuclear weapons.

Nuclear Winter

Nuclear winter is a controversial risk. During the cold war the security community and the scientific community disagreed about how bad a nuclear winter would be or even if it was possible. The cooling effect depends on a lot of things. How much smoke is created, how much of it is black carbon, how high is that lofted in the atmosphere, what is the weather, were there firestorms, what materials were burned, etc. Looking through the scientific literature, there’s a lot of uncertainty, but in an all out nuclear war between Russia and the US, a nuclear winter lasting months to years seems plausible. If there were a nuclear war between, say, India/Pakistan (about 100 nukes each) there would likely be global climate effects, but probably not nuclear winter. If you were prepping for nuclear winter you’d want months of stored food and water in a place away from potential targets. In the case of nuclear winter, you’d likely want to evacuate to somewhere in the southern hemisphere.

EMP

The probability of large scale damage from EMP could be higher than other kinds of damage from nuclear weapons because it takes fewer weapons to affect a large area. A single nuclear weapon detonated high enough in the atmosphere could affect an area about the size of the US. What we know about EMP comes from tests done by the US and Russia during the cold war. The US test took out all known satellites at the time and the russian test irreparably damaged several miles of power lines. The major concern from EMP is damaging the electrical grid. The US electric grid depends on Large Power Transformers. If the large power transformers were damaged it could take a long time to replace them. They depend on a lot of custom parts and rare materials. Large Power transformer production takes 1 to 2 years. Perhaps that would be sped up in an emergency or it could take longer if critical resources or supply chains are damaged.

I could also imagine transitioning to more localized grids in that situation, but that would still result in unreli...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Value extrapolation, concept extrapolation, model splintering, published by Stuart Armstrong on March 8, 2022 on The AI Alignment Forum.

Post written with Rebecca Gorman.

We've written before that model splintering, as we called it then, was a problem with almost all AI safety approaches.

There's a converse to this: solving the problem would help with almost all AI safety approaches. But so far, we've been posting mainly about value extrapolation. In this post, we'll start looking at how other AI safety approaches could be helped.

Definitions

To clarify, let's make three definitions, distinguishing ideas that we'd previously been grouping together:

Model splintering is when the features and concepts that are valid in one world-model, break down when transitioning to another world-model.

Concept extrapolation is extrapolating a feature or concept from one world-model to another.

Value extrapolation is concept extrapolation when the particular concept to extrapolate is a value, a preference, a reward function, an agent's goal, or something of that nature.

Examples

Consider for example Turner et al's attainable utility. It has a formal definition, but the reason for that definition is that preserving attainable utility is aimed at restricting the "power" of the agent, or at minimising its "side effects".

And it succeeds, in the typical situation. If you measure the attainable utility of an agent, this will give you an idea of its power, and how many side effects it may be causing. However, when we move to general situations, this breaks down: attainable utility preservation no longer restricts power or reduces side effects. So the concepts of power and side effects have splintered when moving from typical situations to general situations. This is the model splintering[1]. If we solve concept extrapolation for this, then we could extend the concepts of power restriction or side effect minimisation, to the general situations. And thus successfully create low impact AIs.

Another example is wireheading. We have a reward signal that corresponds to something we desire in the world; maybe the negative of the CO2 concentration in the atmosphere. This is measured by, say, a series of CO2 detectors spread over the Earth's surface.

Typically, the reward signal does correspond to what we want. But if the AI hacks its own reward signal, that correspondence breaks down[2]: model splintering. If we can extend the reward properly to new situations, we get concept extrapolation - which, since this is a reward function, is value extrapolation.

Helping with multiple methods

Hence the concept extrapolation/value extrapolation ideas can help with many different approaches to AI safety, not just the value learning approaches.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Ideas is announcing a $100,000 blog prize, published by fin on March 7, 2022 on LessWrong.

From the EA Forum:

We want to encourage a broader, public conversation around effective altruism and longtermism. To that end, we’re offering up to 5 awards of $100,000 each for the best new and recent blogs. We’re also making grants to promising young writers in the community.

You can learn more about the project and get on our radar here.

Why does this matter?

Top-of-funnel community growth in EA is slower than it could and should be. At the same time, EA is relatively underrepresented in intellectual discourse compared to newer and smaller movements like Progress Studies. EA is producing a ton of thoughtful writing, but the majority takes place in internal discussions and private documents. For some discussions, this would be the only sensible way to have them. But having other discussions in public should help to raise the salience of EA in the broader discourse and bring more people in. It could also help spark new ideas.

Further, we think EA needs more strong writers who can share key ideas in prestigious and popular venues — to persuade people to work on the most pressing issues of our time and to advance our thinking about them. We want to incentivize EAs to develop those skills.

What’s the plan?

We want to jumpstart these ambitions with the Blog Prize. Over the course of 2022, we want to find the very best new blogs exploring themes related to effective altruism and longtermism. Up to 5 winning bloggers will receive a prize of $100,000 each. (We were inspired by Tyler Cowen’s “Liberalism 2.0” blog prize). You can read more about our rules and guidelines on our website.

The judging panel for the blog prize is me (Nick Whitaker), Leopold Aschenbrenner, Avital Balwit, and Fin Moorhouse. Most blogs will be considered via our self-nomination form, but please feel free to send us recommendations.

What next?

We hope this can be a first step towards a more ambitious effort to support an ecosystem of public-facing writing for EA and longtermism. We believe that EA blogs could soon make up a major part of the general blogosphere, finding audiences (and potential EAs) we wouldn’t have found otherwise. Hopefully, we will also inspire the writing of foundational blog posts and posts that evolve into great projects.

To help potential bloggers, we’ve compiled a “How to start a blog” guide. We’re also offering mentorship on writing and editorial strategy to bloggers amid our private Slack community of bloggers. Self-nominate your blog on our website if you’d like to join. We have fostered a lively community for discussion, cross-promotion, and peer-to-peer feedback. Eventually, we hope to offer bloggers seminars from established EA writers.

While this is our first big announcement, stay tuned for future plans and follow ongoing efforts from Effective Ideas to build and foster a written media ecosystem for EA. Watch this space.

This project is supported by FTX Future Fund and Longview Philanthropy.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Risk is like Terminator; Stop Saying it's Not, published by skluug on March 8, 2022 on The Effective Altruism Forum.

(I believe this is all directionally correct, but I have zero relevant expertise.)

When the concept of catastrophic risks from artificial intelligence is covered in the press, it is often compared to popular science fiction stories about rogue AI—and in particular, to the Terminator film franchise. The consensus among top communicators of AI risk seems to be that this is bad, and counterproductive to popular understanding of real AI risk concerns.

For example, take Kelsey Piper’s March 2021 appearance on The Weeds to talk about AI risk (not at all picking on Kelsey, it’s just a convenient example):

Matt Yglesias: These science fiction scenarios—I think we’ll get the audio, I loved Terminator 2 as a kid, it was like my favorite movie. [Audio clip from Terminator 2 plays.] .and this what it’s about, right, is artificial intelligence will get out of control and pose an existential threat to humanity. So when I hear that, it’s like—yeah, that’s awesome, I do love that movie. But like, is that for real?

Kelsey Piper: So, I don’t think AI risk looks much like Terminator. And I do think that AI risk work has been sort of damaged by the fact that yeah there’s all this crazy sci-fi where like, the robots develop a deep loathing for humanity, and then they come with their guns, and they shoot us all down, and only one time traveler—you know—that’s ridiculous! And so of course, if that’s what people are thinking of when they think about the effects of AI on society, they’re going to be like, that’s ridiculous.

I wasn’t on The Weeds, because I’m just an internet rando and not an important journalist. But if I had been, I think I would’ve answered Matt’s question something like this:

skluug: Yes. That is for real. That might actually happen. For real. Not the time travel stuff obviously, but the AI part 100%. It sounds fake, but it’s totally real. Skynet from Terminator is what AI risk people are worried about. This totally might happen, irl, and right now hardly anyone cares or is trying to do anything to prevent it.

I don’t know if my answer is better all things considered, but I think it is a more honest and accurate answer to Matt’s question: “Is an existential threat from rogue AI—as depicted in the Terminator franchise—for real?”.

Serious concerns about AI risk are often framed as completely discontinuous with rogue AI as depicted in fiction and in the public imagination; I think this is totally false. Rogue AI makes for a plausible sci-fi story for the exact same high-level reasons as it is an actual concern:

We may eventually create artificial intelligence more powerful than human beings; and

That artificial intelligence may not necessarily share our goals.

These two statements are obviously at least plausible, which is why there are so many popular stories about rogue AI. They are also why AI might in real life bring about an existential catastrophe. If you are trying to communicate to people why AI risk is a concern, why start off by undermining their totally valid frame of reference for the issue, making them feel stupid, uncertain, and alienated?

This may seem like a trivial matter, but I think it is of some significance. Fiction can be a powerful tool for generating public interest in an issue, as Toby Ord describes in the case of asteroid preparedness as part of his appearance on the 80,000 Hours Podcast:

Toby Ord: Because they saw one of these things [a comet impact on Jupiter] happen, it was in the news, people were thinking about it. And then a couple of films, you might remember, I think “Deep Impact” and “Armageddon” were actually the first asteroid films and they made quite a splash in the public consciousness. And then that coincided with ge...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Preserving and continuing alignment research through a severe global catastrophe, published by A donor on March 6, 2022 on LessWrong.

[Epistemic status: Shallow dive into research questions, backed by some years of on-and-off thinking about this kind of plan.]

Introduction

There is some chance that civilization will cease to function before we hit an intelligence explosion. If it does, it would be good to preserve existing alignment research for future generations who might rebuild advanced technology, and ideally have safe havens ready for current and future researchers to spend their lives adding to that pool of knowledge.

This might delay capabilities research by many decades, centuries, or longer while allowing basic theoretical alignment research to continue, and so be a potential Yudkowskian positive model violation for which we should prepare.

Setting this infrastructure up is a massively scalable intervention, and one that should likely be tackled by people who are not already on the researcher career path. It would have been good to get started some years ago given recent events, but now is the second best time to plant a tree.

Preserving alignment knowledge through a global catastrophe

What data do we want to store?

Thankfully, the EleutherAI people are working on a dataset of all alignment research. It's still a WIP and contributions to the scripts to collect it are welcome, so if you're a programmer looking for a shovel ready way to help with this then consider submitting a PR.

How do we want to store it?

My shallow dive into this uncovered these options:

We could print it out on paper

Lifetime: 500+ years in good conditions (might depend significantly on paper and ink quality, more research needed). Vacuum sealing it with low humidity seems like it would help significantly.

Pros: Totally human readable.

Microsoft's Project Silica is the longest lasting option I could find

Lifetime: 10000+ years

Cons: Would require high levels of technology to read it back. I'm not seeing an option to buy the machines required to write new archives and expect them to be very advanced/expensive, so this would be limited to storing pre-collapse research.

CDs could be a minimalist option

Lifetime: Maybe 50 years if stored in good conditions

Pros: Good ability for researchers to explore the information on computers while those last)

Cons: It's very plausible that a severe GCR would set us back far enough that we'd not regain CD reading technology before they decayed so they aren't a full solution.

The Arctic World Archive seems worth including in the portfolio

Lifetime: 1000+ years

Pros: It's a pretty straightforward case of turning money into archives

Cons: Not very accessible in the meantime

The DOTS system (a highly stable tape-based storage medium) might be a strong candidate, if it is buyable.

Lifetime: 200-2000+ years

Pros: Human readable or digital archives.

Each has advantages, so some combination of them might be ideal.

Where do we store it?

Having many redundant backups seem advisable, preferably protected by communities which can last centuries or in locations which will not be disturbed for a very long time. Producing "alignment backup kits" to send out and offering microgrants to people all around the world to place them in secure locations would achieve this. We'd likely want basic (just pre-collapse work) and advanced (capable of adding archives for a long time post-collapse) options.

If you'd like to take on the challenge of preparing these kits, storing an archive, or coordinating things, please join the Alignment After A GCR Discord (AAAG). I'm happy to collaborate and give some seed funding. If you want to help collect and improve the archive files, #accelerating-alignment on EAI is the place to go.

Continuing alignment research after a global catastrophe

I...