Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Luna Lovegood and the Chamber of Secrets, Part 4: Luna Lovegood and the Chamber of Secrets , published by lsusr.

Luna located the Ravenclaw Common Room on the map. She climbed up to Ravenclaw Tower, then climbed up Ravenclaw Tower. All that separated Luna from making day-one friends was a door with a bronze knocker in the shape of an eagle which spoke riddles appropriate for first-years. She could almost hear muffled sounds of the other children partying. Luna knocked once. The eagle spoke with Rowena Ravenclaw's voice.

"Where is my diadem?"

Luna lacked sufficient hours-until-breakfast to find the Lost Diadem of Ravenclaw. Luna curled up in an alcove close to where heat leaked out of the Common Room. She wrote a thank you note to the house-elves for keeping the floor clean.

"I'm going to call you Wanda," Luna said to her Wrackspurt.

Wanda appeared on the Marauder's Map. Luna clicked her tongue and then spellotaped the Comed-Tea can to her ear so Wanda could feed. Luna's brain went fuzzy. Then she was asleep.

Luna woke to the sound of Ravenclaw going down to breakfast. Luna removed Wanda from her ear. Someone had placed a blanket over Luna while she slept. Luna threw the blanket off before anyone else spotted her and realized she hadn't solved the riddle. She knocked once on the eagle. The riddle hadn't changed.

On her way to breakfast, Luna passed the Forgotten Library at the base of Ravenclaw Tower. She had to eat food because she was a human being. Or she could explore a room that had stood for a thousand years and would probably still be there tomorrow. The entrance to the Forgotten library was a grey heptagon embedded in the wall. Its interior emitted monochromatic grey light. Luna stepped into the Forgotten Library.

Luna stepped out of the Forgotten library. She checked her satchel for quill, parchment and inkwell. Luna wrote "Exploration Journal" at the top of the parchment. Luna stepped into the Forgotten Library.

Luna stepped out of the Forgotten Library. She held a sheet of parchment with "Exploration Journal" at the top. Luna stepped into the Forgotten Library.

Luna stepped out of the Forgotten Library. She had left her exploration journal inside. Luna stepped into the Forgotten Library.

Luna stepped out of the Forgotten Library. She held a blank sheet of parchment. She had missed her first Charms class. If she hurried she could still reach Battle Magic on time. The Marauder's Map showed a shortcut leading from a door disguised as a window to a window disguised as a door. Luna refenestrated herself onto the top floor of prosaic geometric space just outside the Battle Magic classroom.

Professor Lockhart had preserved Former Professor Quirrel's tradition of combining all four Houses into a single class. He had also combined all seven years into a single class. Professors Burbage, Lapsusa, Sinistra, Trelawney, Vector and Yue were there too, plus Headmistress McGonagall, Miss Pince and Madam Pomfrey. Hogwarts had grown an auditorium that rotated with the Sun under a geodesic dome to ensure Gilderoy Lockhart could always be seen in the best possible light. Gilderoy Lockhart's smile shouted I love you to you, personally.

"Before we begin I have a demonstration. Protego," Lockhart said, "On the count of three I want someone to hex me. One."

Gilderoy Lockhart had given this demonstration dozens of times to witches and wizards around the world. Most were not duelists. Those who were tended to be too surprised to cast anything.

This was Gilderoy Lockhart's first interaction with the survivors of Former Professor Quirrel's armies.

A hundred hexes hit Gilderoy Lockhart before he could say "two". The Defense Professor's podium was molten slag. His body resembled a Blast-Ended Skrewt. Purple smoke billowed up from the shattered dome. The professors rushed to his aid. Head...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Luna Lovegood and the Chamber of Secrets, Part 3: Preface, published by lsusr.

"I solemnly swear that I am up to no good."

Fred and George had carried concealed broomsticks at all times since the previous year's troll attack. When they heard Harry Potter's prophecy at the Quidditch final they flew just over the castle walls and then Apparated to the graveyard where they salvaged their map along with several rings, amulets and strange devices.

"Have you visited every room on this map?" Luna asked.

"Excuse me?" George said.

"What part of the school have you never visited?" Luna enunciated each word.

"Did she just insult us?" Fred said.

"Perhaps our reputation is in disrepair," George said.

"What makes you think there exists a single room in the school we hadn't explored by the end of our first year?" Fred said.

"Are you telling me you visited every room in Hogwarts in your first year?" Luna said.

"We make no such claim," George said, "For if were we to make such a claim then we would include not just mere rooms but also secret passages, pocket dimensions, secret dimensions, pocket passages, and docket sassages."

"Surely someone of your reputation must have visited every room, passage, dimension, sassage and chamber by the end of your first year," Luna said.

"Surely," Fred said.

"Had you visited the Chamber of Secrets before Headmistress McGonagall announced its existence today?" Luna said.

Someone said a rude word.

"Where is the Chamber of Secrets?" Luna asked.

"It's this complex of tunnels," George said, "It connects to this painting of Salazar Slytherin to this girls' bathroom and these places over here. This path goes to the Hogsmeade graveyard."

"Has the Chamber of Secrets always been on this map?" Luna asked.

"Yes," Fred said.

"How do you know?" Luna asked.

"I remember it," Fred said, "I just never really noticed it before McGonagall's announcement."

"Are there any other rooms on this map you haven't noticed?" Luna asked.

The students failed to find anything they hadn't found before.

"I have an idea," Luna said, "First we are going to look at this map normally. We are going to list every room and secret passage we know of, including the Chamber of Secrets. We are going to count them. Then you are going to conjure a grid over this map. We are going to count every single room without identifying them. Then we will compare the two numbers."

The two numbers came up exactly the same.

"Wait a minute. I have another idea. Give me that list. Is there any room in this castle you haven't been to?" Luna said.

"No," George said, "We've been everywhere important except the Chamber of Secrets."

"Let's go over the map again," Luna said, "List each room you've been inside."

".and that's the broom closet we trapped Percy in along with his girlfriend, his ex-girlfriend, his ex-ex-girlfriend and Peeves," Fred finished.

Luna tore up little bits of parchment and covered up each room Fred and George had visited. She had covered nearly all of the Marauder's Map. There were just a few unimportant rooms that didn't really count. Then Luna caught herself. This must be what a Muggle-Repelling Charm felt like.

Luna deliberately read off the unimportant rooms she had just nearly written off.

The Stone Citadel (under construction)

The Chamber of Secrets

The Forgotten Library

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Luna Lovegood and the Chamber of Secrets, Luna Lovegood and the Chamber of Secrets - Part 2, published by lsusr.

"I wonder what pulls the boats," Luna said.

"Whatever pulls the horseless carriages must also pull the boats," a first year said.

"But.but.. You can SEE horses pulling the carriages and you CAN'T SEE anything pulling the boats." Luna said.

Luna's Failed Friendships Counter ticked up to four.

"They're called thestrals," said a gentle voice behind her, "Why don' yeh sit with me. I’m Keeper of Keys and Grounds at Hogwarts. I take care of the magical creatures in the Forbidden Forest."

Luna sat in a boat with the gameskeeper.

"You must know all about invisible creatures, Mister," Luna said.

"Call me Hagrid," Hagrid said, "I knew a few. There's threstrals, obviously. Imps can't be seen or heard or remembered. But there's lots of spells that'll move the imps about so you can cage 'em up."

"It sounds like there's no creature in the world you're scared of," Luna said.

"What? I'm just as scared of dangerous creatures as the next fella," Hagrid said.

"Like dragons?" Luna said.

"Dragons aren't dangerous!" said Hagrid.

"Trolls?" Luna said.

"They might pull your arms off but it's nothing personal," Hagrid said.

"Dementors?" Luna said.

"You don' geddit. Dementors aren't dangerous the way a nargle is dangerous," Hagrid said.

"What's a nargle?" Luna asked.

"I shouldn't have said that," Hagrid muttered, "I should NOT have said that."

Headmistress McGonagall gave her first opening speech.

".I am also proud to announce to you that the Chamber of Secrets has been opened. Archaeologists have swept it for traps and curses. Guided tours have been added to your History of Magic curriculums this year."

Luna tuned out the Ministry propaganda. There were new teachers. Lady Yue would be teaching Potions, Professor Susan Lapsusa would be teaching Transfiguration and—oh my god

Gilderoy Lockhart would be teaching Battle Magic.

Gilderoy Lockhart was renowned as the greatest combat wizard of the era—a modern-day David Monroe. Every week the Daily Prophet printed a story where he described an encounter with ghouls, hags, trolls, vampires and werewolves. Rumor was this week he would tell about his encounter with a yeti in Antarctica. He had more confirmed kills of dark wizards than the best Ministry Auror. He had won Witch Weekly's Most Charming Smile Award five times.

Gilderoy' golden hair flowed with painstakingly-crafted effortlessness. He thrust his shoulders back as if he had just saved the world. And that nose

"Lovegood!", Prof. Lapsusa shouted.

Luna skipped to the front of the Great Hall. Students sniggered. This is the price you pay for being the daughter of someone who speaks truth to power Luna thought.

The sorting hat covered Luna's head all the way down to her shoulders. It was cozily quiet inside.

Hmm, said a small voice in her ear. Difficult. Very difficult. Plenty of courage, I see. Not a bad mind, either. There’s talent, oh my goodness, yes – and a nice thirst to prove yourself, now that’s interesting . So where shall I put you?

Do you say that to every girl? Luna thought, Do you call all of us complicated and courageous and talented and

"RAVENCLAW!" shouted that hat.

Food appeared on the golden plates. Luna wondered where it came from.

"Where does this food come from?" she asked a mousy-haired boy next to her.

"It's created from nothing," he said.

"That's magically impossible," Luna said, "Food is one of the five Principal Exceptions to Gamp’s Law of Elemental Transfiguration."

"Then where do you think it comes from?" he said testily.

Luna's Failed Friendships Counter ticked up to five.

"First-years follow me," the Ravenclaw prefect said.

The logical place for the kitchens is under the Great Hall. Luna waited to see what general direction the prefects went and then stalked h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Luna Lovegood and the Chamber of Secrets, Part 1: Luna Lovegood and the Chamber of Secrets, published by lsusr.

Luna Lovegood walked through the barrier between Platforms Nine and Ten to Platform Nine and Three-Quarters. Luna wondered what happened to Platform Nine and One-Half. Numbers like "three quarters" only appear when you divide an integer in half twice in a row.

Luna looked around for someone who might know the answer and spied a unicorn. She wore clothes, walked on two feet and had curly brown hair. None of that fooled Luna. The unicorn radiated peace and her fingernails were made out of alicorn.

"What happened to Platform Nine and One-Half?" Luna asked the unicorn.

"There is no Platform Nine and One-Half," the unicorn replied.

"How do you know?" Luna asked.

"It would have been in Hogwarts: A History," the unicorn replied, "nor is there mention of a Platform Nine and One-Half in Modern Magical History, Important Modern Magical Discoveries, or any other book in the Hogwarts library. There is only a Platform Nine and Three Quarters."

"What about Platform Nine and Seven Eighths?" Luna asked.

"There is no Platform Nine and Seven Eights either." The unicorn turned around and walked away before Luna could ask "How do you know?"

If Platform Nine and Three Quarters does not appear in Muggle libraries then Platform Nine and One-Half is unlikely to appear in wizard libraries, except for double-witches' libraries. The Hogwarts library is not a double-witch library.

"How are you?" a Weasley-headed first-year girl asked Luna.

"I'm trying to find Platform Nine and One-Half. The unicorn told me it doesn't exist. If it does exist then it must be hidden by powerful magics. How are you?" said Luna.

"What unicorn?" the first-year girl asked.

"That one, right there," Luna said, pointing.

The girl invented an excuse to end the conversation.

Luna didn't know how to make friends. She had a vague idea that as a first-year, the Hogwarts Express was a one-time opportunity to do so. She wore a necklace she had painted herself which nobody else seemed to notice. She had brought kettle of home-brewed Comed-Tea, but it had got her jeered out of a compartment.

Nobody was interested in the troll she had smelled at Platform Nine and Three Quarters or her discovery of a lich in the second year or that the Chamber of Secrets had been opened or any of Dark Lord Harry Potter's various plots. The other first-years seemed unfocused and confused.

Confused..

Wrackspurts are invisible creatures that float into your ears and make your brain go fuzzy. The train could be full of them. They could be floating into her ears right now. Luna stuck her index fingers in her ears to block the possible Wrackspurts. The first-years in the nearby compartment looked at Luna as if she were already unpopular.

Wrackspurts are cognitohazardous which means they mess with your thoughts. Luna learned all about Wrackspurts and other cognitohazards in her work on The Quibbler. The most important thing about cognitohazards is to check yourself regularly and figure out if you've already been affected by one.

Luna observed her own mind. Fuzzy? No. Unfocused? No. Confused? No. Wrackspurts had not yet entered her brain. (Unless it already had and was inhibiting her meta-cognition—but she didn't think that was happening.) Luna observed the other students. Maybe they were infected by Wrackspurts or maybe they were behaving normally. It was hard to tell without a Wrackspurt-free baseline to compare them to.

Before she could unplug her ears, Luna had to figure out if there were Wrackspurts roaming the train. But Wrackspurts are invisible. How can she test whether such a thing exists?

Wrackspurts are attracted to people so the safest place to go would be an empty compartment. Smaller would be better since that would decrease the like...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 7: Commentary on AGI Safety from First Principles, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

My AGI safety from first principles report (which is now online here) was originally circulated as a google doc. Since there was a lot of good discussion in comments on the original document, I thought it would be worthwhile putting some of it online, and have copied out most of the substantive comment threads here. Many thanks to all of the contributors for their insightful points, and to Habryka for helping with formatting. Note that in some cases comments may refer to parts of the report that didn't make it into the public version.

Discussion on the whole report

Will MacAskill

Thanks so much for writing this! Huge +1 to more foundational work in this area.

My overall biggest worry with your argument is just whether it's spending a lot of time defending something that's not really where the controversy lies. (This is true for me; I don't know if I'm idiosyncratic.) Distinguish two claims one could argue for:

Claim 1: At some point in the future, assuming continued tech progress, history will have primarily become the story of AI systems doing things. The goals of those AI systems, or the emergent path that results from interactions among these systems, will probably not be what you reading this document want to happen.

I find claim 1 pretty uncontroversial. And I do think that this alone is enough for far more of the world to be thinking about AI than currently is.

But it feels like at least for longtermist EAs trying to prioritise among causes (or for non-longtermists deciding how much to prioritise safety vs speed on AI), the action is much more on a more substantial claim like:

Claim 2: Claim 1 is true, and the point in time at which the transition from a human-driven world to an AI-driven world is in our lifetime, and the transition will be fast, and we can meaningfully affect how this transition goes with very long-lasting impacts, and (on the classic formulations at least) the transition will be to a single AI agent with more power than all other agents combined, and what we should try to do in response to all this is ensure that the AI systems that get built have goals that are the same as the goals of those who design the AI systems.

Each of the new sub-claims in claim 2, I find (highly) controversial. And you talk a little bit about some of these sub-claims, but it's not the focus.

Interested if you think that's an unfair characterisation. Perhaps you see yourself as arguing for something in between Claim 1 and Claim 2.

Richard Ngo

I think it's fair to say that I'm defending claim 1. I think that a lot of people would disagree with it, because:

a) They don't picture AI systems having goals in a way that's easily separable from the goals of the humans who use them; or

b) They think that humans will retain enough power over AIs that the "main story" will be what humans choose to do, even if some AIs have goals we don't like; or

c) They think that it'll be easy to make AIs have the goals we want them to have; or

d) They think that, even if the outcome is not specifically what they want, it'll be within some range of acceptable variation (in a similar way to how our current society is related to our great-great-grandparents').

My thoughts on the remaining parts of claim 2:

a) "The point in time at which the transition from a human-driven world to an AI-driven world is in our lifetime"

OpenPhil are investigating timelines very thoroughly, so I'm happy to defer to them.

b) The transition will be fast.

I make some arguments about this in the "speed of AI development" section. But broadly speaking, I don't want this version of the argument to depend on the claim tha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 6: AGI safety from first principles: Conclusion, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Let’s recap the second species argument as originally laid out, along with the additional conclusions and clarifications from the rest of the report.

We’ll build AIs which are much more intelligent than humans; that is, much better than humans at using generalisable cognitive skills to understand the world.

Those AGIs will be autonomous agents which pursue long-term, large-scale goals, because goal-directedness is reinforced in many training environments, and because those goals will sometimes generalise to be larger in scope.

Those goals will by default be misaligned with what we want, because our desires are complex and nuanced, and our existing tools for shaping the goals of AIs are inadequate.

The development of autonomous misaligned AGIs would lead to them gaining control of humanity’s future, via their superhuman intelligence, technology and coordination - depending on the speed of AI development, the transparency of AI systems, how constrained they are during deployment, and how well humans can cooperate politically and economically.

Personally, I am most confident in 1, then 4, then 3, then 2 (in each case conditional on all the previous claims) - although I think there’s room for reasonable disagreement on all of them. In particular, the arguments I’ve made about AGI goals might have been too reliant on anthropomorphism. Even if this is a fair criticism, though, it’s also very unclear how to reason about the behaviour of generally intelligent systems without being anthropomorphic. The main reason we expect the development of AGI to be a major event is because the history of humanity tells us how important intelligence is. But it wasn’t just our intelligence that led to human success - it was also our relentless drive to survive and thrive. Without that, we wouldn’t have gotten anywhere. So when trying to predict the impacts of AGIs, we can’t avoid thinking about what will lead them to choose some types of intelligent behaviour over others - in other words, thinking about their motivations.

Note, however, that the second species argument, and the scenarios I’ve outlined above, aren’t meant to be comprehensive descriptions of all sources of existential risk from AI. Even if the second species argument doesn’t turn out to be correct, AI will likely still be a transformative technology, and we should try to minimise other potential harms. In addition to the standard misuse concerns (e.g. about AI being used to develop weapons), we might also worry about increases in AI capabilities leading to undesirable structural changes. For example, they might shift the offense-defence balance in cybersecurity, or lead to more centralisation of human economic power. I consider Christiano’s “going out with a whimper” scenario to also fall into this category. Yet there’s been little in-depth investigation of how structural changes might lead to long-term harms, so I am inclined to not place much credence in such arguments until they have been explored much more thoroughly.

By contrast, I think the AI takeover scenarios that this report focuses on have received much more scrutiny - but still, as discussed previously, have big question marks surrounding some of the key premises. However, it’s important to distinguish the question of how likely it is that the second species argument is correct, from the question of how seriously we should take it. Often people with very different perspectives on the latter actually don’t disagree very much on the former. I find the following analogy from Stuart Russell illustrative: suppose we got a message from space telling us that aliens would be ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 5: AGI safety from first principles: Control, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

It’s important to note that my previous arguments by themselves do not imply that AGIs will end up in control of the world instead of us. As an analogy, scientific knowledge allows us to be much more capable than stone-age humans. Yet if dropped back in that time with just our current knowledge, I very much doubt that one modern human could take over the stone-age world. Rather, this last step of the argument relies on additional predictions about the dynamics of the transition from humans being the smartest agents on Earth to AGIs taking over that role. These will depend on technological, economic and political factors, as I’ll discuss in this section. One recurring theme will be the importance of our expectation that AGIs will be deployed as software that can be run on many different computers, rather than being tied to a specific piece of hardware as humans are.[1]

I’ll start off by discussing two very high-level arguments. The first is that being more generally intelligent allows you to acquire more power, via large-scale coordination and development of novel technological capabilities. Both of these contributed to the human species taking control of the world; and they both contributed to other big shifts in the distribution of power (such as the industrial revolution). If the set of all humans and aligned AGIs is much less capable in these two ways than the set of all misaligned AGIs, then we should expect the latter to develop more novel technologies, and use them to amass more resources, unless strong constraints are placed on them, or they’re unable to coordinate well (I’ll discuss both possibilities shortly.)

On the other hand, though, it’s also very hard to take over the world. In particular, if people in power see their positions being eroded, it’s generally a safe bet that they’ll take action to prevent that. Further, it’s always much easier to understand and reason about a problem when it’s more concrete and tangible; our track record at predicting large-scale future developments is pretty bad. And so even if the high-level arguments laid out above seem difficult to rebut, there may well be some solutions we missed which people will spot when their incentives to do so, and the range of approaches available to them, are laid out more clearly.

How can we move beyond these high-level arguments? In the rest of this section I’ll lay out two types of disaster scenarios, and then four factors which will affect our ability to remain in control if we develop AGIs that are not fully aligned:

Speed of AI development

Transparency of AI systems

Constrained deployment strategies

Human political and economic coordination

Disaster scenarios

There have been a number of attempts to describe the catastrophic outcomes that might arise from misaligned superintelligences, although it has proven difficult to characterise them in detail. Broadly speaking, the most compelling scenarios fall into two categories. Christiano describes AGIs gaining influence within our current economic and political systems by taking or being given control of companies and institutions. Eventually “we reach the point where we could not recover from a correlated automation failure” - after which those AGIs are no longer incentivised to follow human laws. Hanson also lays out a scenario in which virtual minds come to dominate the economy (although he is less worried about misalignment, partly because he focuses on emulated human minds). In both scenarios, biological humans lose influence because they are less competitive at strategically important tasks, but no single AGI is able to seize control of the wor...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 4: AGI safety from first principles: Alignment, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Parts of this section were rewritten in mid-October.

In the previous section, I discussed the plausibility of ML-based agents developing the capability to seek influence for instrumental reasons. This would not be a problem if they do so only in the ways that are aligned with human values. Indeed, many of the benefits we expect from AGIs will require them to wield power to influence the world. And by default, AI researchers will apply their efforts towards making agents do whatever tasks those researchers desire, rather than learning to be disobedient. However, there are reasons to worry that despite such efforts by AI researchers, AIs will develop undesirable final goals which lead to conflict with humans.

To start with, what does “aligned with human values” even mean? Following Gabriel and Christiano, I’ll distinguish between two types of interpretations. Minimalist (aka narrow) approaches focus on avoiding catastrophic outcomes. The best example is Christiano’s concept of intent alignment: “When I say an AI A is aligned with an operator H, I mean: A is trying to do what H wants it to do.” While there will always be some edge cases in figuring out a given human’s intentions, there is at least a rough commonsense interpretation. By contrast, maximalist (aka ambitious) approaches attempt to make AIs adopt or defer to a specific overarching set of values - like a particular moral theory, or a global democratic consensus, or a meta-level procedure for deciding between moral theories.

My opinion is that defining alignment in maximalist terms is unhelpful, because it bundles together technical, ethical and political problems. While it may be the case that we need to make progress on all of these, assumptions about the latter two can significantly reduce clarity about technical issues. So from now on, when I refer to alignment, I’ll only refer to intent alignment. I’ll also define an AI A to be misaligned with a human H if H would want A not to do what A is trying to do (if H were aware of A’s intentions). This implies that AIs could potentially be neither aligned nor misaligned with an operator - for example, if they only do things which the operator doesn’t care about. Whether an AI qualifies as aligned or misaligned obviously depends a lot on who the operator is, but for the purposes of this report I’ll focus on AIs which are clearly misaligned with respect to most humans.

One important feature of these definitions: by using the word “trying”, they focus on the AI’s intentions, not the actual outcomes achieved. I think this makes sense because we should expect AGIs to be very good at understanding the world, and so the key safety problem is setting their intentions correctly. In particular, I want to be clear that when I talk about misaligned AGI, the central example in my mind is not agents that misbehave just because they misunderstand what we want, or interpret our instructions overly literally (which Bostrom calls “perverse instantiation”). It seems likely that AGIs will understand the intentions of our instructions very well by default. This is because they will probably be trained on tasks involving humans, and human data - and understanding human minds is particularly important for acting competently in those tasks and the rest of the world.[1] Rather, my main concern is that AGIs will understand what we want, but just not care, because the motivations they acquired during training weren’t those we intended them to have.

The idea that AIs won’t automatically gain the right motivations by virtue of being more intelligent is an implication of Bostrom’s orthogonality th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 1: AGI safety from first principles: Goals and Agency, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

The fundamental concern motivating the second species argument is that AIs will gain too much power over humans, and then use that power in ways we don’t endorse. Why might they end up with that power? I’ll distinguish three possibilities:

AIs pursue power for the sake of achieving other goals; i.e. power is an instrumental goal for them.

AIs pursue power for its own sake; i.e. power is a final goal for them.

AIs gain power without aiming towards it; e.g. because humans gave it to them.

The first possibility has been the focus of most debate so far, and I’ll spend most of this section discussing it. The second hasn’t been explored in much depth, but in my opinion is still important; I’ll cover it briefly in this section and the next. Following Christiano, I’ll call agents which fall into either of these first two categories influence-seeking. The third possibility is largely outside the scope of this document, which focuses on dangers from the intentional behaviour of advanced AIs, although I’ll briefly touch on it here and in the last section.

The key idea behind the first possibility is Bostrom’s instrumental convergence thesis, which states that there are some instrumental goals whose attainment would increase the chances of an agent’s final goals being realised for a wide range of final goals and a wide range of situations. Examples of such instrumentally convergent goals include self-preservation, resource acquisition, technological development, and self-improvement, which are all useful for executing further large-scale plans. I think these examples provide a good characterisation of the type of power I’m talking about, which will serve in place of a more explicit definition.

However, the link from instrumentally convergent goals to dangerous influence-seeking is only applicable to agents which have final goals large-scale enough to benefit from these instrumental goals, and which identify and pursue those instrumental goals even when it leads to extreme outcomes (a set of traits which I’ll call goal-directed agency). It’s not yet clear that AGIs will be this type of agent, or have this type of goals. It seems very intuitive that they will because we all have experience of pursuing instrumentally convergent goals, for example by earning and saving money, and can imagine how much better we’d be at them if we were more intelligent. Yet since evolution has ingrained in us many useful short-term drives (in particular the drive towards power itself), it’s difficult to determine the extent to which human influence-seeking behaviour is caused by us reasoning about its instrumental usefulness towards larger-scale goals. Our conquest of the world didn’t require any humans to strategise over the timeframe of centuries, but merely for many individuals to expand their personal influence in a relatively limited way - by inventing a slightly better tool, or exploring slightly further afield.

Furthermore, we should take seriously the possibility that superintelligent AGIs might be even less focused than humans are on achieving large-scale goals. We can imagine them possessing final goals which don’t incentivise the pursuit of power, such as deontological goals, or small-scale goals. Or perhaps we’ll build “tool AIs” which obey our instructions very well without possessing goals of their own - in a similar way to how a calculator doesn’t “want” to answer arithmetic questions, but just does the calculations it’s given. In order to figure out which of these options is possible or likely, we need to better understand the nature of goals and goal-directed agency. That’s the focu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is AGI safety from first principles, Part 2: AGI safety from first principles: Superintelligence, published by Richard_Ngo.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

In order to understand superintelligence, we should first characterise what we mean by intelligence. We can start with Legg’s well-known definition, which identifies intelligence as the ability to do well on a broad range of cognitive tasks.[1] The key distinction I’ll draw in this section is between agents that understand how to do well at many tasks because they have been specifically optimised for each task (which I’ll call the task-based approach to AI), versus agents which can understand new tasks with little or no task-specific training, by generalising from previous experience (the generalisation-based approach).

Narrow and general intelligence

The task-based approach is analogous to how humans harnessed electricity: while electricity is a powerful and general technology, we still need to design specific ways to apply it to each task. Similarly, computers are powerful and flexible tools - but even though they can process arbitrarily many different inputs, detailed instructions for how to do that processing needs to be individually written to build each piece of software. Meanwhile our current reinforcement learning algorithms, although powerful, produce agents that are only able to perform well on specific tasks at which they have a lot of experience - Starcraft, DOTA, Go, and so on. In Reframing Superintelligence, Drexler argues that our current task-based approach will scale up to allow superhuman performance on a range of complex tasks (although I’m skeptical of this claim).

An example of the generalisation-based approach can be found in large language models like GPT-2 and GPT-3. GPT-2 was first trained on the task of predicting the next word in a corpus, and then achieved state of the art results on many other language tasks, without any task-specific fine-tuning! This was a clear change from previous approaches to natural language processing, which only scored well when trained to do specific tasks on specific datasets. Its successor, GPT-3, has displayed a range of even more impressive behaviour. I think this provides a good example of how an AI could develop cognitive skills (in this case, an understanding of the syntax and semantics of language) which generalise to a range of novel tasks. The field of meta-learning aims towards a similar goal.

We can also see the potential of the generalisation-based approach by looking at how humans developed. As a species, we were “trained” by evolution to have cognitive skills including rapid learning capabilities; sensory and motor processing; and social skills. As individuals, we were also “trained” during our childhoods to fine-tune those skills; to understand spoken and written language; and to possess detailed knowledge about modern society. However, the key point is that almost all of this evolutionary and childhood learning occurred on different tasks from the economically useful ones we perform as adults. We can perform well on the latter category only by reusing the cognitive skills and knowledge that we gained previously. In our case, we were fortunate that those cognitive skills were not too specific to tasks in the ancestral environment, but were rather very general skills. In particular, the skill of abstraction allows us to extract common structure from different situations, which allows us to understand them much more efficiently than by learning about them one by one. Then our communication skills and theories of mind allow us to share our ideas. This is why humans can make great progress on the scale of years or decades, not just via evolutionary adaptation over many lifetimes.

I should note tha...