Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ELK Computational Complexity: Three Levels of Difficulty, published by Abram Demski on March 30, 2022 on The AI Alignment Forum.

ELK is described as a problem we wish to solve in the "worst case"; IE, for any potential failure case we can describe, we want to have a strong argument about why we won't run into that particular problem. The ELK document lists several cases we want to handle in that way. First in this list is the case where the correct translation is arbitrarily computationally complex.

However, this assumption implies that ELK is intractable. At best, we might hope to define some notion of approximate solution, and prove that there are efficient approximate solutions. So how are we supposed to solve ELK, if we are to assume that it's intractable?

The answer (at least, as I see it) is by arguing that the correct translation cannot be too computationally complex. (Or, failing that, by providing a strong argument that it is possible, at which point we would have to narrow the scope of ELK ambitions.)

A proposed algorithm for solving ELK which (1) has a strong argument for its correctness, and (2) is itself a tractable algorithm, is usually already an argument that correct translations are tractable. For example, a search procedure usually has to evaluate candidate translations by running them; so if we could prove that the search terminates in a reasonable amount of time, we've probably already proven that the solution it finds is tractable (at least in certain cases, EG, on the training data).

But this doesn't really answer the question. Since we don't already have such a solution, we may need to explicitly think about tractability in order to produce one.

So, in this post, I'm going to think about how we might possibly bound the computational complexity of ELK.

Level 3: Probabilistic Knowledge

The third level of difficulty for ELK (IE, the most computationally difficult that I'll discuss here) is the most conceptually simple to understand: we define the "correct translation" of a pattern of activity in the Predictor to be any and all things correlated with such activity.

In other words, we define the "meaning" of Predictor activity as just whatever we can predictably infer from it. In my previous post I formalized this by imagining probability distributions called "third person perspectives" which can be conditioned on the full execution trace of the Predictor, and infer as much as possible about the world from this information.

This seems intractable for two main reasons:

It requires logical omniscience. For example, this criterion seemingly requires that the correct translation knows all the predictable consequences of a plan, not just the immediate consequences. So long as the consequences can be predicted by running the world-model forward, they count as probabilistic implications of the Predictor's knowledge. In particular, the future outcomes of any computations running in the world are included in this. Therefore, computing this form of ELK requires access to a halting oracle in the worst case.

Even if we can side-step the first issue, this form of ELK is at least as hard as finding marginals in arbitrary graphical models, which is #P-complete. As complexity classes go, this is worse than NP-complete.

By trying to detect any correlation, we impose an arbitrarily high requirement of detailed simulation. As a consequence, this Reporter would know far more than the Predictor can realistically be said to know. If the Predictor knows how the dominoes have been set up, the Reporter is required to know how they fall.

From a safety perspective, we're fine with the reporter knowing too much -- we're only concerned with cases where the Predictor knows something which the humans don't know. However, this does seem like a sign that we don't have to solve such...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing What We Owe The Future, published by William MacAskill on March 30, 2022 on The Effective Altruism Forum.

Summary

My new book on longtermism, What We Owe The Future, will come out this summer

You can now pre-order a copy (US link, UK link)

If you want to help with the book launch, I think pre-ordering is probably the highest-impact thing you can do right now

Longer Summary

I’ve written a book!!

It’s called What We Owe The Future. It makes the case for longtermism — the view that positively affecting the long-run future is a key moral priority of our time — and explores what follows from that view. As well as the familiar topics of AI and extinction risk, it also discusses value lock-in, civilisational collapse and recovery, technological stagnation, population ethics, and the expected value of the future. I see it as a sequel to Doing Good Better, and a complement to The Precipice.

I think I’ve probably worked harder on this project than on any other in my life, and I’m excited that the launch date is finally now in sight: Aug 16th in the US and Sep 1st in the UK. I’m looking forward to being able to share it and discuss it with you all!

I’m now focused on trying to make the book launch go well. I’d really like to use the launch as a springboard for wider promotion of longtermism, trying to get journalists and other influencers talking about the idea. In order to achieve that, a huge win would be to hit The New York Times Bestseller list. And in order to achieve that, getting as many pre-orders as possible is crucial.

In particular, there’s a risk that the book is perceived by booksellers (like Amazon and Barnes and Noble) as being “just another philosophy book”, which means they buy very few copies to sell to consumers. This means that the booksellers don’t put any effort into promoting the book, and they can even literally run out of copies (as happened with Superintelligence after the Elon tweet). Preorders are the clearest way for us to demonstrate that WWOTF (or, as I prefer, “WTF?”) is in a different reference class.

For these reasons, I think that pre-ordering WWOTF is probably the highest value thing you can do to help with the book launch right now. The US link to pre-order is here, the UK link is here, and for all other countries you can use your country’s Amazon link.

US version

UK version

About the book

My hope for WWOTF is that it will be like an Animal Liberation for future generations: impacting broadly how society thinks about the interests of future people, and inspiring people to take action to safeguard the long term. If the launch goes well, then significantly more people — including people who are deciding which careers to pursue, philanthropists, and political decision-makers and policy-makers — will be exposed to the core ideas.

The book is aimed to be both readable for a general audience and informative for EA researchers or interested academics. (Though I’m not sure if I’ve succeeded at this!) So there’s a wide breadth of content: everything from stories of historical instances of long-run impact to discussion of impossibility theorems in population ethics. And, following Toby Ord’s lead, there is an ungodly number of endnotes.

The table of contents can give you the gist:

Introduction

Part I. The Long View

Chapter 1: The Case for Longtermism

Chapter 2: You Can Shape the Course of History

Part II. Trajectory Changes

Chapter 3: Moral Change

Chapter 4: Value Lock-In

Part III. Safeguarding Civilization

Chapter 5: Extinction

Chapter 6: Collapse

Chapter 7: Stagnation

Part IV. Assessing the End of the World

Chapter 8: Is It Good to Make Happy People?

Chapter 9: Will the Future Be Good or Bad?

Part V. Taking Action

Chapter 10: What to Do

In the course of writing the book, I’ve also changed my mind on a number of issues. I hope to share my ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA and the United Nations, published by Lisa G on March 30, 2022 on The Effective Altruism Forum.

Call to action for the EA community

Take the role of the UN more into consideration in EA activities incl. discussions, workshops and other type of EA-events as well as impact research

Seek positions in and around the UN and affiliated organizations and improve institutional decision-making at the UN and its partners

Seek a seat at the table at multi-stakeholder discussions and processes to engage with the UN and its member states. Advocate for high impact policies at relevant UN events and processes for instance in the run up to the 2023 Summit of the Future.

EA research into funding of the UN and affiliated organizations as well as how EA recommended charities (can) work with the UN given the clear distinguished mandate, reputation and footprint the UN has in global human rights, peace and security and development issues.

The EA community seems to largely neglect an enormous system that has had significant impact in the past and present in preventing and solving worldwide issues in the areas of human rights, peace and security and development. These are the three pillars of the United Nations, the one and only intergovernmental organization where 193 member states come together to discuss common problems and find shared solutions to benefit all of humanity. It was founded in 1945 after the Second World War with the main purpose at the time to prevent another World War in the future. It has been successful in this manner, as the data on deaths in conflict where state governments are a participant of has tremendously declined since 1946. Over the years it has developed into an organization with many specialized agencies, funds and programmes. They do not all start with the UN in their names like United Nations Children's Fund (UNICEF) or United Nations Population Fund (UNFPA).

The World Bank, the World Food Programme (WFP) and the World Health Organization (WHO) are also all part of the UN system.

As a unique system with organizations that head global responses to global issues, it is remarkable that there is so little attention to this system in the EA community. If you look at where EA started from, looking into very focused, measurable impactful interventions it makes sense that this system was overlooked. It is big, slow and bureaucratic and for sure the UN has not always delivered sustainable and impactful results. In some cases it may have also caused harm in their interventions despite good intentions. However, it is too big and too important to neglect. Moreover, there is only one UN, where all member states come together to negotiate global solutions to global problems. It has a unique mandate and reputation and a scale and reach that no other non-profit organization could come close to.

Bill knows

Bill Gates and Melinda French, who in the EA community have a certain level of respect due to comprehensive approaches and research for interventions in achieving global development goals through the Gates foundation, acknowledge the importance of the UN system. Did you know the Gates Foundation is the second-largest contributor to the WHO? As of September 2021, it had invested nearly $780 million in its programs. Germany, the biggest contributor, had contributed more than $1.2 billion, while the United States donated $730 million. Bill also realized there is an opportunity for his company in engaging with the UN. In 2021 Microsoft opened a UN Affairs Office in New York. The first private company to do so and it has been widely engaged in collaborations with different UN organizations specifically focused on delivering international digital transformation (e.g. providing accessible, equitable, safe internet for all).

The Secretary-General of the United Nations develop...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Debating myself on whether “extra lives lived” are as good as “deaths prevented”, published by HoldenKarnofsky on March 29, 2022 on LessWrong.

If we could do something to lower the probability of the human race going extinct,1 that would be really good. But how good? Is preventing extinction more like “saving 8 billion lives” (the number of people alive today), or “saving 80 billion lives” (the number who will be alive over the next 10 generations) . or "saving 625 quadrillion lives" (an Our World in Data estimate of the number of people who could ever be born) ... or “saving a comically huge number of lives" (Nick Bostrom argues for well over 10^46 as the total number of people, including digital people, who could ever exist)?

More specifically, is “a person getting to live a good life, when they otherwise would have never existed” the kind of thing we should value? Is it as good as “a premature death prevented?”

Among effective altruists, it’s common to answer: “Yes, it is; preventing extinction is somewhere around as good as saving [some crazy number] of lives; so if there’s any way to reduce the odds of extinction by even a tiny amount, that’s where we should focus all the attention and resources we can.”

I feel conflicted about this.

I am sold on the importance of specific, potentially extinction-level risks, such as risks from advanced AI. But: this is mostly because I think the risks are really, really big and really, really neglected. I think they’d be worth focusing on even if we ignored arguments like the above and used much more modest estimates of “how many people might be affected.”

I’m less sold that we should work on these risks if they were very small. And I have very mixed feelings about the idea that “a person getting to live a good life, when they otherwise would have never existed” is as good as “a premature death prevented.”

Reflecting these mixed feelings, I'm going to examine the philosophical case for caring about “extra lives lived” (putting aside the first bullet point above), via a dialogue between two versions of myself: Utilitarian Holden (UH) and Non-Utilitarian Holden (NUH).2

This represents actual dialogues I’ve had with myself (so neither side is a pure straw person), although this particular dialogue serves primarily to illustrate UH's views and how they are defended against initial and/or basic objections from NUH. In future dialogues, NUH will raise more sophisticated objections.

This is part of a set of dialogues on future-proof ethics: trying to make ethical decisions that we can remain proud of in the future, after a great deal of (societal and/or personal) moral progress. (Previous dialogue here, though this one stands on its own.)

A couple more notes before I finally get started:

The genre here is philosophy, and a common type of argument is the thought experiment: "If you had to choose between A and B, what would you choose?" (For example: "is it better to prevent one untimely death, or to allow 10 people to live who would otherwise never have been born?")

It's common to react to questions like this with comments like "I don't really think that kind of choice comes up in real life; actually you can usually get both A and B if you do things right" or "actually A isn't possible; the underlying assumptions about how the world really works are off here." My general advice when considering philosophy is to avoid reactions like this and think about what you would do if you really had to make the choice that is being pointed at, even if you think thI ae author's underlying assumptions about why the choice exists are wrong. Similarly, if you find one part of an argument unconvincing, I suggest pretending you accept it for the rest of the piece anyway, to see whether the rest of the arguments would be compelling under that assumption....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unsurprising things about the EA movement that surprised me, published by Ada-Maaria Hyvärinen on March 30, 2022 on The Effective Altruism Forum.

I learned a lot about EA from my local group before reading that much by myself. So when I finally started to actively read EA material, I was quite confused about EA as a global social movement and who the people behind all those texts were. Here is a list that would have helped me put EA texts in the correct context; maybe it will be helpful for someone else.

There is a social EA movement

EA is not just the way some people happen to think. There is also an active global EA movement that runs the EA forum, organizes events and facilitates collaboration between EA individuals and organizations.

There are a lot of real professional people in EA, and those people are influencing things in the real world – EA is by no means just a philosophy discussion club, even if your local EA club is one (and it does not have to be one forever!)

It is even possible to work in “movement building”, for example helping EAs have more positive impact, communicating about EA to new people and supporting local volunteer groups. There is funding for this kind of work because a lot of people believe that right now, it is even more important to establish a good EA movement than to work directly on EA cause areas.

EA related work might already influence the world around you. (I was really surprised to learn that the Finnish Ministry for Foreign Affairs had funded a research project on x-risk in 2017. It made EA seem a lot like something in the real world and less like just a bunch of well-meaning people on the internet.)

Size of EA

Even if EA has relevant real world influence, the EA movement has way fewer people than I initially thought. There are currently (March 2022) less than 21 000 members in the Effective Altruism FB-group and the EA survey 2020 collected 2 166 answers, so probably the number of engaged EAs is somewhere between those numbers. (To understand what that number means, I remind myself that my favorite role-playing event Ropecon had almost 5 000 attendees in 2019. You should use something you can easily imagine instead.)

There are a lot of “famous EA figures” whose names most engaged EAs would recognize at least on the level of “this is an important person in the movement”. Also, a lot of EAs know each other personally, because they have worked together, run EA groups together or even live together. (Understanding these two things made it suddenly a lot easier to read the EA forum: knowing that users might have other information than just the contents of a certain post or comment helped to make sense of some interactions.)

EA material

When you first get to EA, it feels like there is an EA text about everything. That is not true: actually, the amount of texts commenting on issues from an EA perspective is limited. (Personally, I would love for someone to write about balancing between moral impartiality and wanting to help locally because they live in a middle-to-low-income country. Being from Finland myself, it is relatively easy for me to let our Nordic welfare state take care of local issues, but this is not the case everywhere.)

In particular, there is no secret EA database of estimates of effectiveness of every possible action (sadly). When you tell people effective altruism is about finding effective, research-based ways of doing good, it is a natural reaction to ask: “so, what are some good ways of reducing pollution in the Baltic Sea / getting more girls into competitive programming / helping people affected by [current crisis that is on the news]” or “so, what does EA think of the effectiveness of [my favorite charity]”. Here, the honest answer is often “nobody in EA knows”, and it is easy to sound dismissive by adding “and w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Procedurally evaluating factual accuracy: a request for research, published by Jacob Hilton on March 30, 2022 on The AI Alignment Forum.

I am grateful to Daniel Kokotajlo, Beth Barnes and John Schulman for feedback on this post.

The purpose of this post is to request research on the design of precise procedures for evaluating how factually accurate pieces of text are. This stands out to me as an area that is potentially valuable for reducing risks from advanced AI, while not requiring detailed knowledge of ML.

Problem statement

Suppose that you are given:

A real-world setting in which text appears, such as "Google search snippets", or "OpenAI API outputs", or "arXiv abstracts"

A collection of contexts in that setting, such as a "the search queries made in 2021", or "the questions from the ELI5 dataset", or "all existing papers on the arXiv"

A text distribution for each context, such as "actual current Google search snippets", or "responses produced by an ML model", or "the abstract a certain person would write after reading the paper"

The problem is to define a procedure that takes in a context and a piece of text from the given distribution, and outputs a numeric score for factual accuracy.

The procedure should have the following properties:

It is feasible for humans to carry out.

It is as unambiguous as possible: different people following the same procedure should end up with the same score, except perhaps for a minimal number of "irreducible" judgment calls.

It can be optimized to produce good outcomes: to the extent that factual accuracy is important in the given setting, selecting text with a high score is beneficial, compared to alternative procedures.

Included in this is the effect that knowledge of the procedure could have, e.g. people will have more trust in the text if they understand and endorse the procedure.

In practice, whether good outcomes are produced will depend on the method and extent of optimization (e.g., Goodhart's law will eventually kick in), and on criteria other than factual accuracy. This makes the problem statement a little fuzzy.

Note that the problem statement doesn't mention ML models (except as examples). The problem is of course motivated by ML models, but I think it can be studied relatively independently of ML, at least initially.

Motivation

The main motivation for this request is that it is a problem that arises very naturally when attempting to train truthful LMs. The most straightforward way to optimize the factual accuracy of a language model is to have humans evaluate the factual accuracy of model outputs, and to then optimize those evaluations using techniques like reinforcement learning. The procedure needs to be unambiguous because label noise hurts both ML training and labeler monitoring (not to mention other benefits). Subject to this constraint, the main criterion for the procedure should be that it produces good outcomes in the given real-world setting.

My main reasons for thinking that this research could be important for reducing risks from advanced AI are:

Direct benefits. The research could result in procedures that are directly used to train more beneficial AI. The sorts of risks this could help mitigate are discussed in more depth in Truthful AI and Risks from AI persuasion.

Preparation for harder versions of the problem. I expect advanced AI systems to pose analogous versions of the same problem, but to require more advanced solutions, in which AI systems assist with human evaluation, for example. Solutions to the simpler problem can serve both as a starting point for more advanced solutions and as a baseline against which they can be compared, and can help build relevant capacity.

Supporting ML work on truthful LMs. Currently, the responsibility for this research falls largely on ML researchers trying to improv...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some observations about LM monologue limitations, published by leogao on March 30, 2022 on The AI Alignment Forum.

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and Laria for discussion.

One potential way we might think to interpret LMs is to have them explain their thinking as a monologue or justification or train of thought or something. Particularly, by putting the explanation before the answer, we might hope to encourage the model to actually use the monologue to come to its conclusion and try to avoid the model coming up with the bottom line first. However, there are a bunch of ways this could go wrong. For instance:

Humans are really good at coming up with the bottom line first and then rationalizing a reasonable sounding process to arrive at that conclusion, so imitating humans encourages rationalization by default, even if the explanation is put temporally before the bottom line. Even though the model has uncertainty over the explanation, and in some sense the explanation determines the answer, the problem is that the distribution over the explanations may not be determined by the validity of the explanation, but rather be an integral over possible bottom lines.

It's not sufficient for the answer to merely depend on the explanation: the model could be storing arbitrary data in the explanation in a way that doesn't line up with the meaning of the words. It could also be using the additional tokens to do more thinking in the latents but not in the tokens, etc.

If the model is deceptive and we use the monologues to try and tell if the model is deceptive and apply any optimization pressure against that whatsoever (this includes rejecting actions/models based on it), then it would actually be incentivized to not tell us about its scheming, because we would be punishing (blatant monologue, evil action) more than (benign monologue, evil action).

This also has a reduction to ELK: if the model's abstractions are sufficiently different from human abstractions, it may actually be more work for the model to translate its actual thought process to human understandable text than to just simulate the rationalization process a human would go through. To make the connection to ELK more clear, we can think of the reporter as the circuit that converts the model's actual thought process to text (or simulates the human thought process).

I think it's possible that we're simply lucky and that GPTs of the relevant size just don't have problems (in the same way that ELK could just be easy), but I'm not sure we want to bet on that. I also think lots of these problems correspond to problems with mechanistic interpretability, especially the ELK and deception related ones.

I don't think this is a particularly novel observation but it's nice having a reference to point to anyways.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the EU Tech Policy Fellowship, published by Jan-WillemvanPutten on March 30, 2022 on The Effective Altruism Forum.

Announcing the EU Tech Policy Fellowship, a cost-covered programme to catapult relevant graduates into high-impact career paths in European policy and Tech Policy roles.

Summary

Training for Good is excited to announce the EU Technology Policy Fellowship. This programme enables promising EU citizens to launch careers focused on regulating high-priority emerging technologies, especially AI and cybersecurity.

This fellowship consists of three components:

Remote study group (July - August, 4 hours a week): A 6 week part-time study group covering AI governance & technology policy fundamentals.

Policy training in Brussels (late August / early September - exact date TBD): An intensive week-long bootcamp in Brussels featuring workshops, guest lectures from relevant experts and networking events.

Tracks: Fellows will then participate in one of two tracks depending on their interests.

Track 1 (September - December full time): Fellows will be matched with a host organisation working on European tech regulation for a ~3 month placement between September and December 2022.

Host organisations include The Future Society, Centre for European Policy Studies, and NLAI Coalition (among others).

Track 2 (September): Fellows will receive job application support and guidance to pursue a career in the European Commission, party politics or related policy jobs in Europe.

This will include career workshops, feedback on CVs, interview training and mentorship from experienced policy professionals.

Other important points:

If you have any questions or would like to learn more about the program and whether or not it’s the right fit for you, Training for Good will be hosting an informal information session on Thursday April 7 (5.30pm CET), please subscribe here for that session.

This fellowship is only open to EU citizens.

Modest stipends are available to cover living and relocation costs. We expect most stipends to be between €1,250 and €1,750 per month (though will take individual circumstances into consideration).

For track 1, stipends are available for up to 4 months while participating in placements.

For track 2, stipends are available for up to 1 month while exploring and applying for policy roles.

Apply here by April 19th.

The Programme

The programme spans ~5 months from July to December, is fully cost-covered, and where needed, participants can avail of stipends to cover living costs. It consists of 3 major parts:

Remote study group (July - August): A 6 week study group covering AI governance & technology policy fundamentals (draft curriculum). This study group will run from July 18 - August 28, including ~4 hours of readings and a 1 hour discussion each week.

Policy training in Brussels (exact date TBD): An intensive week-long bootcamp in Brussels featuring workshops, guest lectures from relevant experts and networking events.

This training will either run the week commencing August 29 or September 5 (exact date to be decided).

All accomodation, food & travel costs will be fully covered by Training for Good.

Tracks: Fellows will then participate in one of two tracks depending on their interests.

Track 1 (September - December): Fellows will be matched with a host organisation working on European tech regulation for a ~3 month placement between September and December 2022.

Host organisations include The Future Society, Centre for European Policy Studies, and NL AI Coalition (among others).

Modest stipends are available to fellows during these placements to cover living and relocation costs for up to 4 months.

Track 2 (September): Fellows will receive job application support and guidance to pursue a career in the European Commission, party politics or related policy ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Intro to brain-like-AGI safety] 10. The alignment problem, published by Steve Byrnes on March 30, 2022 on The AI Alignment Forum.

Part of the “Intro to brain-like-AGI safety” post series.

10.1 Post summary / Table of contents

In this post, I discuss the alignment problem for brain-like AGIs—i.e., the problem of making an AGI that’s trying to do some particular thing that the AGI designers had intended for it to be trying to do.

The alignment problem is (I claim) the lion’s share of the AGI safety problem. I won’t defend that claim here—I’ll push it off to the next post (Post #11), which will cover exactly how AGI safety is related to AGI alignment, including the edge-cases where they come apart.

This post is about the alignment problem, not its solution. What are the barriers to solving the alignment problem? Why do straightforward, naïve approaches seem to be insufficient? I’ll talk about possible solution approaches in later posts. (Spoiler: Nobody knows how to solve the alignment problem, and neither do I.)

Table of contents

In Section 10.2, I’ll define “inner alignment” and “outer alignment” in the context of our brain-like-AGI motivation system. To oversimplify a bit:

If you prefer neuroscience terminology: “Outer alignment” entails having “innate drives” (cf. Post #3, Section 3.4.2) that fire in a way that reflects how well the AGI is following the designer’s intentions. “Inner alignment” is a situation where an imagined plan (built out of concepts, a.k.a. latent variables, in the AGI’s world-model) has a valence that reflects the innate drives which would be triggered by that plan.

If you prefer reinforcement learning terminology: “Outer alignment” entails having a ground-truth reward function that spits out rewards that agree with what we want. “Inner alignment” is having a value function that estimates the value of a plan in a way that agrees with its eventual reward.

In Section 10.3, I’ll talk about two key issues that make alignment hard in general (whether “inner” or “outer”):

The first is “Goodhart’s Law”, which implies that an AGI whose motivation is just a bit off from what we intended can nevertheless lead to outcomes that are wildly different from what we intended.

The second is “Instrumental Convergence”, which says that a wide variety of possible AGI motivations—including generic, seemingly-benign motivations like “I want to invent a better solar cell”—will lead to AGIs which try to do catastrophically-bad things like escape human control, self-reproduce, gain resources and influence, act deceptively, and kill everyone (cf. Post #1, Section 1.6).

In Section 10.4, I’ll discuss two challenges to achieving “outer alignment”: first, translating our design intentions into machine code, and second, the possible inclusion of rewards for behaviors that are not exactly what we ultimately want the AGI to do, such as satisfying its own curiosity (see Post #3, Section 3.4.3).

In Section 10.5, I’ll discuss numerous challenges to achieving “inner alignment”, including reward ambiguity, bad credit assignment, “ontological crises”, and the AGI manipulating itself or its training process.

In Section 10.6, I’ll discuss some reasons that “outer alignment” and “inner alignment” should probably not be viewed as two independent problems with two independent solutions. For example, neural network interpretability would cut through both layers.

10.2 Inner & Outer (mis)alignment

10.2.1 Definition

Here, yet again, is that figure from Post #6, now with some helpful terminology (blue) and a little green face at the bottom left:

I want to call out three things from this diagram:

The designer’s intentions (green face): Perhaps there’s a human who is programming the AGI; presumably they have some idea in their head as to what the AGI is supposed to be trying to do. That’s just ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mental nonsense: my anti-insomnia trick, published by AllAmericanBreakfast on March 28, 2022 on LessWrong.

My dad used to suffer from insomnia, holding imaginary meetings in his head late into the night. I'm the same way.

One stress-relief technique from Johns Hopkins is a breathing + mantra exercise. I find their mantra recommendation cringey:

Breathing in I am calm, breathing out I am coping.

Mantras, guided imagery, and autogenic training are generally hard for me, because I feel pressure to "stay focused on the mantra."

My weird trick is to replace the mantra with free-form inner-monologue nonsense syllables. Here's an example:

Nimbla doobla deeble dee... simba dimba lima nooble doo...

I let the pacing and "mental voice" vary, but mainly aim for a sleepy-sounding tone.

At the same time, I let myself visualize nonsense images. My visual imagination isn't very vivid, and I don't try to create any specific images. It's nothing more than a patchy and low-res mental screensaver. Not even this exciting:

This puts me right to sleep.

If you have refinements of anti-insomnia tricks that aren't on bog-standard lists like this one, go ahead and add them in the comments!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.