Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI ruin scenarios are likely (and disjunctive), published by So8res on July 27, 2022 on The Effective Altruism Forum.

Note: As usual, Rob Bensinger helped me with editing. I recently discussed this model with Alex Lintz, who might soon post his own take on it.

Some people seem to be under the impression that I believe AGI ruin is a small and narrow target to hit. This is not so. My belief is that most of the outcome space is full of AGI ruin, and that avoiding it is what requires navigating a treacherous and narrow course.

So, to be clear, here is a very rough model of why I think AGI ruin is likely. (>90% likely in our lifetimes.)

My real models are more subtle, take into account more factors, and are less articulate. But people keep coming to me saying "it sounds to me like you think humanity will somehow manage to walk a tightrope, traverse an obstacle course, and thread a needle in order to somehow hit the narrow target of catastrophe, and I don't understand how you're so confident about this". (Even after reading Eliezer's AGI Ruin post—which I predominantly agree with, and which has a very disjunctive character.)

Hopefully this sort of toy model will at least give you some vague flavor of where I’m coming from.

Simplified Nate-model

The short version of my model is this: from the current position on the game board, a lot of things need to go right, if we are to survive this.

In somewhat more detail, the following things need to go right:

The world’s overall state needs to be such that AI can be deployed to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

The world needs to admit of an AGI deployment strategy (compatible with realistic alignable-capabilities levels for early systems) that prevents the world from being destroyed if executed.

At least one such strategy needs to be known and accepted by a leading organization.

Somehow, at least one leading organization needs to have enough time to nail down AGI, nail down alignable AGI, actually build+align their system, and deploy their system to help.

This very likely means that there needs to either be only one organization capable of building AGI for several years, or all the AGI-capable organizations need to be very cautious and friendly and deliberately avoid exerting too much pressure upon each other.

It needs to be the case that no local or global governing powers flail around (either prior to AGI, or during AGI development) in ways that prevent a (private or public) group from saving the world with AGI.

Technical alignment needs to be solved to the point where good people could deploy AI to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

There need to be people who think of themselves as working on technical alignment, whose work is integrated with AGI development and is a central input into how AGI is developed and deployed.

They need to be able to perceive every single lethal problem far enough in advance that they have time to solve them.

They need to be working on the problems in a way that is productive.

The problems (and the general paradigm in which they're attacked) need to be such that people's work can stack, or such that they don't require much serial effort; or the research teams need a lot of time.

Significant amounts of this work have to be done without an actual AGI to study and learn from; or the world needs to be able to avoid deploying misaligned AGI long enough for the research to complete.

The internal dynamics at the relevant organizations need to be such that the organizations deploy an AGI to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

The teams that first gain access to AGI, need to care in the right ways about AGI alignment.

E....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levels of Pluralism, published by Adam Shimi on July 27, 2022 on The AI Alignment Forum.

This work was done while at Conjecture.

Preparing the Pluralism Question

When do we want all our research eggs in the same paradigm basket?

Although most people don't go as far as the extreme paradigmatism of Thomas Kuhn in The Structure of Scientific Revolutions, which only allows one paradigm at time (for a given science), the preference for less rather than more options is still pervasive. In the ideal, a convergence to one, even if it's not always feasible. After all, there's only one correct answer, right?

Putting that debatable question aside, I've become more and more convinced that pluralism, the pursuit of multiple lines of research in parallel, is far more prevalent and integral to the process of science than Kuhn's naive paradigmatism. This realization has emerged from studying History and Philosophy of Science, especially outside of physics which for many reasons is quite an unrepresentative science.

But one crucial preliminary point is that pluralism can appear at multiple levels. And the value of pluralism also depends on the level at which it is applied.

So this post proposes a decomposition of the activity of research into four levels, and introduces the corresponding pluralism at each level. Although the point here is not (yet) to argue for pluralism, I offer some examples of pluralistic successes, as well as arguments for the epistemic circumstances where pluralism seems the most valuable. I also finish by proposing a geometric model for when each level of pluralism makes sense, based around considering bits of evidence as objects in high-dimensional space.

The four levels of pluralism I discuss are:

Individual pluralism: pluralism of the methods, ideas, and analogies used by a single researcher or a single research tradition.

Approach pluralism: pluralism of approaches to the same operationalization of the problem.

Operationalization pluralism: pluralism in the way that the problem itself is operationalized.

Problem pluralism: pluralism in the problem itself.

Thanks to Andrea Motti for feedback on a draft of this post.

Simplifying Assumption: Focus on Epistemic Circumstances

When investigating under which circumstances some epistemic strategy applies, there are many confusing and complicating factors coming from psychology and sociology. Taking pluralism as an example, the following non-exhaustive list comes to mind:

How easy/difficult is it for researchers to keep multiple approaches at different levels?

How confusing is it for researchers to keep multiple approaches at different levels?

Do we have enough resources for implementing the ideal level of pluralism?

How should we implement it, given the social structures and the psychological difficulties?

My stance here, and more generally, is to neglect these issues so I can focus on the ideal epistemic algorithm under the circumstances studied. The rationale is that the sociological and psychological factors can be better dealt with once we know the ideal strategy, and removing them gives us an idealization that is easier to work with. In some this emulates how most physics approximations remove the details (often ultimately important details) to get to the core insight.

Although I expect this to work, note that this is in tension with epistemological vigilance: there is some chance that the sociological and psychological factors matter so much that it makes more sense to include part of them in the ideal answer.

Levels: from Individuals to Problems

Individual Pluralism

If we zoom in on a particular research approach or tradition, we might expect to be too low-level for pluralism to appear. Yet pluralism isn't only about portfolios of approaches — a single tradition can be pluralist in its methods, ideas, and an...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI ruin scenarios are likely (and disjunctive), published by Nate Soares on July 27, 2022 on The AI Alignment Forum.

Note: As usual, Rob Bensinger helped me with editing. I recently discussed this model with Alex Lintz, who might soon post his own take on it.

Some people seem to be under the impression that I believe AGI ruin is a small and narrow target to hit. This is not so. My belief is that most of the outcome space is full of AGI ruin, and that avoiding it is what requires navigating a treacherous and narrow course.

So, to be clear, here is a very rough model of why I think AGI ruin is likely. (>90% likely in our lifetimes.)

My real models are more subtle, take into account more factors, and are less articulate. But people keep coming to me saying "it sounds to me like you think humanity will somehow manage to walk a tightrope, traverse an obstacle course, and thread a needle in order to somehow hit the narrow target of catastrophe, and I don't understand how you're so confident about this". (Even after reading Eliezer's AGI Ruin post—which I predominantly agree with, and which has a very disjunctive character.)

Hopefully this sort of toy model will at least give you some vague flavor of where I’m coming from.

Simplified Nate-model

The short version of my model is this: from the current position on the game board, a lot of things need to go right, if we are to survive this.

In somewhat more detail, the following things need to go right:

The world’s overall state needs to be such that AI can be deployed to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

The world needs to admit of an AGI deployment strategy (compatible with realistic alignable-capabilities levels for early systems) that prevents the world from being destroyed if executed.

At least one such strategy needs to be known and accepted by a leading organization.

Somehow, at least one leading organization needs to have enough time to nail down AGI, nail down alignable AGI, actually build+align their system, and deploy their system to help.

This very likely means that there needs to either be only one organization capable of building AGI for several years, or all the AGI-capable organizations need to be very cautious and friendly and deliberately avoid exerting too much pressure upon each other.

It needs to be the case that no local or global governing powers flail around (either prior to AGI, or during AGI development) in ways that prevent a (private or public) group from saving the world with AGI.

Technical alignment needs to be solved to the point where good people could deploy AI to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

There need to be people who think of themselves as working on technical alignment, whose work is integrated with AGI development and is a central input into how AGI is developed and deployed.

They need to be able to perceive every single lethal problem far enough in advance that they have time to solve them.

They need to be working on the problems in a way that is productive.

The problems (and the general paradigm in which they're attacked) need to be such that people's work can stack, or such that they don't require much serial effort; or the research teams need a lot of time.

Significant amounts of this work have to be done without an actual AGI to study and learn from; or the world needs to be able to avoid deploying misaligned AGI long enough for the research to complete.

The internal dynamics at the relevant organizations need to be such that the organizations deploy an AGI to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

The teams that first gain access to AGI, need to care in the right ways about AGI alignment.

E.g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distillation Contest - Results and Recap, published by Aris Richardson on July 26, 2022 on The Effective Altruism Forum.

This post:

Announces the winning submissions to the Distillation Contest

Gives further insight into the scoring process for the contest

Examines the effectiveness of our advertising strategies

Gives a brief impact estimate of the contest

Shares my advice for community builders who are planning to run contests

Notes:

A huge thank you to Akash and all of the judges for this contest! This wouldn’t have been possible without their work. I’m definitely not perfect! I imagine there are better ways to advertise, run, and score a contest, so I wanted to be transparent about my process so that other people could make suggestions if they have ideas for improvements.

Want to use the materials from the Distillation Contest to run your own version? I’m in the process of creating a platform for EA contests and will soon have a demo site up! I’m planning to upload all of the Distillation contest materials into a “bundle” so that anyone can host their own contest easily. Once the site is up (hopefully in the next couple of weeks), feel free to use anything there and iterate to make better resources. If you have your own EA contest resources and would like to make them available to other people, I’d love to add them to the notion!

Winners

The submission winning first place in the Distillation Contest is Understanding Selection Theorems by UC Berkeley’s Adam Khoja – distilling John Wentworth's Selection Theorems: A Program for Understanding Agents. Our second-place winner is The Geometry of Adversarial Perturbations by Gabriel Wu from Harvard University – a distillation of Universal Adversarial Perturbations.

We’ve granted 15 other prizes, six $500 awards, and nine $250 awards, plus three honorable mentions. The six $500 winners are listed below, with their distillations linked to their names. You can find the other finalists, and their distillations, on the EA Berkeley distillation contest winners page.

Callum McDougal, Cambridge

Jasper Day, University of Edinburgh (has not yet given permissions to share submission)

Harrison Gietz, Louisiana State University

Sasha Sato, UC Berkeley

Chinmay Deshpande, Harvard University

Yash Dave, UC Berkeley

Scoring

Each submission was scored by two judges (of which we had five total, all of whom actively work in the alignment space). Our rubric took into account the submission’s Depth of Understanding, Clarity of Presentation, Concision/Length, Originality of Insight, Accessibility, and two extra-subjective measures: X-Factor and Subjective Rating, explained below.

X-Factor

Some submissions may end up scoring low despite being amazing because they are exceptional for a quality that’s missed by the factors listed above (they make really great applications of the material, they synthesize multiple sources, they have unexpectedly unique and useful dimensions, etc.). You can grant as many additional points to a distillation as you’d like when you score with this X-Factor zone. Most papers will not have an X-factor effect, so do not feel required to give out X-Factor points.

Subjective Rating

Ignore the rubric. Assume you just had to rate the submission on a scale from 1-10 (including decimals). What rating would you give this submission?

Once the judges had scored their submissions, we found the average score for that judge and then divided each submission score by the average to get an adjusted score. If the average score was 40, a score of 50 would turn into a 1.25.

There were 19 submissions that were rated above average by both of their judges. Given that this was already above the number of submissions we said we’d award, we moved on to mostly comparing these submissions to one another. We also looked at “controversial” submission...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: «Boundaries», Part 1: a key missing concept from utility theory, published by Andrew Critch on July 26, 2022 on The AI Alignment Forum.

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.

This is Part 1 of my «Boundaries» Sequence on LessWrong.

Summary: «Boundaries» are a missing concept from the axioms of game theory and bargaining theory, which might help pin down certain features of multi-agent rationality (this post), and have broader implications for effective altruism discourse and x-risk (future posts).

1. Boundaries (of living systems)

Epistemic status: me describing what I mean.

With the exception of some relatively recent and isolated pockets of research on embedded agency (e.g., Orseau & Ring, 2012; Garrabrant & Demsky, 2018), most attempts at formal descriptions of living rational agents — especially utility-theoretic descriptions — are missing the idea that living systems require and maintain boundaries.

When I say boundary, I don't just mean an arbitrary constraint or social norm. I mean something that could also be called a membrane in a generalized sense, i.e., a layer of stuff-of-some-kind that physically or cognitively separates a living system from its environment, that 'carves reality at the joints' in a way that isn't an entirely subjective judgement of the living system itself. Here are some examples that I hope will convey my meaning:

a cell membrane (separates the inside of a cell from the outside);

a person's skin (separates the inside of their body from the outside);

a fence around a family's yard (separates the family's place of living-together from neighbors and others);

a digital firewall around a local area network (separates the LAN and its users from the rest of the internet);

a sustained disassociation of social groups (separates the two groups from each other)

a national border (separates a state from neighboring states or international waters).

Figure 1: Cell membranes, skin, fences, firewalls, group divisions, and state borders as living system boundaries.

Comparison to Cartesian Boundaries.

For those who'd like a comparison to 'Cartesian boundaries', as in Scott Garrabrant's Cartesian Frames work, I think what I mean here is almost exactly the same concept. The main differences are these:

(life-focus) I want to focus on boundaries of things that might naturally be called "living systems" but that might not broadly be considered "agents", such as a human being that isn't behaving very agentically, or a country whose government is in a state of internal disagreement. (I thought of entitling this sequence "membranes" instead, but stuck with 'boundaries' because of the social norm connotation.)

(flexibility-focus) Also, the theory of Cartesian Frames assumes a fixed cartesian boundary for the agent, rather than modeling the boundary as potentially flexible, pliable, or permeable over time (although it could be extended to model that).

Comparison to social norms.

Certain social norms exist to maintain separations between livings systems. For instance:

Personal space boundaries. Consider a person Alex who wants to give his boss a hug, in a culture with a norm against touching others without their consent. In that case, the boss's personal space creates a kind of boundary separating the boss from Alex, and there's a protocol — asking permission — that Alex is expected to follow before crossing the boundary.

Information boundaries for groups. Consider a person Betty who's having a very satisfying romantic relationship, in a culture where there's a norm of not discussing romantic relationships with colleagues at work. In that case, Alice maintains an information barrier between the details of her romantic life and her workplace. The workplace is kind of living...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humanity’s vast future and its implications for cause prioritization, published by evelynciara on July 26, 2022 on The Effective Altruism Forum.

Summary

Humanity could last millions of years and spread beyond Earth, resulting in an unimaginably large number of people being born. Even under conservative assumptions, at least 100 trillion people could exist in the future.

The best way to help future people have good lives is to create positive trajectory changes—durable improvements to the world at every point in the future. Reducing existential and suffering risks would improve humanity’s entire future trajectory, whereas speeding up economic growth would only create benefits over the next few thousand years (which is still a very big deal for the next few generations).

Based on this information, I’ve pivoted toward reducing suffering risks (S-risks), or risks of astronomical suffering in the future, because it is an important and neglected strategy for creating positive long-term trajectory changes.

How many people could there be?

There are almost 8 billion people in the world today. The United Nations estimates that by 2100, the global population will stabilize around 11 billion people with 125 million births per year. If we have an idea of how long humanity will survive, then we can estimate how many people will be born in the future.

Modern humans have been around for 200,000 years, and the average mammalian species lasts for a million years. So let’s suppose that humanity will survive for another 800,000 years. If 125 million people are born each year, then 100 trillion people will eventually be born. That’s 12,500 times as many people as are alive today. If we last 10 million years, like some mammalian species, then we could eventually give rise to over a quadrillion people.

Of course, human civilization could survive for much longer than that—or much shorter. Even in extreme scenarios, climate change is unlikely to destroy civilization or render humans extinct, but it could indirectly lead to our extinction by driving political instability and global conflict. By some accounts, artificial general intelligence could be developed this century, and if we don’t have the technology to align AGI systems with our values, one could go rogue and kill us all simply because it is indifferent to our survival.

On the other hand, we could use our technological capabilities to live for billions of years in the Solar System and beyond. This century, we will probably start establishing human settlements on other planets, like Mars, and satellites like Saturn’s largest moon Titan. Some scientists have speculated that we will be able to start traveling through deep space by the end of the 24th century. Earth will remain habitable for at least another 500 million years, but if we have a presence elsewhere in the Solar System, we can survive for much longer than that. When the Sun eventually becomes a white dwarf in 8 billion years’ time, we could still live in artificial space habitats orbiting what’s left of it, but most of us will likely be living on exoplanets and space habitats orbiting other stars.

Trajectory changes vs. speeding up growth

This vast potential has profound implications for how best to help as many people as possible in the present and future. Many longtermists believe that creating trajectory changes—durable changes to the amount of good in the world at every point in the future—is more valuable for the trillions of people yet to be born than trying to speed up the future. Preventing an existential catastrophe, such as human extinction or the collapse of civilization, is a prototypical trajectory change, since the entire value of the future hangs in the balance.

In Stubborn Attachments, economist Tyler Cowen argues that humanity must focus on three things to m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA's Culture and Thinking are Severely Limiting its Impact, published by Peter Elam on July 26, 2022 on The Effective Altruism Forum.

Executive Summary

EA has a distinct culture, patterns of thinking, and commitments to certain conclusions that stem from its origins.

The culture emphasizes analysis over practice, and it does not attract many of the leaders and builders that are critical for maximizing impact.

Ideas around EA capacity building have lacked an imagination of what’s possible, and tend to be designed around reinforcing and propagating the existing subculture and current ways of doing things.

If your goal truly IS to maximize impact you do not want to only focus on specific projects, programs, or individuals, you want to focus on building ecosystems. And you want to use these ecosystems to attract and support people that share your values and push your mission forward, even if they don’t fit in with the current EA subculture.

If you want to have maximum impact you often want to focus on leadership and governance. Most solvable problems in the world are really leadership and governance problems at their core.

If you want that impact to be lasting, you should focus on building organizations, institutions, or ecosystems that endure over time.

I give an example of what ecosystem building would look like in the context of Haiti, show how this approach is different from current EA approaches, and give real-world examples of different aspects of an ecosystem building approach.

There are so many wonderful people with so much potential to drive positive change that feel unwelcome in the movement as it currently stands. Let them in. And there are so many people that could become effective altruistic leaders if they just received the right encouragement, inspiration, and support. Build ecosystems welcome them, and that inspire, encourage, and support them at scale. This is the type of movement the world is waiting for.

Introduction

I first want to establish that I am an Effective Altruism outsider. I have been vaguely aware of the movement for some time, mainly from guest appearances by EA leaders on podcasts that I listen to. I learned about the FTX Future Fund from Tyler Cowen’s Marginal Revolution blog, and through that I have submitted three proposals to EA-aligned grant opportunities (none were accepted).

However, I was very impressed with the value system expressed in the grant proposal solicitations, and since then I’ve done a deeper dive to try to understand what EA is all about. I have read quite a few forum posts, and spoken to probably a dozen EA affiliated people (all of whom have been kind, intelligent, earnest, and impressive), but that’s more or less where my involvement ends.

So this criticism is based on my observations as a curious outsider, and my lack of insider experience will undoubtedly be obvious in parts of what I write. However, even though I may miss the mark on certain details, I do think that my core point is worth considering, so I would ask for a charitable reading despite these mistakes.

My core criticism is this: Due to its origins and the specific path the movement has taken thus far, there exists a subculture and patterns of thinking that are severely limiting EA’s potential future impact. EA now has the resources to legitimately change things for the better, in a way that few institutions in the world do, and in a way it has not had in the past. However, in order to realize that potential it will require a shift in thinking and strategy by the EA movement that may or may not be possible.

Please understand the core of what I am claiming. EA has an embarrassment of riches. Not just in money, but also in a core of extremely motivated, good-hearted, talented people. The opportunity in front of the movement is truly incredible, but that...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The most important climate change uncertainty, published by cwa on July 26, 2022 on The Effective Altruism Forum.

Short summary: we don't know how much warming will make civilization collapse, and it might not be unreasonable to think of climate change as a major x-risk on these grounds.

Background/Epistemic status: I’m a climate scientist who has been in and around the Effective Altruism community for quite a few years and has spent some time thinking about worst-case outcomes. The following content was partially inspired by this forum comment by SammyDMartin; it includes many of the same points, but also takes things further and potentially in some different directions. I claim no certainty (or even high confidence) in anything said here — the main goal is to encourage discussion.

Introduction

Outside of the EA world, discussion of climate change as a top-priority emergency and/or existential threat is ubiquitous: from politicians to scientists to popular movements and books. EA literature typically takes a different view: climate change is bad but not an existential threat, and there are many other threats that could cause more damage (like unaligned AI or engineered pandemics) and deserve our marginal resources more.

The disconnect between these two views on climate change has bothered me for a long time. Not because I necessarily agree strongly with one side or the other, but because I just haven’t had a good model for why this disconnect exists. I don’t think the answer is quite as simple as “the people who think climate change is a top priority haven’t thought seriously about existential risks and cause prioritization”; anecdotally, I’ve met many people within EA who have, and who still have a nagging feeling that EA is somehow downplaying the risks from climate change. So what is going on?

This post examines two distinct, related, and potentially provocative claims that I think cut to the heart of the disconnect:

most of the subjective existential risk from climate change comes from the uncertainty in how much damage is caused by a given amount of warming

it's not obviously unreasonable to think the existential risk from climate change is as large as that from AI or pandemics.

The focus is on the importance of climate change as a cause area, not neglectedness or tractability; I’ll revisit this omission a bit at the end. The discussion is also purposely aimed at a longtermist/existential-risk-prioritizing audience; the importance of climate change under other moral views can and should be discussed, but I will not do so here.

Background

Following, e.g., Halstead, it is instructive to split the question of climate change damages into three numbered questions:

How much greenhouse gas will humanity emit?

How much warming will there be per unit CO2-equivalent greenhouse gas?

How much damage will be caused (to humanity) by a given amount of warming?

Question 1 can in principle be addressed by economic and political projections, thinking about the economics of decarbonization, and so on. Question 2 is addressed by climate science: climate modeling, looking at past climate changes, and so on. Question 3 has typically been addressed by economic modeling (more on this later) and by considering hard limits for habitability by humans.

Much existing EA analysis of climate change as an existential risk focuses on Question 2. This makes sense in part because it’s the most well-quantified: scientists and the IPCC publish probability distributions of this value, called the “climate sensitivity”. Conditional on a given carbon emission scenario, we can then find out the probability of a certain amount of warming by a certain date (e.g. 2100). If you think that some extreme level of warming will cause existential catastrophe (say 6/10/14 °C, whichever you fancy), then the existe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Neartermists should consider AGI timelines in their spending decisions, published by Tristan Cook on July 26, 2022 on The Effective Altruism Forum.

Summary

Neartermist funders should factor AGI timelines in their cost effectiveness analyses and possibly prepare to spend all their money quickly in light of an AGI arrival ‘heads-up’. When considering two types of interventions: those which produce value in a short period of time and those that produce value over a long time, factoring in AGI timelines pushes to marginally increase spending on the former.

Main text

Suppose for simplicity there are two types of interventions that neartermists can spend money on:

'Fast' interventions are interventions that have an immediate effect but have little to no flow-through effects beyond the time for which they’re applied. Examples could include

providing pain relief medicines

paying people to eat fewer animal products (who revert their diet after the intervention)

'Slow' interventions are interventions that help in the first year of implementation but they continue producing value for many years afterwards. Examples could include

improving someone’s education, which improves their future earnings

improving animal welfare laws

Most of the most cost-effective neartermist interventions are slow interventions.

Suppose we only considered the benefit from the first year of the interventions. Then it seems plausible that some fast interventions would be more cost effective than the cost effective slow interventions. If this was not the case, it may be a case of surprising and suspicious convergence.

Now suppose we knew AGI was one year away and that the slow interventions have no flow-through effects after AGI arrives. Then given the choice between spending on slow or fast interventions, we should spend our money entirely on the latter. At some point, we may be in this epistemic state.

However, we may never be in such an epistemic state where we know AGI is one year away. In this case, we should marginally spend more on fast interventions if we are 50% sure AGI will arrive next year than if we are 10% sure AGI will arrive next year.

In both cases the AGI timeline considerations effectively discounts the slow interventions. With sufficient AGI implied discounting, fast interventions can become the most cost effective.

Simple optimal control model

I have created a simple optimal controlmodel that I describe in the appendix, and that you can play with here. The model (which I explain the appendix) takes parameters

The real interest on capital

The diminishing returns to spending on each of slow & fast interventions each year

The ratio of (benefit in the first year of the best fast intervention per cost) to (benefit in the first year of the best slow intervention per cost)

The rate at which slow interventions ‘wash out’ (alternatively framed as the degree to which their benefit is spread out over time). [Fast interventions are only beneficial in the first year by assumption].

AGI timelines

The probability that at some point we have a one year ‘heads-up’ on the arrival of AGI and so spend all our money on fast intervention

A non-AGI discount rate to account for non-AGI existential risks, philanthropic opportunities drying up, etc

The model returns the optimal spending on fast inventions (xF(t)) and slow interventions (xS(t)) each year.

Example model results

Example 1

This example above has AGI timelines (50% by 2040, 75% by 2061). The best fast interventions have 5x more impact in the first year than slow interventions, but 0.25x impact over the long run (modulo AGI timelines & discounting). The growth in our capital is 5% and there is a 3% discount rate. The diminishing returns to spending on fast and slow interventions are equal. We have a 50% chance of a one year AGI arrival heads-up.

Som...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How much should you optimize for the short-timelines scenario?, published by SoerenMind on July 26, 2022 on The Effective Altruism Forum.

(x-posted on LW)

Some have argued that one should tend to act as if timelines are short since in that scenario it's possible to have more expected impact. But I haven't seen a thorough analysis of this argument.

Question: Is this argument valid? And if yes how strong do you think it is?

The basic argument seems to be: if timelines are short, the field (AI alignment) will be relatively smaller and have made less progress. So you can pick more low-hanging fruits that wouldn't otherwise be picked.

The question affects career decisions. For example, if you optimize for long timelines, you can invest more time into yourself and delay your impact.

The question interacts with the following questions in somewhat unclear ways:

How fast do returns to additional work diminish (or increase)?

If returns don't diminish, the argument above fails.

If the field will grow very quickly, returns will diminish faster.

Is your work much more effective when it's early?

This may happen because work can be hard to parallelize - ‘9 women can't make a baby in 1 month’. And field-building can be more effective earlier as the field can compound-grow over time. So someone should start early.

If work is most effective earlier, you shouldn’t lose too much time investing in yourself.

Is work much more effective at crunch time?

If yes, you should focus more on investing in yourself (or do field-building for crunch time) instead of doing preparatory research.

If timelines are longer, is this evidence that we'll need a paradigm shift in ML that makes alignment easier/harder?

(This question seems less tractable than the others.)

Is your comparative advantage to optimize for short or long timelines?

For example, young people can contribute more easily given longer timelines and vice versa.

If someone would like to seriously research the overall question, please reach out. The right candidate can get funding.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.