Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levels of goals and alignment, published by zeshen on September 16, 2022 on The AI Alignment Forum.

This post was written as part of Refine. Thanks to Adam Shimi, Lucas Teixeira, Linda Linsefors, and Jonathan Low for helpful feedback and comments.

Epistemic status: highly uncertain. This post reflects my understanding of the terminologies and may not reflect the general consensus of AI alignment researchers (if any).

Motivation

I have been very confused with the various terminologies of alignment e.g. ‘inner alignment’, ‘outer alignment’, ‘intent alignment’, etc for the longest time. For instance, someone might talk about a particular example of inner misalignment and I would be left wondering how the presence of a mesa-optimizer was established, only to find out much later that we were never on the same page at all.

Through many more conversations with people and reading more posts, I thought I finally had cleared this up and was able to stop asking ‘what do you mean by inner alignment’ in every conversation I was in. However, this confusion came back to bite me when I came across the terms ‘robustness’ and ‘alignment’ as being different concepts in the context of ML safety.

In this post, I attempt to clarify my understanding on the different levels of goals and alignment, as well as give examples of each type of misalignment. I expect a lot of disagreements and I welcome suggestions to and pushback.

Levels of goals

Humanity’s ultimate terminal goals

These are the ultimate goals that humanity will have, given enough time to ruminate over them. They can be thought of as our true goals after a long reflection or humanity’s coherent extrapolated volition. To moral anti-realists, these ultimate goals may not even exist. Nevertheless, there are some goals that are almost certainly closer to these hypothetical ultimate goals than others (e.g. reducing suffering is almost certainly better than increasing it).

Current human intent / goals

These are the intentions and goals of humans that are somewhat in line to our values. In other words, these include the things we want and exclude the things we don’t want (e.g. we’d like to eat when we’re hungry but not to the point of puking).

Of course, there are different levels of human goals, where some are instrumental in different ways while others are more terminal. Humans are notoriously bad at knowing what we really want, but we shall ignore these issues for now and assume there is a coherent set of goals that we want.

AI’s base goals

These are the goals that the AI pursues after, whether it is trained or programmed. They can either be explicitly specified by humans or inferred through some kind of training process.

AI’s mesa-optimizer’s goals

These are the goals of a mesa-optimizer which may exist under certain conditions. This will be further discussed in the next section.

Levels of alignment

Solving philosophy

This is about aligning ‘what we currently want’ with ‘what we truly want if we had all the time and wisdom to think about it’.

An example of such a misalignment is when people in the previous centuries would want to own slaves, but we now know is morally reprehensible. A hypothetical AI fully aligned to human values at the time would probably be very effective slave-drivers.

An obvious issue with failing to solve moral philosophy is that a powerful AI fully aligned only to our current human values may lead us to some bad value lock-in scenarios, where it would continue to pursue presently set goals that we find morally reprehensible in the future. Proposed solutions to this problem include some form of corrigibility, where an AI allows its goals to be modified by its human programmers. That being said, if we can have an AGI that is only aligned to what we want now, it would already be a huge win.

This misalignment may a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ordering capability thresholds, published by Tamsin Leake on September 16, 2022 on The AI Alignment Forum.

(this post has been written for the third Refine blog post day)

given an AI which is improving towards ever more capabilities, such as by way of recursive self-improvement, in what order will it pass the following points?

throughout this post i'll be using PreDCA as an example of a formal goal to be maximized, because it appears to me as a potentially promising direction; but you can imagine adapting this post to other formal goals such as insulated goal-programs, or other alignment strategies altogether. we can even use this time-ordering framework to compare the various thresholds of multiple alignment strategies, though i won't do that here.

Start: we start the AI

Math: it can figure out relatively complicated math, such as whether P equals PSPACE, or whether this world looks like it has finite compute if we can make it do physics.

PreDCA: it can figure out what is entailed in maximizing PreDCA — notably that that goal is best entailed by not destroying the earth too much

sub-PreDCA: it can figure out some individual parts of PreDCA, such as the identity of the user or what is entailed in maximizing a human's utility function, in a way that we can use to modify those parts if they need adjusting

Escape: it becomes able to escape the environment over which we have control — and typically starts replicating across the internet

Influence: it gets the ability to significantly influence the timeline, for example enough to eg save us from facebook destroying everything six months later

DSA: it achieves decisive strategic advantage

Doom: it becomes capable of destroying the earth too much (without necessarily using that capability)

Cone: it takes over a significant portion of the universe, or at least of the lightcone

with a few notes:

"decisive strategic advantage" is a term i'm taking from bostrom's superintelligence book, describing the point at which an AI has sufficiently ensured its continuation that we can't turn it off or change its goals anymore; it is effectively the point of no return.

by "destroying the earth too much" i mean destroying so much of earth that it can't reasonably be resimulated. if resimulating earth is too unethical, computationally costly, or anthropically costly, then "destroying too much of earth" might straightforwardly mean destroying all of humankind or something like that. note that for PreDCA, preserving earth in some way is important not just because it's pretty bad that we all die, but also because the AI might need to preserve its user and possibly their environment in order to figure out their utility function.

in the case of knowing mathematical statements (Math, PreDCA, and sub-PreDCA), i imagine the AI being pretty sure about them, not necessarily having proven them. in addition, for simplicity, i'm assuming that we can use the AI to figure out some mathematical fact if and only if the AI can figure it out for itself — in practice, this need not be the case.

one thing that can be noticed is that humans might serve as evidence. for example, we can examine history to figure out whether we passed Math or would've been able to pass PreDCA (given a reasonable description of it) before getting to Doom — my guess is yes at least for that latter one.

now, we can reasonably guess the following pieces of ordering, where as usual in ordering graphs X Y means X < Y and transitive edges are not shown.

in addition, for any two quantities X < Y, it can be the case that they're pretty close in time X ≈ Y, or it can be that there's a bunch of time between them X ≪ Y. whether the threshold between those two possibilities is more like a day or a year, is gonna depend on context.

depending on how the rest of the ordering graph turns out and h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Pugwash Conferences and the Anti-Ballistic Missile Treaty as a case study of Track II diplomacy, published by rani martin on September 16, 2022 on The Effective Altruism Forum.

Track II (unofficial) diplomacy is potentially a useful intervention for reducing risks of great power conflict. In this post, I discuss what a paradigmatic case of track II diplomacy tells us about the value of this intervention. This case is the influence of the Pugwash Conferences on the Soviet government agreeing to place limitations on anti-ballistic missiles.

I first motivate interest in track II diplomacy as an intervention. I then identify two important theories of change for track II diplomacy and claims about its effects which support these theories of change. I test these claims using the Pugwash case - largely discussing and scrutinising the evidence and narrative about the impact of Pugwash from political science Professor Matthew Evangelista's authoritative work on the subject, 'Unarmed Forces'. I argue that this case suggests high quality access to official decision makers, windows of opportunity for influence over policy, and the development of trusting communities of non-officials are important to the success of track II diplomacy. This case should make us more enthusiastic about using track II diplomacy to reduce risks of great power conflict.

1. Introduction

1.1 Risks from great power conflict

Great power conflict is an existential risk factor: the probability of an existential catastrophe grows as a major conflict becomes more likely. There are several pathways by which great power conflict raises existential risk. For example, a war between great powers could cause or accelerate the deployment of weapons with catastrophic potential, including bioweapons and nuclear weapons. Tensions or war between great powers would make it harder to coordinate on the safe development of artificial intelligence and other destabilising emerging technologies.¹

If, as seems plausible,² great power conflict significantly contributes to total existential risk, interventions to reduce the probability of such conflict could significantly decrease existential risk.

1.2 Track II diplomacy and great power conflict

Track II diplomacy is a potentially-effective intervention to reduce risks from great power conflict. It is unofficial policy discussion, usually between parties from different sides of an international conflict or representing groups that are in tension with each other.³ Though they are not officials themselves, Track II participants can influence official actions by sharing information and policy ideas, and by advocating for their preferred approach to their contacts back home. Their non-official status distinguishes Track II diplomacy programs from Track I (official) diplomacy. Track 1.5 dialogues are unofficial discussions which include official participants.⁴

Philanthropists have funded Track II diplomacy,⁵ suggesting that, if effective, it could be a good way to use philanthropic funding to reduce risks from great power conflict. However, few formal evaluations of the effectiveness of Track II diplomacy programs exist. In this report, I present a case study of a paradigmatic example of Track II diplomacy: the influence of the Pugwash Conferences on Science and World Affairs on the Soviet government’s decision to sign the 1972 Anti-Ballistic Missile Treaty.⁶ While a case study approach raises concerns about generalisability, it can be useful when it is not feasible to conduct more systematic evaluations, such as randomised controlled trials. I use this case to test several assumptions about the effect of Track II diplomacy on relations between the countries involved.

In the rest of this post, I first briefly discuss existing work on the effectiveness and theory of change of Track ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Representational Tethers: Tying AI Latents To Human Ones, published by Paul Bricman on September 16, 2022 on The AI Alignment Forum.

This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by ideological inference engines.

Thanks Adam Shimi, Alexander Oldenziel, Tamsin Leake, and Ze Shen for useful feedback.

TL;DR: Representational tethers describe ways of connecting internal representations employed by ML models to internal representations employed by humans. This tethering has two related short-term goals: (1) making artificial conceptual frameworks more compatible to human ones (i.e. the tension in the tether metaphor), and (2) facilitating direct translation between representations expressed in the two (i.e. the physical link in the tether metaphor). In the long-term, those two mutually-reinforcing goals (1) facilitate human oversight by rendering ML models more cognitively ergonomic, and (2) enable control over how exotic internal representations employed by ML models are allowed to be.

Intro

The previous two proposals in the sequence describe means of deriving human preferences procedurally. Oversight leagues focus on the adversarial agent-evaluator dynamics as the process driving towards the target. Ideological inference engines focus on the inference algorithm as the meat of the target-approaching procedure. A shortcoming of this procedural family is that even if you thankfully don't have to plug in the final goal beforehand (i.e. the resulting evaluator or knowledge base), you still have to plug in the right procedure for getting there. You're forced to put your faith in a self-contained preference-deriving procedure instead of an initial target.

In contrast, the present proposal tackles the problem from a different angle. It describes a way of actively conditioning the conceptual framework employed by the ML model to be compatible with human ones, as an attempt to get the ML model to form accurate conceptions of human values. If this sounds loosely relates to half a dozen other proposals, that's because it is — consider referring to the Discussion for more details on tangents. In the meantime, following the structure of the previous posts in the sequence, here are some assumptions underlying representational tethers:

Assumption 1, "Physicalism": Our thoughts are represented as neural dynamics. In the limit of arbitrarily large amounts of data on neural dynamics aligned with external stimuli (in the sense of parallel corpora), our thoughts can be accurately reconstructed.

Assumption 2, "Bottleneck Layer": There is a bottleneck layer in the architecture of the ML model being tethered to human representations. This bottleneck refers to a low-dimensional representation through which all the information being processed by the ML model is forced to pass.

Assumption 3, "AGI Hard, Human Values Harder": We are unlikely to formulate the True Name of human values in closed-form before deploying transformative AI. The best we are likely to do before takeoff is model human values approximately and implement an imperfect evaluator.

Proposal

Representational tethers suggest a way of aligning human and AI latents for the purpose of facilitating later interaction. There are two steps to this:

First Bring Them Closer

Incentivize the ML model to employ internal representations which are compatible with human ones, thus bringing them "closer." This can be operationalized by conditioning latent activations which arise in the ML model to be expressible in human representations. Concretely, optimization pressure would be exerted on the ML model to push it to internalize a conceptual framework which can successfully be translated to and from a human one without significant loss of information. If the artificial repre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dan Luu: Futurist prediction methods and accuracy, published by Linch on September 15, 2022 on The Effective Altruism Forum.

tl;dr: Dan Luu has a detailed post where he tracks in detail past predictions and argues that contra Karnofsky, Arb, etc, the track record of futurists is overall quite bad. Relevantly to this audience, he further argues that this is evidence against the validity of current longtermist efforts in long-range predictions.

(I have not finished reading the post).

I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.

The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists1) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibility of predictions that use similar methodologies.

Unfortunately, every single predictor from the independently chosen set had a poor record and, on spot checking some predictions from other futurists, it appears that futurists often have a fairly poor track record of predictions so, in order to contrast techniques that worked with techniques that I didn't, I sourced predictors that have a decent track record from my memory, an non-independent source which introduces quite a few potential biases.

Something that gives me more confidence than I'd otherwise have is that I avoided reading independent evaluations of prediction methodologies until after I did the evaluations for this post and wrote 98% of the post and, on reading other people's evaluations, I found that I generally agreed with Tetlock's "Superforecasting" on what worked and what didn't work despite using a wildly different data set.

In particular, people who were into "big ideas" who use a few big hammers on every prediction combined with a cocktail party idea level of understanding of the particular subject to explain why a prediction about the subject would fall to the big hammer generally fared poorly, whether or not their favored big ideas were correct. Some examples of "big ideas" would be "environmental doomsday is coming and hyperconservation will pervade everything", "economic growth will create near-infinite wealth (soon)", "Moore's law is supremely important", "quantum mechanics is supremely important", etc. Another common trait of poor predictors is lack of anything resembling serious evaluation of past predictive errors, making improving their intuition or methods impossible (unless they do so in secret).

Instead, poor predictors often pick a few predictions that were accurate or at least vaguely sounded similar to an accurate prediction and use those to sell their next generation of predictions to others.

By contrast, people who had (relatively) accurate predictions had a deep understanding of the problem and also tended to have a record of learning lessons from past predictive errors. Due to the differences in the data sets between this post and Tetlock's work, the details are quite different here. The predictors that I found to be relatively accurate had deep domain knowledge and, implicitly, had access to a huge amount of information that they filtered effectively in order to make good predictions. Tetlock was studying peo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Animations' Script Writing Contest, published by Writer on September 15, 2022 on The Effective Altruism Forum.

Introduction

I'm announcing a script writing contest for the Rational Animations YouTube channel. Write scripts of 2500 words maximum about a topic of longtermist relevance, and win:

5000 USD.

We will make an animated video based on your script to be published on the YouTube channel with due credits. It will be animated by a team of 9 animators or an animation studio and will be of much higher quality than anything we've done so far.

A potential job offer as a scriptwriter for the channel.

Bonus: you may suggest other people's already written work. If we bring your suggestion to the channel as it is (without having to adapt it), you will get $500, but nothing if I am already considering the work.

Soft deadline: October 15th.

Topics

The topic of your script should be of longtermist relevance. Examples: existential risks of any kind, the future of humanity, philosophy related to longtermism, technologies of longtermist relevance. I don't mean to be too strict, I'll accept scripts about a broad range of topics, and I might even make exceptions and offer a prize to scripts not relevant to longtermism if I find them a lot better than the competition. As a reference, the videos already on the channel that I would classify as longtermism-relevant are:

Can we make the future a million years from now go better?

Everything might change forever this century (or we’ll go extinct)

Will we grab the universe? Grabby aliens predictions.

Humanity was born way ahead of its time. The reason is grabby aliens.

We are failing to see how much better off humanity could be

Longtermism: an idea that could save 100 billion trillion lives

Length and number of scripts

The maximum length of the script should be 2500 words. Of course, you may split a longer script into two or more parts. You may send an unlimited number of scripts.

Are there any more restrictions?

Not really.

So far, we've only published explainers on the YT channel, but you don't have to limit yourself to explainers. Do you want to write a Fable of the Dragon-Tyrant style short story? OK. Do you want to write an anime episode? OK. You can do whatever you want.

You may also adapt or send older work you've written that you think would make a good video for the YouTube channel if narrated as it is or with some tweaks.

Assorted things that make a good script, in order of importance:

For any claim, you should include sources. The script should be as epistemically legible as possible for the audience.

Focus on what really matters. If you write about complex topic X, there will be a few core points that are necessary and sufficient to build a solid understanding of topic X. Focus on those points. Yes, I understand this is a bit vague as a suggestion.

Don't simplify, but explain. Write for a really smart audience that is ignorant about the topic. Some background explanations can be skipped, and some others are required. Recognizing the difference is a bit of an art.

Be concise, and avoid useless words and filler phrases.

Telling stories and using concrete details helps with engagement and works better for teaching.

How you start matters: if the beginning of the script is boring, that's especially bad for engagement on YouTube.

Challenge the preconceived assumptions a viewer might have about a topic, if the topic lends itself to it.

See old topics from new surprising angles, but as with point 7, don't force it.

How to send scripts

Send your scripts to [email protected] (beware of the spelling, it's "rationalanimations", not "rationalanimation").

In addition to sending the script to the YouTube channel's e-mail, you may also publish it as a comment under this post or as a top-level post in the EA Forum or Less ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 9/15/22: Permanent Normal, published by Zvi on September 15, 2022 on LessWrong.

I may or may not have had Covid about two weeks ago.

At the time, I knew I had come down with something. I largely lost my voice and spent about a week carefully conserving what remained. I spent over a week coughing a decent amount. I was having some trouble breathing when any pressure was applied to my chest and occasionally at other times.

I never felt the situation was that serious and thought it probably wasn’t Covid, and never got tested or took any treatment beyond the normal resting and staying home for a minor illness plus resting my voice, expecting it to get better. Which it did.

My personal trainer, however, immediately assumed that of course it was Covid. It certainly is possible. In the end, it does not much matter. I presume I will never know for sure.

I am fully recovered now, except that I lost some progress exercising, especially on the elliptical machine, and am still working to get back to where I was before, which I anticipate will take another week or two.

This is permanent normal. We get sick. We don’t know exactly what it was. Over time, we mostly feel better. There is not much Covid news, so it seems fine to have the introduction be a personal tale like this.

That also, combined with doing some longer term work on various things, is why I haven’t been putting out any additional content beyond the weekly posts for a bit. That will change. I also am considering how much I should be ‘spinning off’ various things into their own posts.

Executive Summary

Covalent boosters now available wherever vaccines are administered.

Covid cases dropped unexpectedly.

The dog that didn’t bark: Zero monkeypox news of note.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 490k cases (+9%) and 3,000 deaths (+10%)

Results: 389k cases (-14%) and 2,878 deaths (+5%).

Prediction for next week: 350k cases (-10%) and 2,625 deaths (-9%).

California normally reports cases twice a week and this week only reported once. I am going to assume this is a delay and conservatively added 20,000 additional cases and 70 additional deaths. These will be subtracted from next week’s numbers, unless I discover something weird going on that makes me do something else instead. Deaths are typically jumpy, so only making the adjustment there because of the very obvious need to adjust cases. Not going to worry too much about it all given the low value of information.

I cannot think of a reason for general lack of reporting last week, and California has been adjusted. This reduction is on top of coming back from Labor Day weekend, so I am going to assume it is a real drop, and expect it to continue faster than expected. Deaths should also be steadily falling.

Deaths

Cases

Physical World Modeling

Bloom Labs thread about origins of Omicron. Makes a strong case.

Bloom Lab reports that mouse tests show updated vaccines work better against BA.5.

New Long Covid research initiative, currently has $15 million in funding and looking to get to $100 million, with a focus on exploring treatments. There is shade thrown towards RECOVER, saying it is being slow to recruit patients, and also the implication it won’t be taking enough active steps, and so a smaller but less risk averse effort is needed and can do better. This seems likely to me. Post includes CDC estimate that 40% or more of Americans have had Covid, which seems quite low.

Thread and Atlantic article on brain fog. To extent people do have brain fog it is indeed a major drag on ability to live life and be productive. Doesn’t update me on how often this is caused by Covid.

South African results find vaccination surprisingly ineffective and fast to fade. (Study)

These numbers are against severe disease not infection.

Whenever I see someone...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We’re still (extremely) funding constrained (but don’t let fear of getting funding stop you trying)., published by Luke Freeman on September 16, 2022 on The Effective Altruism Forum.

Lots has been written about this so I wrote a poem instead. I lead the team at Giving What We Can, views are my own.

Poem

Years ago we were struck by big problems: they were so extremely funding constrained.

One-by-one we saw a big impact: by making them less extremely funding constrained.

We didn't wait for permission, we gave from our own pockets first. It became our mission to put others first.

Our spendthrift community dug into the data. We made money go further, we made things go better.

Each dollar paid dividends, each DALY gained a good end. Progress felt slow, but was needed, we know.

Constraints were consistent, opportunity cost felt: "Should I pledger further? Should I become a researcher?"

A driven community with compassion so big: we found more neglected problems, solvable, and big.

We said "more research needed", traded money for time: found researchers, founders and then funders aligned.

Some problems found funders more quickly than founders, yet others found moneypits so desperate to fill.

Give trillions in cash or keep coal in the ground? What about the backlash if our decisions aren't sound?

As we made progress we hit the mainstream. Among the first questions: "Why's my cause unseen?"

We're resource constrained, I wish it weren't such: "Yes, we want to help everyone, but we only have so much!"

Our work's still so small in the scheme of the world. Still, let's be more ambitious: let's build a dreamworld.

We need many folks to be stoaked by our mission. We need many funders, founders, and passion.

Experimentation is something we now know we can try: don't let fear of funding be why you don't apply.

But for the foreseeable future your dollars still count: for every life that you help we mustn't discount.

Our mission 'aint over, we're at the start of our road. We need your help: let's make some inroads.

So give what you can and get others involved. Let’s keep working together to get these problems solved.

Postscript

It can be quite difficult to ‘feel’ the fact that all of these things are true at the same time:

We have increased available funding by an order of magnitude over the past decade and increased the rate at which that funding is being deployed

We don't want lack of funds to be the reason that people don't do important and ambitious things; and yet

Yet in most cases we are still extremely funding constrained

I find it painful (and counter-productive) to see these messages floating around:

EA has “too much money”

EA has “more money than it knows what to do with”

There is “such a glut of money in EA right now”

EA has a “funding overhang”

Donations aren’t needed, or they don’t count

Pursuing a high-impact careers is mutually exclusive with donating (although there are tradeoffs)

If you don’t get funded it means your project is not worthwhile (in some cases it could be just below a bar)

If you miss out on a job when “there’s plenty of money” it means there’s nothing for you (some organisations are still very funding constrained and may have happily hired you if they were less funding constrained)

Whereas I think the better (more truthful and constructive narratives) are:

We have a more decent shot at having a significant impact

We have more resources which helps us:

Double down on things we have good evidence for

Justifiably spending more on research and experimentation

Become more diverse (e.g. doesn’t require someone to have enough personal resources to take big risks, we can fund people to attend a conference/retreat they couldn’t otherwise afford etc) and therefore find more excellent people to participate in this grand project.

The situation is nuanced:

The funding...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longterm cost-effectiveness of Founders Pledge's Climate Change Fund, published by Vasco Grilo on September 14, 2022 on The Effective Altruism Forum.

Summary

This analysis estimates the cost-effectiveness of reducing existential risk via the Climate Change Fund (CCF) of Founders Pledge (FP) (see Methodology).

The results were obtained with this Colab, and the key ones are summarised in the table below (for more, see Results and Discussion). Comments about how to interpret them are welcome.

The cost-effectiveness bar for reducing existential risk (8 kt/$) is estimated to be 3 times as high as (an overestimate for?) the cost-effectiveness of CCF (2 kt/$) (see Discussion). This suggests the best interventions to fight climate change are not amongst the most effective ways of reducing existential risk.

ResultMeanExistential risk due to climate change (bp)1.00Cumulative GHG emissions between 2020 and 2100 (Tt)3.76Existential risk reduction caused by removing GHG emissions (bp/Tt)0.273Cost-effectiveness of removing GHG emissions via CCF (t/$)2.34 kCost-effectiveness of CCF (bp/G$)0.640Cost-effectiveness bar for reducing existentiat risk (bp/G$)2.17Cost-effectiveness bar for removing GHG emissions (t/$)7.94 k

Acknowledgements

Thanks to Alexey Turchin, Anonymous, David Denkenberger, Johannes Ackva, Luke Kemp, and Nuño Sempere.

Introduction

Removing greenhouse gas (GHG) emissions decreases heat-related deaths: Bressler 2021 estimated the 2020 mortality cost of carbon to be 2.2610^-4 life/t. However, most of the benefits of reducing GHG emissions respect the decrease in existential risk due to climate change.

This analysis estimates the existential risk reduction caused by removing GHG emissions, and the cost-effectiveness of CCF, which is compared with an estimate for the cost-effectiveness bar for reducing existential risk. Nonetheless, it should be noted that, according to this comment from Johannes Ackva:

The goal of the Climate [Change] Fund is to optimize spending that is not cause-neutral, to have as big as possible a positive impact given constraints, it is not intended or marketed as "the top bet on reducing existential risk" and we are careful to not crowd in resources that would otherwise go to areas we think of as higher marginal priority.

I encourage the readers to check this comment from Matt Lerner, FP's research director, to better understand FP's mission:

To be absolutely clear, FP's goal is to do the maximum possible amount of good, and to do so in a cause-neutral way.

Methodology

The (marginal) cost-effectiveness of CCF was estimated from the product between:

The existential risk reduction caused by removing GHG emissions, which was estimated from the ratio between:

The existential risk due to climate change.

The cumulative GHG emissions between 2020 and 2100 assuming current climate policies.

The cost-effectiveness of removing GHG emissions via CCF, which was estimated from:

The reciprocal of the cost to remove GHG emissions via CCF.

The assumption regarding the cumulative GHG emissions is consistent with the current Metaculus' cummunity prediction of 2.6 ºC for how much greater (in ˚C) will the average global temperature in 2100 be than the average global temperature in 1880. According to the Climate Action Tracker (CAT), current climate policies are predicted to result in a global warming between 2.5 ºC and 2.9 ºC (interval which contains 2.6 ºC). The modelling of the cumulative GHG emissions would ideally consider more scenarios.

Existential risk due to climate change

The existential risk due to climate change was modelled as a beta distribution with:

Mean equal to 0.01 %, which was determined from the geometric mean between:

The 0.1 % guessed by Toby Ord in The Precipice for the next 100 years (2021 to 2120).

The upper bound of 0.01 % guessed by 80,000 ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding Conjecture: Notes from Connor Leahy interview, published by Akash on September 15, 2022 on LessWrong.

I recently listened to Michaël Trazzi interview Connor Leahy (co-founder & CEO of Conjecture)'s on a podcast called The Inside View (Youtube video here; full video & transcript here).

The interview helped me better understand Connor’s worldview and Conjecture’s theory of change.

I’m sharing my notes below. The “highlights” section includes the information I found most interesting/useful. The "full notes" section includes all of my notes.

Disclaimer #1: I didn’t take notes on the entire podcast. I selectively emphasized the stuff I found most interesting. Note also that these notes were mostly for my understanding, and I did not set out to perfectly or precisely capture Connor’s views.

Disclaimer #2: I’m always summarizing Connor (even when I write with “I” or “we”— the “I” refers to Connor). I do not necessarily endorse or agree with any of these views.

Highlights

Timelines

20-30% in the next 5 years. 50% by 2030. 99% by 2100. 1% we already have it (but don’t know this yet). Higher uncertainty than Eliezer but generally buys the same arguments. Is mostly like “Eliezer’s arguments seem right but how can anyone be so confident about things?”

Thoughts on MIRI Dialogues & Eliezer’s style

An antimeme is something that by its very nature resists being known. Most antimemes are just boring—things you forget about. If you tell someone an antimeme, it bounces off them. So they need to be communicated in a special way. Moral intuitions. Truths about yourself. A psychologist doesn’t just tell you “yo, you’re fucked up bro.” That doesn’t work.

A lot of Eliezer’s value as a thinker is that he notices & comprehends antimemes. And he figures out how to communicate them.

What happened in the MIRI dialogues is that Eliezer was telling Paul “hey, I’m trying to communicate an antimeme to you, but I’m failing because it’s really really hard.”

Thoughts on Death with Dignity & optimizing for “dignity points” rather than utility

The Death with Dignity post is a perfect example of an antimeme. A great way to convey antimemes is through jokes and things outside the Overton Window.

The antimeme is that utilitarianism is hard, and no, it’s not actually a good idea to advocate for really stupid “pivotal acts” that sound ridiculous. Consequentialism is really hard. I have to reason about all of my possible choices and all of their possible consequences. If you have an infinitely big brain, this works. If not, it doesn’t.

It’s too computationally hard to be a perfect consequentialist. And being an imperfect consequentialist is really really bad. If you do one step of reasoning, you might be like “yeaaa let’s get rid of GPUs!” But you don’t realize how that would be super bad for the world, would make cooperation extremely difficult, would make everything become super secretive, etc.

The antimeme is that most people shouldn’t be thinking like consequentialists. Instead of thinking about how to maximize utility, they should be thinking about how to maximize dignity. This is easier. This is computationally tractable. This heuristic will make you do better.

I see so many people come into this arena with the anime protagonist “I’m going to save the world” complex, and then they burnout after 3 months and go do DMT.

I know two humans who can maybe reason better under the consequentialist frame. But for everyone else, if you’re going to do 5 years of soul-crushing difficult research without much support from the outside world, you should think under the dignity frame.

Thoughts on the importance of playing with large models

One mistake I see people make is that they underestimate the importance of getting actual hands-on experience with the thing you are studying. I think it’s important to ...