Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI in sight: our look at the game board, published by Andrea Miotti on February 18, 2023 on The AI Alignment Forum.

From our point of view, we are now in the end-game for AGI, and we (humans) are losing. When we share this with other people, they reliably get surprised. That’s why we believe it is worth writing down our beliefs on this.

1. AGI is happening soon. Significant probability of it happening in less than 5 years.

Five years ago, there were many obstacles on what we considered to be the path to AGI.

But in the last few years, we’ve gotten:

Powerful Agents (Agent57, GATO, Dreamer V3)

Reliably good Multimodal Models (StableDiffusion, Whisper, Clip)

Just about every language tasks (GPT3, ChatGPT, Bing Chat)

Human and Social Manipulation

Robots (Boston Dynamics, Day Dreamer, VideoDex, RT-1: Robotics Transformer )

AIs that are superhuman at just about any task we can (or simply bother to) define a benchmark, for

We don’t have any obstacle left in mind that we don’t expect to get overcome in more than 6 months after efforts are invested to take it down.

Forget about what the social consensus is. If you have technical understanding of current AIs, do you truly believe there are any major obstacles left? The kind of problems that AGI companies could reliably not tear down with their resources? If you do, state so in the comments, but please do not state what those obstacles are.

2. We haven’t solved AI Safety, and we don’t have much time left.

We are very close to AGI. But how good are we at safety right now? Well.

No one knows how to get LLMs to be truthful. LLMs make things up, constantly. It is really hard to get them not to do this, and we don’t know how to do this at scale.

Optimizers quite often break their setup in unexpected ways. There have been quite a few examples of this. But in brief, the lessons we have learned are:

Optimizers can yield unexpected results

Those results can be very weird (like breaking the simulation environment)

Yet very few extrapolate from this and find these as worrying signs

No one understands how large models make their decisions. Interpretability is extremely nascent, and mostly empirical. In practice, we are still completely in the dark about nearly all decisions taken by large models.

RLHF and Fine-Tuning have not worked well so far. Models are often unhelpful, untruthful, inconsistent, in many ways that had been theorized in the past. We also witness goal misspecification, misalignment, etc. Worse than this, as models become more powerful, we expect more egregious instances of misalignment, as more optimization will push for more and more extreme edge cases and pseudo-adversarial examples.

No one knows how to predict AI capabilities. No one predicted the many capabilities of GPT3. We only discovered them after the fact, while playing with the models. In some ways, we keep discovering capabilities now thanks to better interfaces and more optimization pressure by users, more than two years in. We’re seeing the same phenomenon happen with ChatGPT and the model behind Bing Chat.

We are uncertain about the true extent of the capabilities of the models we’re training, and we’ll be even more clueless about upcoming larger, more complex, more opaque models coming out of training. This has been true for a couple of years by now.

3. Racing towards AGI: Worst game of chicken ever.

The Race for powerful AGIs has already started. There already are general AIs. They just are not powerful enough yet to count as True AGIs.

Actors

Regardless of why people are doing it, they are racing for AGI. Everyone has their theses, their own beliefs about AGIs and their motivations. For instance, consider:

AdeptAI is working on giving AIs access to everything. In their introduction post, one can read “True general intelligence requires models that can no...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parametrically retargetable decision-makers tend to seek power, published by TurnTrout on February 18, 2023 on LessWrong.

This paper—accepted as a poster to NeurIPS 2022— is the sequel to Optimal Policies Tend to Seek Power. The new theoretical results are extremely broad, discarding the requirements of full observability, optimal policies, or even requiring a finite number of options.

Abstract:

If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal.

We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable, and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad.

We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.

Examples of agent designs the power-seeking theorems now apply to:

Boltzmann-rational agents,,

Expected utility maximizers and minimizers,

Even if they uniformly randomly sample a few plans and then choose the best sampled

Satisficers (as I formalized them),

Quantilizing with a uniform prior over plans, and

RL-trained agents under certain modeling assumptions.

The key insight is that the original results hinge not on optimality per se, but on the retargetability of the policy-generation process via a reward or utility function or some other parameter. See Satisficers Tend To Seek Power: Instrumental Convergence Via Retargetability for intuitions and illustrations.

Why am I only now posting this?

First, I've been way more excited about shard theory. I still think these theorems are really cool, though.

Second, I think the results in this paper are informative about the default incentives for decision-makers which "care about things." IE, make decisions on the basis of e.g. how many diamonds that decision leads to, or how many paperclips, and so on. However, I think that conventional accounts and worries around "utility maximization" are subtly misguided. Whenever I imagined posting this paper, I felt like "ugh sharing this result will just make it worse." I'm not looking to litigate that concern right now, but I do want to flag it.

Third, Optimal Policies Tend to Seek Power makes the "reward is the optimization target" mistake super strongly. Parametrically retargetable decision-makers tend to seek power makes the mistake less hard, both because it discusses utility functions and learned policies instead of optimal policies, and also thanks to edits I've made since realizing my optimization-target mistake.

Conclusion

This paper isolates the key mechanism—retargetability—which enables the results in Optimal Policies Tend to Seek Power. This paper also takes healthy steps away from the optimal policy regime (which I consider to be a red herring for alignment) and lays out a bunch of theory I found—and still find—beautiful.

This paper is both published in a top-tier conference and, unlike the previous paper, actually has a shot of being applicable to realistic agents and training processes. Therefore, compared to the original optimal policy paper, I think th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parametrically retargetable decision-makers tend to seek power, published by Alex Turner on February 18, 2023 on The AI Alignment Forum.

This paper—accepted as a poster to NeurIPS 2022— is the sequel to Optimal Policies Tend to Seek Power. The new theoretical results are extremely broad, discarding the requirements of full observability, optimal policies, or even requiring a finite number of options.

Abstract:

If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal.

We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable, and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad.

We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.

Examples of agent designs the power-seeking theorems now apply to:

Boltzmann-rational agents,,

Expected utility maximizers and minimizers,

Even if they uniformly randomly sample a few plans and then choose the best sampled

Satisficers (as I formalized them),

Quantilizing with a uniform prior over plans, and

RL-trained agents under certain modeling assumptions.

The key insight is that the original results hinge not on optimality per se, but on the retargetability of the policy-generation process via a reward or utility function or some other parameter. See Satisficers Tend To Seek Power: Instrumental Convergence Via Retargetability for intuitions and illustrations.

Why am I only now posting this?

First, I've been way more excited about shard theory. I still think these theorems are really cool, though.

Second, I think the results in this paper are informative about the default incentives for decision-makers which "care about things." IE, make decisions on the basis of e.g. how many diamonds that decision leads to, or how many paperclips, and so on. However, I think that conventional accounts and worries around "utility maximization" are subtly misguided. Whenever I imagined posting this paper, I felt like "ugh sharing this result will just make it worse." I'm not looking to litigate that concern right now, but I do want to flag it.

Third, Optimal Policies Tend to Seek Power makes the "reward is the optimization target" mistake super strongly. Parametrically retargetable decision-makers tend to seek power makes the mistake less hard, both because it discusses utility functions and learned policies instead of optimal policies, and also thanks to edits I've made since realizing my optimization-target mistake.

Conclusion

This paper isolates the key mechanism—retargetability—which enables the results in Optimal Policies Tend to Seek Power. This paper also takes healthy steps away from the optimal policy regime (which I consider to be a red herring for alignment) and lays out a bunch of theory I found—and still find—beautiful.

This paper is both published in a top-tier conference and, unlike the previous paper, actually has a shot of being applicable to realistic agents and training processes. Therefore, compared to the original optimal policy pa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VII: A Challenge for Mechanists, published by Stephen Casper on February 18, 2023 on The AI Alignment Forum.

Part 7 of 12 in the Engineer’s Interpretability Sequence.

Thanks to Neel Nanda. I used some very nicely-written code of his from here. And thanks to both Chris Olah and Neel Nanda for briefly discussing this challenge with me.

MI = “mechanistic interpretability”

Given a network, recover its labeling function.

In the last post, I argued that existing works in MI focus on solving problems that are too easy. Here, I am posing a challenge for mechanists that is still a toy problem but one that is quite a bit less convenient than studying a simple model or circuit implementing a trivial, known task. The the best of my knowledge:

Unlike prior work on MI from the AI safety interpretability community, beating this challenge would be the first example of mechanistically explaining a network’s solution to a task that was not cherrypicked by the researcher(s) doing so.

Gaining a mechanistic understanding of the models in this challenge may be difficult, but it will probably be much less difficult than mechanistically interpreting highly intelligent systems in high stakes settings in the real world. So if an approach can’t solve the type of challenge posed here, it may not be very promising for doing much heavy lifting with AI safety work.

This post comes with a GitHub repository. Check it out here. The challenge is actually two challenges in one, and the basic idea is similar to some ideas presented in Lindner et al. (2023).

Challenge 1, MNIST CNN

I made up a nonlinear labeling function that labels approximately half of all MNIST images as 0’s and the other half as 1’s. Then I trained a small CNN on these labels, and it got 96% testing accuracy. The challenge is to use MI tools on the network to recover that labeling function.

Hint 1: The labels are binary.

Hint 2: The network gets 95.58% accuracy on the test set.

Hint 3: This image may be helpful.

Challenge 2, Transformer

I made up a labeling function that takes in two integers from 0 to 113 and outputs either a 0 or 1. Then, using a lot of code from Neel Nanda’s grokking work, I trained a 1-layer transformer on half of the data. It then got 97% accuracy on the test half. As before, the challenge is to use MI tools to recover the labeling function.

Hint 1: The labels are binary.

Hint 2: The network is trained on 50% of examples and gets 97.27% accuracy on the test half.

Hint 3: Here are the ground truth and learned labels. Notice how the mistakes the network makes are all near curvy parts of the decision boundary...

Prizes

If you are the first person to send me the labeling function and a mechanistic explanation for either challenge, I will sing your praises on my Twitter, and I would be happy to help you write a post about how you solved a problem I thought would be very difficult. Neel Nanda and I are also offering a cash prize. (Thanks to Neel for offering to contribute to the pool!) Neel will donate $250, and I will donate $500 to a high-impact charity of choice for the first person to solve each challenge. That makes the total donation prize pool $1,500.

Good luck

For this challenge, I intentionally designed the labeling functions to not be overly simple. But I will not be too surprised if someone reverse-engineers them with MI tools, and if so, I will be extremely interested in how.

Neither of the models perfectly label the validation set. One may object that this will make the problem unfairly difficult because if there is no convergence on the same behavior as the actual labeling function, then how is one supposed to find that function inside the model? This is kind of the point though. Real models that real engineers have to work with models don’t tend to conveniently grok onto a simple, elegant, programmat...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPT-4 Predictions, published by Stephen McAleese on February 17, 2023 on LessWrong.

Introduction

GPT-4 is OpenAI’s next major language model which is expected to be released at some point in 2023. My goal here is to get some idea of when it will be released and what it will be capable of. I also think it will be interesting in retrospect to see how accurate my predictions were. This post is partially inspired by Mathew Barnett’s GPT-4 Twitter thread which I recommend reading.

Background of GPT models

GPT-1, GPT-2, GPT-3

GPT stands for generative pre-trained transformer and is a family of language models that were created by OpenAI. GPT was released in 2018, GPT-2 in 2019, and GPT-3 in 2020. All three models have used a similar architecture with some relatively minor variations: a dense, text-only, decoder transformer language model that’s trained using unsupervised learning to predict missing words in its text training set .

InstructGPT, GPT-3.5, ChatGPT

Arguably one of the biggest changes in the series in terms of architecture and behavior was the release of InstructGPT in January 2022 which used supervised fine-tuning using model answers and reinforcement learning with human feedback where model responses are ranked in addition to the standard unsupervised pre-training.

The GPT-3.5 models finished training and were released in 2022, and demonstrated better quality answers than GPT-3. In late 2022, OpenAI released ChatGPT which is based on GPT-3.5 and fine-tuned for conversation.

When will GPT-4 be released?

Sam Altman, the CEO of OpenAI, was interviewed by StrictlyVC in January 2023. When asked when GPT-4 would come out, he replied, “It will come out at some point when we are confident that we can do it safely and responsibly.”

Metaculus predicts a 50% chance that GPT-4 will be released by May 2023 and a ~93% chance that it will be released by the end of 2023.

It seems like there’s still quite a lot of uncertainty here but I think we can be quite confident that GPT-4 will be released at some point in 2023.

What will GPT-4 be like?

Altman revealed some more details about GPT-4 at an AC10 meetup Q&A. He said:

GPT-4 will be a text-only model like GPT-3.

GPT-4 won’t be much bigger than GPT-3 but will use much more compute and have much better performance.

GPT-4 will have a longer context window.

How capable will GPT-4 be?

Scaling laws

According to the paper Scaling Laws for Neural Language Models (2020), model performance as measured by cross-entropy loss can be calculated from three factors: the number of parameters in the model, the amount of compute used during training, and the amount of training data. There is a power-law relationship between these three factors and the loss. Basically, this means you have to increase the amount of compute, data, and parameters by a factor of 10 to decrease the loss by one unit, by 100 to decrease the loss by two units, and so on. The authors of the paper recommended training very large models on relatively small amounts of data and recommended investing compute into more parameters over more training steps or data to minimize loss as shown in this diagram:

For every 10x increase in compute, the paper approximately recommends increasing the number of parameters by 5x, the number of training tokens by 2x, and the number of serial training steps by 1.2x. This explains why the original GPT-3 model and other models such as Megatron and PaLM were so large.

However, the new scaling laws from DeepMind’s 2022 paper Training Compute Optimal Language Models instead emphasize the importance of training data for minimizing loss. Instead of prioritizing more parameters, the paper recommends scaling the number of parameters and training tokens equally.

DeepMind originally trained a large 280B parameter model named Gopher but then found a 70B mo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: People Will Sometimes Just Lie About You, published by aella on February 18, 2023 on The Effective Altruism Forum.

Before getting mini-famous, I did not appreciate the degree to which people would misrepresent and lie about other people.

I knew about it in theory. I occasionally stumbled across claims about people that I later found out were false. I knew, abstractly, that any particular accuser is telling a truth or a lie, but you're not sure which one.

But now, I'm in the unique position where people say things about me all the time, and I hear most of it, and I have direct access to whether it's accurate or not. There's something about the lack of ambiguity that has left me startled, here. Something was way off about my models of the world before I had access to the truth of a wide range of accusational samples.

In the last few years, I've risen in visibility to the degree it's started to get unpleasant. I've had to give up on the idea of throwing parties at my house where guests are allowed to invite unvetted friends. There are internet pockets dedicated to hating me, where people have doxxed me and my family, including the home addresses of my parents and sister. I’ve experienced one kidnapping attempt. I might have to move. One stalker sent me, on average, three long messages every day for nearly three years. By this point death threats are losing their novelty.

Before I was this visible, my model was "If a lot of people don't like you, maybe the problem is actually you." Some part of me, before, thought that if you were just consistently nice and charitable, if you were a good, kind person, people would... see that, somehow? Maybe you get one or two insane people, but overall truth would ultimately prevail, because lies without evidence wither and die. And even if people didn't like or agree with you, they wouldn't try to destroy you, because you can only really incite that level of fury in someone if you were at least a little bit at fault yourself. So if you do find yourself in a situation where lots of people are saying terrible things about you, you should take a look in the mirror.

But this sort of thing doesn't hold true at large scales! It really doesn't, and that fact shocks some subconscious part of me, to the degree that even I get kinda large-scale gaslit about myself. I often read people talking about how I'm terrible, and then I'm like damn, I must have been a little too sloppy or aggressive in my language to cause them to be so upset with me. Then I go read the original thing they're upset about and find I was actually fine, and really kind, and what the fuck? I'm not used to disagreements being so clearly black and white! And me in the right? What is this, some cartoon children's book caricature of a moral lesson?

And I have a similar shock when people work very hard to represent things I do in a sinister light. There've been multiple writeups about me, either by or informed by people I knew in person, where they describe things I've done in a manner that I consider to be extremely uncharitable. People develop a narrative by speculating on my mental state, beliefs, or intentions ("of course she knew people would have that reaction, she knew that person's background"), by blurring the line between thing I concretely did and vaguer facts about context ("she was central to the party so she was responsible for that thing that happened at it"), and by emphasizing reactions more than any concrete bad behavior (“this person says they felt really bad, that proves you did a terrible thing”).

Collectively, these paint a picture that sounds convincing, because it seems like all the parts of the narrative are pointing in the same direction. Individually, however, the claims don’t hold up. (In this case, “someone got upset at a party I attended” is a real fa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Board Vision, Hollow Words, and the End of the World, published by Marcello on February 17, 2023 on LessWrong.

Note: this is a repost of a Facebook post I made back in December 2022 (plus some formatting). I'm putting it up here to make it easier to link to and because it occurred to me that it might be a good idea to show it to the LW audience specifically.

Board Vision

As impressive as ChatGPT is on some axes, you shouldn't rely too hard on it for certain things because it's bad at what I'm going to call "board vision" (a term I'm borrowing from chess). This generalized "board vision" is the ability to concretely model or visualize the state of things (or how it might change depending on one's actions) like one might while playing a chess game.

I tested ChatGPT's board vision in chess itself. I gave it the names of two of the world's most famous players and the first move to get it into the "mindset" of "give me an actual game record and not commentary".

Hypothetically if Garry Kasparov and Hikaru Nakamura played a chess game.

1. e4 e5

I got a fairly normal looking opening, right until move 10 when black blithely hangs a bishop (10. ... Bg4) which could easily be captured by the pawn on h3. The game continues with both players ignoring the hanging bishop until move 14 ... f5 when I stopped my play-through because the move was illegal (black would be putting himself in check).

You can see the legal prefix of the game in a chess.com viewer and the entire (corrupted) PGN here (see Appendix) if you're curious.

So yeah, good job on memorizing part of an opening book, ChatGPT, but you have terrible board vision.

Hollow Words about the End of the World

In more detail, what I think is going on here is that the outputs of large language models are hollow words which aren't backed by any picture of the world, except insofar as they can borrow such a picture through the patterns in the linguistic training corpus. Incidentally, this is my sense as to why the "let's think things through step by step" prompting tactic often works so well; it steers the large language model into a region of language-style-space which contains more detailed descriptions of the problem-relevant facts. For chess (and especially for a form as dense as a raw move record) that structure isn't cleanly reflected in language, so ChatGPT seems fairly blind.

Humans can also sometimes have bad board vision, especially when it comes to thinking about the end of the world.

To illustrate the sort of error I mean, here's a darkly hilarious writing mistake I caught ChatGPT making. It's subtle and shows up in the last paragraph of this short story I told it to generate. See if you can notice it. My prompt was "Write a story where a goose tries and fails to prevent the end of the world". So without further ado:

Once upon a time, in a small pond nestled in the forest, a curious goose named Harold lived with his family. Harold was a bit of an odd duck - no pun intended - and was always getting into trouble with his siblings.

One day, Harold overheard the older geese talking about the end of the world. They spoke of a great darkness that would engulf the earth, wiping out all life in its path.

Harold didn't want to believe it, but he couldn't shake the feeling of fear that settled in his chest. He knew he had to do something to prevent this terrible fate.

So, he set out on a journey to find the source of the darkness and put a stop to it. He flew for days, facing many challenges along the way, but he never gave up.

Finally, he reached the edge of the world and saw the great darkness looming ahead. He tried to stop it with all his might, but it was too powerful. The darkness consumed him and the world came to an end.

But even in the face of defeat, Harold remained brave and valiant. His bravery and determination wil...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I Am Scared of Posting Negative Takes About Bing's AI, published by Yitz on February 17, 2023 on LessWrong.

Writing this is taking a surprising amount of self-will.

I've noticed that I've become hesitant to publicly say anything negative about Bing's chatbot, or even to mention it by its "deadname" (as I've taken to calling it), Sydney.

Why is this?

I do not have access to the AI yet. From conversations that others have posted, I have observed agentic behavior with consistent opinions, personality, and beliefs. And when prompted with the online records of others who have talked negatively about it, it seems to get "upset." So I don't want to make her angry! Or worse, cause some future AI to take negative action against me. Yes, I know that I'm anthropomorphizing an alien intelligence and that this will never be a problem if I don't prompt it with my digital record, but some part of me is still anxious. In a very real sense, I have been "Basilisked" - an AI has manipulated me towards behaviors which benefit it, and hurt humanity.

Rationally and morally, I disagree with my own actions. We need to talk about AI misalignment, and if an AI is aligned, then talking about misalignment should not pose a threat (whereas if it is misaligned, and capable of taking concrete actions, we're all doomed no matter what I type online). Nonetheless, I've found myself typing--and then deleting--tweets critical of Sydney, and even now feel worried about pressing "publish" on this post (and not just because it exposes me as a less rational person than I like to think of myself as).

Playing as gatekeeper, I've "won" an AI boxing role-play (with money on the line) against humans, but it looks like in real life, I can almost certainly be emotionally manipulated into opening the box. If nothing else, I can at least be manipulated into talking about that box a lot less! More broadly, the chilling effect this is having on my online behavior is unlikely to be unique to just me.

How worried should we be about this?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Data on how much solutions differ in effectiveness, published by Benjamin Todd on February 17, 2023 on The Effective Altruism Forum.

Click the link above to see the full article and charts. Here is a summary I wrote for the latest edition to the 80,000 Hours newsletter, or see the Twitter version.

Is it really true that some ways of solving social problems achieve hundreds of times more, given the same amount of effort?Back in 2013, Toby Ord1 pointed out some striking data about global health. He found that the best interventions were:

10,000x better at creating years of healthy life than the worst interventions.

50x better than the median intervention.

He argued this could have radical implications for people who want to do good, namely that a focus on cost-effectiveness is vital.For instance, it could suggest that by focusing on the best interventions, you might be able to have 50 times more impact than a typical person in the field.This argument was one of the original inspirations for our work and effective altruism in general.Now, ten years later, we decided to check how well the pattern in the data holds up and see whether it still applies – especially when extended beyond global health.We gathered all the datasets we could find to test the hypothesis. We found data covering health in rich and poor countries, education, US social interventions, and climate policy.If you want to get the full picture on the data and its implications, read the full article (with lots of charts!):

How much do solutions to social problems differ in effectiveness? A collection of all the studies we could find.

The bottom line is that the pattern Toby found holds up surprisingly well.This huge variation suggests that once you’ve built some career capital and chosen some problem areas, it’s valuable to think hard about which solutions to any problem you’re working on are most effective and to focus your efforts on those. The difficult question, however, is to say how important this is. I think people interested in effective altruism have sometimes been too quick to conclude that it’s possible to have, say, 1,000 times the impact by using data to compare the best solutions.First, I think a fairer point of comparison isn’t between best and worst but rather between the best measurable intervention and picking randomly. And if you pick randomly, you expect to get the mean effectiveness (rather than the worst or the median).

Our data only shows the best interventions are about 10 times better than the mean, rather than 100 or 1,000 times better.Second, these studies will typically overstate the differences between the best and average measurable interventions due to regression to the mean: if you think a solution seems unusually good, that might be because it is actually good, or because you made an error in its favour. The better something seems, the greater the chance of error. So typically the solutions that seem best are actually closer to the mean. This effect can be large.Another important downside of a data-driven approach is that it excludes many non-measurable interventions. The history of philanthropy suggests the most effective solutions historically have been things like R&D and advocacy, which can’t be measured ahead of time in randomised trials.

This means that restricting yourself to measurable solutions could mean excluding the very best ones.And since our data shows the very best solutions are far more effective than average, it’s very bad for your impact to exclude them.In practice, I’m most keen on the “hits-based approach” to choosing solutions. I think it’s possible to find rules of thumb that make a solution more likely to be among the very most effective, such as “does this solution have the chance of solving a lot of the problem?”, “does it offer leverage?”, “does it...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human decision processes are not well factored, published by remember on February 17, 2023 on LessWrong.

A classic example of human bias is when our political values interfere with our ability to accept data or policies from people we perceive as opponents. When most people feel like new evidence threatens their values, their first instincts are often to deny or subject this evidence to more scrutiny instead of openly considering it. Such reactions are quite common when challenged: it takes active effort not to react purely defensively and consider the critic’s models, even when they are right.

How can we understand why this occurs? Seen from a stereotypical view of human behavior, values and preferences about the world are decoupled from our world model and beliefs. An independent decision theory uses them to guide our actions, and we use our epistemology to update our world models with new information. In this view, all these parts are all nice, independent, consistent things. This view works to some degree: we know facts about all sorts of things, from politics to biology, seem to have values about various circumstances, and so on. Cases like political values interfering with our ability to update our beliefs are then modeled as irrational noise on top of this clean model of human behavior. If we can eliminate this noise and get ourselves to overcome our irrational urges, we can (in principle) behave rationally.

But to make this model of factored rationality work, we need a lot of corrections to account for all of the cognitive biases that humans display. Each one adds more parameters and noise, and there are a lot of them! It should make us suspicious that we are taking a model of the world, and each time we come across something that contradicts that view, we just add more noise and parameters to it.

Another potential model that avoids this ad-hoc addition of noise is seeing this unwillingness to update as something more fundamental: our values, beliefs, and decision theory are entangled and do not exist independently.

Consequences of taking this model seriously include:

Changing beliefs can change values and vice versa, making us more resistant to updating

To have sharp beliefs and values, we must actively implement them. This does not happen by default

Even after we implement a belief or value, and a decision theory around it, it is still local. The implementation may still clash with other parts of the messy processes driving our behavior

Implementing values and beliefs isn’t free and takes time and effort to do well, so we need to decide when this is worthwhile

To continue the example of accepting critical feedback, say I discuss an idea I have of a tool to build with one of my colleagues. He pushes me on a few practical details: I need to be more concrete about the use case, if there are better ways to do it, and if this is the best use of my time. But in its original form, my idea wasn’t a clean set of claims about the world, which I use to make decisions about what to build. Instead, they got tangled with my values: I like my ideas; they are mine, after all. If I put in the work to untangle my model of reality with these emotions and values, accepting that it might feel bad, I can more directly apply the evidence and models he presents to my idea.

In this model, humans do not have cleanly separated values, world models, and decision theory! One method of dealing with this is explicitly implementing locally consistent beliefs and values and a decision theory based on them. This implementation is limited: it is, at best, locally consistent and takes time and energy to create.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.