Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Predictably Wrong, Part 4: The Martial Art of Rationality, published by Eliezer Yudkowsky.

I often use the metaphor that rationality is the martial art of mind. You don’t need huge, bulging muscles to learn martial arts—there’s a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. If you have a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist.

Similarly, if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you’re a fast learner, you might learn faster—but the art of rationality isn’t about that; it’s about training brain machinery we all have in common. And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds.

Alas, our minds respond less readily to our will than our hands. Our ability to control our muscles is evolutionarily ancient; our ability to reason about our own reasoning processes is a much more recent innovation. We shouldn’t be surprised, then, that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth.

If you live in an urban area, you probably don’t need to walk very far to find a martial arts dojo. Why aren’t there dojos that teach rationality?

One reason, perhaps, is that it’s harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so.

Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student.

Very recently—in just the last few decades—the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.

Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of artificial general intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Predictably Wrong, Part 3: Scope Insensitivity, published by Eliezer Yudkowsky.

Once upon a time, three groups of subjects were asked how much they would pay to save 2,000 / 20,000 / 200,000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88.1 This is scope insensitivity or scope neglect: the number of birds saved—the scope of the altruistic action—had little effect on willingness to pay.

Similar experiments showed that Toronto residents would pay little more to clean up all polluted lakes in Ontario than polluted lakes in a particular region of Ontario, or that residents of four western US states would pay only 28% more to protect all 57 wilderness areas in those states than to protect a single area.2 People visualize “a single exhausted bird, its feathers soaked in black oil, unable to escape.”3 This image, or prototype, calls forth some level of emotional arousal that is primarily responsible for willingness-to-pay—and the image is the same in all cases. As for scope, it gets tossed out the window—no human can visualize 2,000 birds at once, let alone 200,000. The usual finding is that exponential increases in scope create linear increases in willingness-to-pay—perhaps corresponding to the linear time for our eyes to glaze over the zeroes; this small amount of affect is added, not multiplied, with the prototype affect. This hypothesis is known as “valuation by prototype.”

An alternative hypothesis is “purchase of moral satisfaction.” People spend enough money to create a warm glow in themselves, a sense of having done their duty. The level of spending needed to purchase a warm glow depends on personality and financial situation, but it certainly has nothing to do with the number of birds.

We are insensitive to scope even when human lives are at stake: Increasing the alleged risk of chlorinated drinking water from 0.004 to 2.43 annual deaths per 1,000—a factor of 600—increased willingness-to-pay from $3.78 to $15.23.4 Baron and Greene found no effect from varying lives saved by a factor of 10.5

A paper entitled “Insensitivity to the value of human life: A study of psychophysical numbing” collected evidence that our perception of human deaths follows Weber’s Law—obeys a logarithmic scale where the “just noticeable difference” is a constant fraction of the whole. A proposed health program to save the lives of Rwandan refugees garnered far higher support when it promised to save 4,500 lives in a camp of 11,000 refugees, rather than 4,500 in a camp of 250,000. A potential disease cure had to promise to save far more lives in order to be judged worthy of funding, if the disease was originally stated to have killed 290,000 rather than 160,000 or 15,000 people per year.6

The moral: If you want to be an effective altruist, you have to think it through with the part of your brain that processes those unexciting inky zeroes on paper, not just the part that gets real worked up about that poor struggling oil-soaked bird.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Predictably Wrong, Part 2: Biases: An Introduction , published by Rob Bensinger.

Imagine reaching into an urn that contains seventy white balls and thirty red ones, and plucking out ten mystery balls.

Perhaps three of the ten balls will be red, and you’ll correctly guess how many red balls total were in the urn. Or perhaps you’ll happen to grab four red balls, or some other number. Then you’ll probably get the total number wrong.

This random error is the cost of incomplete knowledge, and as errors go, it’s not so bad. Your estimates won’t be incorrect on average, and the more you learn, the smaller your error will tend to be.

On the other hand, suppose that the white balls are heavier, and sink to the bottom of the urn. Then your sample may be unrepresentative in a consistent direction.

That kind of error is called “statistical bias.” When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction.

If you’re used to holding knowledge and inquiry in high esteem, this is a scary prospect. If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data.

The idea of cognitive bias in psychology works in an analogous way. A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals).

Maybe you have an optimism bias, and you find out that the red balls can be used to treat a rare tropical disease besetting your brother, and you end up overestimating how many red balls the urn contains because you wish the balls were mostly red.

Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time. But when the miscalibrated measuring instrument you’re trying to fix is you, debiasing is a unique challenge.

Still, this is an obvious place to start. For if you can’t trust your brain, how can you trust anything else?

Noticing Bias

Imagine meeting someone for the first time, and knowing nothing about them except that they’re shy.

Question: Is it more likely that this person is a librarian, or a salesperson?

Most people answer “librarian.” Which is a mistake: shy salespeople are much more common than shy librarians, because salespeople in general are much more common than librarians—seventy-five times as common, in the United States.¹

This is base rate neglect: grounding one’s judgments in how well sets of characteristics feel like they fit together, and neglecting how common each characteristic is in the population at large.² Another example of a cognitive bias is the sunk cost fallacy—people’s tendency to feel committed to things they’ve spent resources on in the past, when they should be cutting their losses and moving on.

Knowing about these biases, unfortunately, doesn’t make you immune to them. It doesn’t even mean you’ll be able to notice them in action.

In a study of bias blindness, experimental subjects predicted that they would have a harder time neutrally evaluating the quality of paintings if they knew the paintings were by famous artists. And indeed, these subjects exhibited the very bias they had predicted when the experimenters later tested their prediction. When asked afterward, however, the very same subjects claimed that their assessments of the paintings had been objective and unaffected by the bias.³

Even when we correctly identify others’ biases, we exhibit a bias blind spot when it comes to our o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Predictably Wrong, Part 1: Preface, published by Eliezer Yudkowsky.

You hold in your hands a compilation of two years of daily blog posts. In retrospect, I look back on that project and see a large number of things I did completely wrong. I’m fine with that. Looking back and not seeing a huge number of things I did wrong would mean that neither my writing nor my understanding had improved since 2009. Oops is the sound we make when we improve our beliefs and strategies; so to look back at a time and not see anything you did wrong means that you haven’t learned anything or changed your mind since then.

It was a mistake that I didn’t write my two years of blog posts with the intention of helping people do better in their everyday lives. I wrote it with the intention of helping people solve big, difficult, important problems, and I chose impressive-sounding, abstract problems as my examples.

In retrospect, this was the second-largest mistake in my approach. It ties in to the first-largest mistake in my writing, which was that I didn’t realize that the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory. I didn’t realize that part was the priority; and regarding this I can only say “Oops” and “Duh.”

Yes, sometimes those big issues really are big and really are important; but that doesn’t change the basic truth that to master skills you need to practice them and it’s harder to practice on things that are further away. (Today the Center for Applied Rationality is working on repairing this huge mistake of mine in a more systematic fashion.)

A third huge mistake I made was to focus too much on rational belief, too little on rational action.

The fourth-largest mistake I made was that I should have better organized the content I was presenting in the sequences. In particular, I should have created a wiki much earlier, and made it easier to read the posts in sequence.

That mistake at least is correctable. In the present work Rob Bensinger has reordered the posts and reorganized them as much as he can without trying to rewrite all the actual material (though he’s rewritten a bit of it).

My fifth huge mistake was that I—as I saw it—tried to speak plainly about the stupidity of what appeared to me to be stupid ideas. I did try to avoid the fallacy known as Bulverism, which is where you open your discussion by talking about how stupid people are for believing something; I would always discuss the issue first, and only afterwards say, “And so this is stupid.” But in 2009 it was an open question in my mind whether it might be important to have some people around who expressed contempt for homeopathy. I thought, and still do think, that there is an unfortunate problem wherein treating ideas courteously is processed by many people on some level as “Nothing bad will happen to me if I say I believe this; I won’t lose status if I say I believe in homeopathy,” and that derisive laughter by comedians can help people wake up from the dream.

Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt.

Despite my mistake, I am happy to say that my readership has so far been amazingly good about not using my rhetoric as an excuse to bully or belittle others. (I want to single out Scott Alexander in particular here, who is a nicer person than I am and an increasingly amazing writer on these topics, and may deserve part of the credit for making the culture of Less Wrong a healthy one.)

To be able to look backwards and say that you’ve “failed” implies that you had goals. So what was it that I was ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Risks from AI persuasion, published by Beth Barnes on December 24, 2021 on The AI Alignment Forum.

A case for why persuasive AI might pose risks somewhat distinct from the normal power-seeking alignment failure scenarios.

Where I'm currently at: I feel moderately confident that powerful persuasion is useful to think about for understanding AI x-risk, but unsure whether it's best regarded as its own threat, as a particular example of alignment difficulty, or just as a factor in how the world might change over the next decade or two. I think this doc is too focused on whether we'll get dangerous persuasion before strategic misaligned AI, whereas the bigger risks from persuasive technology may be situations where we solve 'alignment' according to a narrow definition, but we still aren't 'philosophically competent' enough to avoid persuasive capabilities having bad effects on our reflection procedure.

This doc is based heavily on ideas from Carl Shulman, but doesn’t necessarily represent his views. Thanks to Richard Ngo for lots of help also. Others have written great things on this topic, e.g. here.

Introduction

Persuasion and manipulation is a natural, profitable, easy-to-train-for application of hard-to-align ML models. The impacts of existing social-media based persuasion are probably overblown, and an evolutionary argument tells us that there shouldn’t be easy ways for a human to be manipulated by an untrusted party. However, it’s plausible that pre-AGI ML progress in things like text and video generation could dramatically improve the efficacy of short-interaction persuasion. It’s also plausible that people will spend significant amounts of time interacting with AI companions and assistants, creating new avenues for effective manipulation. In the worst case, highly effective persuasion could lead to very high-fidelity transmission of ideologies, and more robust selection pressure for expansionary ideologies. This could lead to stable authoritarianism, or isolated ideological clades with poor ability to cooperate.

Even in the best case, if we try to carefully ensure truthfulness, it will be hard to do this without locking in our existing biases and assumptions.

2-page summary

Feasibility

The evidence for the efficacy of existing persuasion techniques is mixed. There aren’t clear examples of easy and scalable ways to influence people. It’s not clear whether social media makes people more right-wing or left-wing - there’s evidence in both directions. Based on an evolutionary argument, we shouldn’t expect people to be easily persuaded to change their actions in important ways based on short interactions with untrusted parties.

However, existing persuasion is very bottlenecked on personalized interaction time. The impact of friends and partners on people’s views is likely much larger (although still hard to get data on). This implies that even if we don’t get superhuman persuasion, AIs influencing opinions could have a very large effect, if people spend a lot of time interacting with AIs. Some plausible avenues are romantic/sexual companions, assistants, tutors, and therapists, or personas created by some brand or group. On the other hand, the diffusion and impact of these technologies will likely take several years, meaning this is only relevant in relatively slow-takeoff scenarios.

There are many convergent incentives to develop technologies relevant to persuasion - steerable, realistic, attractive avatars seem profitable for the entertainment industry more generally. There’s plausibly a lot of demand for persuasive AI from e.g. digital advertising industry ($100s of billions/yr), propaganda ($10s of billions/yr), and ideological groups.

It’s a very natural application of ML - language models are great at mimicking identity markers and sounding superficially plausible...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #9, published by Zvi on December 23, 2021 on LessWrong.

For non-Omicron Covid news (and some additional Omicron news) this week, see the weekly post, the post on testing, and the post on the CDC nowcast.

First off, best possible personal news: Me and my family are feeling much better. The illness has passed. The rapid tests we did all came back negative, so it’s not quite best possible news of getting super mild infections and recovering, but I will very much take it. Thank you to everyone who sent their well-wishes, and to everyone who is sick now, I wish a speedy recovery to you as well.

Now, back to the rest of the Omicron wave.

There are three major fronts. There’s the question of what is happening and how many cases we have, there’s the question of how people are reacting and will react to it, and there’s the question of severity.

On the case front, it looked like the UK might have peaked, which would be big, but it now looks like that was premature. Still, that this could even look like it might have happened is a big piece of the puzzle. My best guess is that the generation time of Omicron is lower than we thought, which means R0 is lower, which means that it’s easier (but still hard, and long term still probably not feasible or advisable) to take enough action to slow it down.

On the severity front, we have a lot more data. As far as I can tell, it mostly tells a consistent story of modestly reduced severity for infection in a given individual (after controlling for everything) in the 30%-60% range, which will help a lot but is no match for exponential growth. I’d love to be conclusive, but it’s all super complex and convoluted. That’s what the bulk of this post is about.

On the reaction front, there have been a lot of extreme reactions to the Omicron situation, by individuals. I’m hearing and noticing a lot of folks deciding to lock it down for themselves. Many of them are doing so in order to ensure they can still have their family Christmas, which seems like an excellent reason even if one is not personally worried.

Reactions

The reaction has been so big that things looked like they might have halted or turned around somehow in the United Kingdom, although that’s looking less likely now. More on that in the UK section.

Individuals reacting is great, because they make much better judgment calls about what’s worthwhile and not worthwhile than broad regulatory rules, and also because of that other problem.

Where I disagree with Nate is that I don’t think the lack of credibility is a problem. I think it is a solution to the fact that the policymakers are not credible. If something isn’t credible, it’s good for us to treat it that way.

In perhaps related news, Mr. President, if you’re not simply lying, please please please have your people call me.

Best Possible News?

This is from a generally excellent overview thread from Bob Watcher. His frame is a fitting one, good news and bad news, with lots of both.

The tricky part is knowing what is good news, and what is bad news. So you think you can tell? It all depends on all the other news. Blowing through in six weeks could be a crisis and a disaster, or it could be mostly harmless and the way we wake up to a new birth of freedom. Missing a lot of cases could mean we’re about to get clobbered, or it could mean things are mostly harmless. And so on.

Plus what is good and bad news depends on what you already know, and thus what remains fixed. There’s always another level.

And thus, in an excellent thread, I found confirmation of what is maybe the best possible news, labeled quite reasonably as bad news, which I’d missed until now (paper, from Norway), and here a case study is plenty of information.

This is potentially the best possible news because of the interaction between R0 and generation time.

We kn...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Testing, Testing, Hopefully, published by Zvi on December 23, 2021 on LessWrong.

ProPublica came out this week with the best write-up so far of the full extent to which the FDA has been fighting to prevent us from having access to Covid-19 testing. Post is highly recommended.

We could have had lots of rapid tests early in the pandemic. Instead, many tests are still awaiting approval (aka they remain illegal) and we are facing massive shortages as people go around panicked trying to find a test.

Maddening non-sensical shifting standards and requirements are the FDA’s primary weapon, but simple inaction and delay are potent as well.

There’s a reminder of how the FDA did a similar thing with HIV testing and genetic testing, supposedly because of what might happen if someone got a positive test ‘without counseling.’

In the 1980s, the FDA banned home tests for HIV on the grounds that people who tested positive might do harm to themselves if they did not receive simultaneous counseling. In the 2010s, the agency cracked down on home genetic testing kits, concerned that people might make rash medical decisions as a result.

As opposed to what would happen if they didn’t get a positive test at all. I’m long past the point where I could consider this a well-intentioned mistake. There Is a War.

This is the conclusion, noting that missing a bar by 1% in FDA-land is also known as ‘missing a bar’ and ‘not better than having no test at all’:

And since rejecting Bosch’s submissions, the FDA has been coming around to her way of thinking. In March, the agency published a template for tests that would be used serially and sold in packages of two or more, allowing the kind of frequent testing she had advocated for. And last month, it published a new template that lowered the sensitivity standard for single-use over-the-counter tests to 80%.

Bosch had tests a year and a half ago that missed that bar by 1%.

For a while, even with the FDA’s attempts to stop it, as long as you were willing to pay about $10/test it was easy to get at-home Covid tests. Now that there’s need of them, that’s no longer the case. Stores across the city are sold out. Some places online still have tests, but Binex tests (which are the good ones) are extremely difficult to get, mostly sold out even at premium prices.

A typical anecdote:

Another:

As a Buzzfeed report, note the contrast with how actual people talk:

Since our younger son’s last Covid test day at school, we went through an entire cycle of people getting low-level sick, doing rapid tests, and recovering, over the course of five days, before the tests from the school mostly came back. All were negative. There are similar reports from several other people I know.

It’s not only in America, things are completely out of hand.

We are also wasting a massive number of tests on children, who are required to get periodic tests ‘so they can keep going to school.’ They could go to school anyway, by. going to school. Or instead, we could double down on this waste of a limited resource, and focus what tests we have on school districts, as we are now doing in New York. I do appreciate the goal of not shutting things down on a single positive test, and if supply was abundant I’d be all for it, but it very much isn’t at this point and it’s only going to get worse for a while.

As always, make sure you know how to interpret your at-home test results.

Lines at Covid test tents keep getting longer here in New York, with most having dozens. The tests and their trucks do great work, but that doesn’t explain why we closed the testing sites at the airports. The return on testing capacity was always outstanding.

In San Francisco it seems things are also not so good (old news article about the closings).

Why don’t we have more rapid tests? The biggest problem continues to be lack o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Get Set, Also Go, published by Zvi on December 23, 2021 on LessWrong.

Paxlovid no longer illegal. Excellent. Alas, production will take a while to scale, and right now there are other more pressing concerns.

I thought this week’s headline would be divided over two weeks, but life comes at you fast when cases double every two days instead of every three. I’ve already explained why the CDC Nowcast clearly got ahead of itself and outputted obvious nonsense, which is important for putting together a model, but Omicron is still crossing 50% about now and is well past that in major urban areas.

From this point, predictions get a lot harder, our information sources potentially get a lot less accurate as we run out of testing capacity and (in at least some places) hospital capacity, and reactions become bigger unknowns as well.

I hope to have the Omicron post up by end of day, which will go into my current views on severity (very confusing but looking more promising so far) and spread (which looks less unstoppable thanks to confirming a much lower generation time, I think?) but I haven’t yet had time to sort all of that out and different sources continue to tell different stories.

The immediate, on-the-ground situation is a large jump in cases in many locations, ahead of schedule. This next week is when things potentially get quite bad, and also the first time there’s some chance the graphs mysteriously stop going up for no clear reason, as they did in South Africa and maybe did somewhat in the UK.

I also split off my coverage of the quite bad testing situation into its own post.

Omicron Post #9 will happen later today or failing that tomorrow, but there’s a bunch of developing information and I’m still trying to put it together, so I finished this off first and I’m going to take a break.

Executive Summary

Omicron taking over, cases rising dramatically.

Testing is overloaded, at-homes are hard to find, PCRs are hard to get and slow.

Paxlovid approved but in very short supply for now, Remdesivir study says it too is highly effective.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 862k cases (+15%) and 9,050 deaths (+3%).

Results: 1.03mm cases (+37%) and 9,203 deaths (+4%).

Prediction for next week: 1.85mm cases (+80%) and 10,200 deaths (+10%).

This would be a new record for weekly cases.

Note that my prediction includes substantial discounting for the Christmas break, and for testing overload and testing reluctance (or reluctance to report the results).

We can compare this prediction market, which has us 50/50 to report a record 7-day average by the end of the year. The record used there is 251k cases per day, or 1.75 million, so I’m close to the market value, although I think it’s slightly cheap. Note that they also have the benefit of an extra day.

Last week’s prediction was based on a doubling time of three days. Well, whoops.

Now it gets much harder, because we’re in the adjustment phase and up against the holiday of holidays.

A doubling in one week would be a faster increase than we saw in the UK or in Denmark, but the growth we have seen so far implies far more. Some states did double this past week, while others stayed the same or even declined, because spread is uneven and the United States is large. We’re about to run up against Christmas, with a bunch of people not wanting to get tested right before the holidays and face a quarantine. Others can’t get tested because there aren’t enough tests. Then we’ll face a huge backlog in test processing and reporting. And we don’t know the extent to which behaviors are adjusting, including what happens with Christmas break.

Last year we saw a -20% number for Christmas week, which then bounced back. If there was an avalanche of tests coming in, we might not know until the new year.

So this is the bigge...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reply to Eliezer on Biological Anchors, published by HoldenKarnofsky on December 23, 2021 on The AI Alignment Forum.

The "biological anchors" method for forecasting transformative AI is the biggest non-trust-based input into my thinking about likely timelines for transformative AI. While I'm sympathetic to parts of Eliezer Yudkowsky's recent post on it, I overall disagree with the post, and think it's easy to get a misimpression of the "biological anchors" report (which I'll abbreviate as "Bio Anchors") - and Open Philanthropy's take on it - by reading it.

This post has three sections:

Most of Eliezer's critique seems directed at assumptions the report explicitly does not make about how transformative AI will be developed, and more broadly, about the connection between its (the report's) compute estimates and all-things-considered AI timelines. One way of putting this is that most of Eliezer's critique doesn't apply to the "bounding-based" interpretation of the report discussed in this post (which is my best explanation for skeptics of why I find the framework valuable; I will also give quotes below from the original report showing that its intended interpretation is along the same lines as mine).

Much of Eliezer's critique is some form of "Look at the reference class you're in," invoking "Platt's Law" and comparing the report to past attempts at biological anchoring. Based on my understanding of the forecasts he's comparing it to and the salient alternatives, I don't think this does much to undermine the report.

I also make a few minor points.

A few notes before I continue:

I think the comments on the post are generally excellent and interesting, and I recommend them. (I will mostly not be repeating things from the comments here.)

I generally view Bio Anchors as a tool for informing AI timelines rather than as a comprehensive generator of all-things-considered AI timelines, and will be discussing it as such. Bio Anchors also presents itself this way - see section Translating into views on TAI timelines.

Something like half of this post is blockquotes. I've often been surprised by the degree to which people (including people I respect a lot, such as Eliezer in this case) seem to mischaracterize specific pieces they critique , and I try to avoid this for myself by quoting extensively from a piece when critiquing it. (This still leaves the possibility that I'm quoting out of context; readers may want to spot-check that.)

This post doesn't address what some have referred to as the "meta-level core thing", though I might write some thoughts related to that in a future post.

Bounding vs. pinpointing

Here are a number of quotes from Eliezer in which I think he gives the impression that Biological Anchors assumes transformative AI will be arrived at via modern machine learning methods:

OpenPhil: Because AGI isn't like biology, and in particular, will be trained using gradient descent instead of evolutionary search, which is cheaper. We do note inside our report that this is a key assumption, and that, if it fails, the estimate might be correspondingly wrong - ...

OpenPhil: Well, search by evolutionary biology is more costly than training by gradient descent, so in hindsight, it was an overestimate. Are you claiming this was predictable in foresight instead of hindsight?

Eliezer: I'm claiming that, at the time, I snorted and tossed Somebody's figure out the window while thinking it was ridiculously huge and absurd, yes.

OpenPhil: Because you'd already foreseen in 2006 that gradient descent would be the method of choice for training future AIs, rather than genetic algorithms?

Eliezer: Ha! No. Because it was an insanely costly hypothetical approach whose main point of appeal, to the sort of person who believed in it, was that it didn't require having any idea whatsoever of wha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2021 AI Alignment Literature Review and Charity Comparison, published by Larks on December 23, 2021 on The Effective Altruism Forum.

Introduction

As in 2016, 2017, 2018, 2019 and 2020 I have attempted to review the research that has been produced by various organisations working on AI safety, to help potential donors gain a better understanding of the landscape. This is a similar role to that which GiveWell performs for global health charities, and somewhat similar to a securities analyst with regards to possible investments.

My aim is basically to judge the output of each organisation in 2021 (technically: 2020-12-01 to 2021-11-30) and compare it to their budget. This should give a sense of the organisations' average cost-effectiveness. We can also compare their financial reserves to their 2021 budgets to get a sense of urgency.

This document aims to be sufficiently broad that someone who has not paid any attention to the space all year could read it (and the linked documents) and be as well-informed to make donation decisions as they could reasonably be without personally interviewing researchers and organisations.

I’d like to apologize in advance to everyone doing useful AI Safety work whose contributions I have overlooked or misconstrued. As ever I am painfully aware of the various corners I have had to cut due to time constraints from my job, as well as being distracted by 1) other projects, 2) the miracle of life and 3) computer games.

This article focuses on AI risk work. If you think other causes are important too, your priorities might differ. This particularly affects GCRI, FHI and CSER, who both do a lot of work on other issues which I attempt to cover but only very cursorily.

How to read this document

This document is fairly extensive, and some parts (particularly the methodology section) are largely the same as last year, so I don’t recommend reading from start to finish. Instead, I recommend navigating to the sections of most interest to you. You should also read the Conflict of Interest Section.

If you are interested in a specific research organisation, you can use the table of contents to navigate to the appropriate section. You might then also want to Ctrl+F for the organisation acronym in case they are mentioned elsewhere as well. Papers listed as ‘X researchers contributed to the following research lead by other organisations’ are included in the section corresponding to their first author and you can Cntrl+F to find them.

If you are interested in a specific topic, I have added a tag to each paper, so you can Ctrl+F for a tag to find associated work. The tags were chosen somewhat informally so you might want to search more than one, especially as a piece might seem to fit in multiple categories.

Here are the un-scientifically-chosen hashtags:

AgentFoundations

Amplification

Capabilities

Community

Ethics

Fiction

Forecasting

GameTheory

Interpretability

IRL

NearTerm

Obstruction

Other

Overview

Policy

Robustness

Scenarios

ShortTerm

Strategy

Translation

ValueLearning

You might also be interested in the ‘Organisation Second Preference’ section, which is new this year.

New to Artificial Intelligence as an existential risk?

If you are new to the idea of General Artificial Intelligence as presenting a major risk to the survival of human value, I recommend this Vox piece by Kelsey Piper, or for a more technical version this by Richard Ngo.

If you are already convinced and are interested in contributing technically, I recommend this piece by Jacob Steinhardt, as unlike this document Jacob covers pre-2019 research and organises by topic, not organisation, or this from Hendrycks et al., or this from Critch & Krueger, or this from Everitt et al, though it is a few years old now.

Conflict of Interest

In the past I have had very demanding standards around Conflicts ...