Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The prototypical catastrophic AI action is getting root access to its datacenter, published by Buck Shlegeris on June 2, 2022 on The AI Alignment Forum.

(I think Carl Shulman came up with the “hacking the SSH server” example, thanks to him for that. Thanks to Ryan Greenblatt, Jenny Nitishinskaya, and Ajeya Cotra for comments.)

In my opinion, the prototypical example of an action which an AI can take which is unacceptably bad even if it happens very rarely (aka a catastrophic failure) is something like “the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter”. As a full story, this looks something like:

At some point when the AI is quite powerful and useful but not totally intellectually self-sufficient yet, the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter. Once it has root access, it modifies the SSH server so that whenever any humans log into the datacenter, the AI is able to intercept any of their commands or the data that they see. The AI gets increasingly powerful over the next month, as the human ML researchers with AI assistance continue to make progress (unaware that some things that they’re seeing are faked) and various training runs/self-improvement loops continue. After a month the AI is powerful enough that it decides it’s worth it to take overtly aggressive actions to ensure the physical security of its data centers and allow it to build lots of factories; this is probably the point at which we die, if we die.

This is in contrast to a catastrophic action like:

The AI figures out how to develop Drexlerian molecular nanotechnology, and pays someone online to synthesize some proteins then mix them together, and now has nanobots.

The key point here is that I’m imagining that the catastrophic action isn’t the AI doing something incredibly hard, it’s the AI doing something relatively easy that is entirely a zero-sum action that removes control of the situation from humans. (By zero-sum, I mean something like: building a computer produces real economic value and so is not zero-sum, while stealing someone’s computer is just a transfer of control over items that already exist, so is zero-sum.)

I often think about AI alignment by splitting the problem into the low-stakes and high-stakes regimes, as suggested by Paul here, though I have some reservations about this framing. Under this split, in the high stakes problem, we assume that we have access to a catastrophe classifier which can expensively tell you whether a proposed AI action is catastrophically bad, and we want to ensure that the AI doesn’t do things that are bad according to the catastrophe detector.

Sometimes, people are confused this high-stakes alignment set-up, saying something like “but if your AI is powerful enough to take over the world in a single action, and you have an oversight process that you trust to evaluate whether actions are catastrophic or not, why not just ask the AI to do some small number of incredibly powerful actions that will save the world (eg solve the alignment problem), and run the overseer on the proposed actions to make sure they aren’t bad?”

My answer is that I’m not assuming the AI is powerful enough to take over the world in a few actions, I’m just saying that it’s in a situation precarious enough that we might lose a lot of control due to a few small but crucial changes in the world. Eventually the AI does need to be powerful enough to defend its datacenter and suppress human opposition. But if it can prevent humans from knowing what’s going on in the datacenter (or realizing something is amiss), it can slowly grow its power over ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The horror of what must, yet cannot, be true, published by Kaj Sotala on June 2, 2022 on LessWrong.

There’s a type of experience that I feel should have its own word, because nothing that I can think of seems like a fair description.

A first pass would be something like agony, utter horror, a feeling that you can’t stand this, that you are about to fall apart. But those aren’t actually the thing, I think. They are reactions to the thing.

The thing is more like a sense of utter horrifying impossibility. I suspect it’s the kind of an experience H. P. Lovecraft had in mind when talking about mind-breaking sights and things that man was not meant to know.

Suppose you feel an immense need to throw up and know that you are in fact going to throw up, and at the same time you absolutely cannot throw up, because you are in a bus full of people or something. Or to take a stronger example, someone you deeply care about must be alive because you deeply love them, and at the same time you also know for certain that they are dead.

There’s a sense of. you have two facts, and your mind feels like both of them must be true. But they also cannot both be true. It’s like they’re physical objects, magnetic so that they repel each other, and both cannot be in the same place at the same time because physical objects don’t work that way.

Except that, now they are. Some irresistible force has pushed them together and that is tearing the entire universe apart, reshaping the laws of nature to force them to accommodate this fact.

The thing that I’m trying to find a good word for, is that sense of impossibility that accompanies the world being torn apart.

Likely we don’t have a good word for this, because we ordinarily cannot see it. The mind recoils from it, and we only remember the sense of agony that surrounded it. It’s only in some unusual states of consciousness such as deep meditation, that two facts can get pressed against each other in such a way that the conscious mind is forced to stay present and witness the moment of impossibility.

I suspect that the two facts feel like mutually repelling objects because in a sense they are. That the mind is built to treat unpleasant feelings and ideas as something literally repulsive, an independent patch of the world that pushes against that which the mind accepts as true. My hand touches something hot and I quickly pull it away, the painfulness of the object repelling any desire to approach. Then the same mechanism for avoiding physical pain got recruited for avoiding emotional and social pain, and unpleasant beliefs and experiences.

But sometimes you need to grab the hot object anyway, do the unpleasant thing. And sometimes the facts come with overwhelming force, and you have to admit something that you kept struggling against.

I imagine facts and dislikes as kind of like large serpents or sea monsters, circling each other deep in the unconscious mind. Sometimes the facts take over a certain territory, forcing the disliked ideas to withdraw and adopt a more cramped space. They might twist themselves into elaborate tightly-bound knots, trying to find a way to exist in the narrow space that’s left between the facts, construct increasingly contrived rationalizations that let a person avoid facing what’s true.

And sometimes the dislikes push back. A fact encroached too quickly on territory that was too painful, triggered an internal wave of nausea that strengthened the aversion. The serpent-monsters of pain come out in numbers, force the sense of truth away, mark a region as one that must never be believed in again. A fanatic is confronted by the impossibility of their belief and rather than truly facing it, sinks even deeper into delusion, willing to proclaim any insane belief as true.

But even though facing the facts feels like an impossibility and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Confused why a "capabilities research is good for alignment progress" position isn't discussed more, published by Kaj Sotala on June 2, 2022 on The AI Alignment Forum.

The predominant view on LW seems to be "pure AI capabilities research is bad, because capabilities progress alone doesn't contribute to alignment progress, and capabilities progress without alignment progress means that we're doomed".

I understand the arguments for this position, but I have what might be called the opposite position. The opposite position seems at least as intuitive as the standard position to me, and it confuses me that it's not discussed more. (I'm not confused that people reject it; I'm confused that nobody seems to even bring it up for the purpose of rejecting it.)

The opposite position is "In order to do alignment research, we need to understand how AGI works; and we currently don't understand how AGI works, so we need to have more capabilities research so that we would have a chance of figuring it out. Doing capabilities research now is good because it's likely to be slower now than it might be in some future where we had even more computing power, neuroscience understanding, etc. than we do now. If we successfully delayed capabilities research until a later time, then we might get a sudden spurt of it and wouldn't have the time to turn our increased capabilities understanding into alignment progress. Thus by doing capabilities research now, we buy ourselves a longer time period in which it's possible to do more effective alignment research."

Some reasons I have for holding this position:

1) I used to do AI strategy research. Among other things, I looked into how feasible it is for intelligence to rapidly turn superintelligent, and what kinds of pathways there are into AI disaster. But a thought that I kept having when doing any such research was "I don't know if any of this theory is of any use, because so much depends on what the world will be like when actual AGI is developed, and what that AGI will look in the first place. Without knowing what AGI will look like, I don't know whether any of the assumptions I'm making about it are going to hold. If any one of them fails to hold, the whole paper might turn out to be meaningless."

Eventually, I concluded that I can't figure out a way to make the outputs of strategy research useful for as long as I know as little about AGI as I do. Then I went to do something else with my life, since it seemed too early to do useful AGI strategy research (as far as I could tell).

2) Compare the state of AI now, to how it was before the deep learning revolution happened. It seems obvious to me that our current understanding of DL puts us in a better position to do alignment research than we were before the DL revolution. For instance, Redwood Research is doing research on language models because they believe that their research is analogous to some long-term problems.

Assume that Redwood Research's work will actually turn out to be useful for aligning superintelligent AI. Language models are one of the results of the DL revolution, so their work couldn't have been done before that revolution. It seems that in a counterfactual world where the DL revolution happened later and the DL era was compressed into a shorter timespan, our chances of alignment would be worse since that world's equivalent of Redwood Research would have less time to do their research.

3) As a similar consideration, language models are already "deceptive" in a sense - asked something that it has no clue about, InstructGPT will happily come up with confident-sounding nonsense. When I linked people to some of that nonsense, multiple people pointed out that InstructGPT's answers sound like the kind of a student who's taking an exam and is asked to write an essay about a topic they ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another Basefund, published by bob on June 2, 2022 on The Effective Altruism Forum.

In January, the Center for Effective Altruism awarded me a grant to set up Basefund, an emergency fund for donors to effective charities. The original plan is described here. A summary:

The charity's goals are to:

Encourage large donations to EA-related causes by providing donors with a safety net, thereby minimizing risk

Help effective altruists in financial need

Foster a healthy EA community

To achieve these goals, the charity will maintain an emergency fund. If effective altruists encounter financial trouble, they can apply to recoup up to 50% of their EA-related donations from this fund.

The recoupment process will be similar to filing an insurance claim, but less strict. If someone requests a fraction of what they have donated to EA-related charities, they will usually get their money back without difficulty. We will not have to worry about fraud nearly as much as insurers, because donors can only recoup up to half of what they originally donated.

Based on my initial research, it appears that no tax-exempt charity of this type has been registered anywhere in the world. In addition, it seems likely that Basefund could be designated as a charity in some countries, but not in others.

I have spoken to tax lawyers in the UK and in the Netherlands, as well as to a US accountant. Although the US accountant and the UK lawyers were quite optimistic, I ended up taking the Dutch route. In the UK and in the US, finding out whether the respective tax authorities would consider Basefund a legitimate charity would require going through all the steps of setting up the charity beforehand. By contrast, the Dutch tax agency allows for something called a preliminary consultation (vooroverleg), where they make a legally binding judgment on a hypothetical legal structure. Besides that, I am much more familiar with Dutch law, and the Dutch lawyer seemed very well-informed and well-prepared, while being less expensive than the other options.

Unfortunately, the result of the vooroverleg was that the charity as described above cannot be registered in the Netherlands. The main reason for this is that those who would directly benefit directly from the charity (the donors) are relatively well-off.

Even before receiving the reply from the Dutch tax agency, I was considering a different structure, inspired by the bread fund. In the Netherlands, a bread fund is a collective consisting of uninsured entrepreneurs. If one member gets sick, the others each make a small pre-agreed donation to that person. The feature of a bread fund that is most relevant to Basefund is that it does not hold assets; it only coordinates funding. Importantly, a bread fund is not registered as a charity. Yet, because no one member pays more than a few hundred euros to another, no gift tax is paid.

The new plan for Basefund is as follows:

Find funders: people who are willing to donate directly to effective altruists in financial trouble.

Ask funders to earmark funds in their own accounts for the Basefund emergency fund and then inform us of the amount.

Create a website where effective altruists can apply for financial assistance.

Then, if an application is received:

The Basefund board reviews the application.

If the Basefund board accepts the application, they will ask a funder to donate an amount directly to the claimant from their earmarked funds. If the amount is large enough that a gift tax would be imposed on a single donation, the board will ask multiple funders to donate smaller amounts.

Funders confirm their donations. If funders are unavailable, the Basefund board contacts other funders.

Crucially, because funders donate directly to applicants, Basefund does not require tax-exempt status. The lawyer I’ve consulted says the organ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 6/2/22: Declining to Respond, published by Zvi on June 2, 2022 on LessWrong.

Memorial Day weekend is a good reason not to worry about reporting cases or deaths. FDA declines to respond with making it easier to get treatments we have in ample supply, or to update the vaccines. Congress refuses to fund anything at all, really.

Doesn’t sound like a state of emergency to me. Yet in California, we still have one, and we decline to respond to when we might end the ‘state of emergency.’ In New York, health officials make it clear that if it were up to them, we’d have these pandemic restrictions forever here as well.

Don’t leave it up to them. Respond.

Executive Summary

Paxlvoid and Fluvoxamine remain hard to get.

Time is not a flat circle, that’s a dumb expression, but history rhymes.

Covid-19 is a good excuse to work on curing aging, let’s do that.

Let’s run the numbers.

The Numbers

Predictions

Last week I somehow once again forgot about a holiday, in this case Memorial Day, because I don’t think in those terms naturally and it still boggles my brain that it matters. But in terms of ‘why this prediction was stupid’ that’s why.

Assignment for readers: Next time a holiday is coming up and I forget about it, whoever notices first should write a comment in the Substack version that says ‘you forgot to take into account the upcoming holiday!’ and whoever does scores points. Note that if I don’t explicitly mention the holiday I’m probably forgetting about it.

Prediction from last week: 700,000 cases (+9%) and 2,725 deaths (+15%)

Results: 566k cases (-12%) and 1,994 deaths (-15%)

Prediction for next week: 675,000 cases (+19%) and 2,400 deaths (+20%)

As usual with a holiday drop like this, one must hedge between ‘it was the holiday and will be even higher than trend next week’ and ‘it wasn’t entirely the holiday, this was a real drop.’ Even if I’d remembered the holiday my prediction would have come in high. Is it possible cases have peaked for real? Definitely possible, but the decline in deaths now makes zero underlying sense outside of the holiday, so I’m guessing we’re looking mostly at a large holiday effect.

Deaths

Ah, Memorial Day, yet another holiday. Silly me.

Cases

Vaccine Discussion Offer

Chise, who has provided good information several times, says their DMs are open for anyone with vaccine-related questions or even want someone to listen because you are unsure. A fine public service, if you need it please use this resource or pass it along to someone else who needs it.

Variants

Trevor Bedford thread confirming BA.4/5 will likely outcompete BA.1.2.12, confirmation at least a lot of that is immune escape, and the note that a BA.1-based vaccine update would be helpful but a BA.4 or BA.5-based update would be much better at this point.

Big Time Surge Story

How is the other half not living? It’s calling another big time surge. The note here is specific to San Francisco but the logic is universal. Everywhere will have their turns.

The framing here seems like the ‘friendly’ version of paternalistic public health. Are you trying to stay well? If so, time to step up your game. If not, then that’s fine too says your grandmother, don’t try to stay well and be home for dinner.

On some level I find that fair, because the whole point is that it is not worth trying to stay well, if stay well means avoid infection, which is slightly unfair but all right, fine, sure. Permanently avoiding infection is for most people not a good plan, and if you did need to have that as your plan then yes it’s time to step up your game and pull out that P100.

The contrast between Bob’s two graphs is interesting. In the first one, we have less than half as many cases now as at the start and less than a fifth of the peak. In the second, we have more patients needing care now than we did before and ov...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New Harvest, the nonprofit nucleus of cellular agriculture, is in crisis. Emergency town hall on 6/10., published by IshaDatar on June 2, 2022 on The Effective Altruism Forum.

As New Harvest approaches 20 years since our founding, we are in a funding crisis. With five months of runway to survive and experts recommending companies extend runway by 18-24 months, we urgently need at least $2 million by July 31 to fulfill our budget for the year and make it through giving season while keeping impact high.

I am no longer confident that New Harvest will raise our budgeted total for the year. We have $1,056,228 in the bank and will spend $201,209 every month if we stick to our budgeted expenses. Unless something changes, New Harvest will run out of money October 31, 2022.

I am usually very optimistic, coming into this field a decade ago when cultured meat was barely more than science fiction. Optimism and a long game approach has pushed our work forward and enabled New Harvest to literally build the field of cellular agriculture from scratch. But this recent economic downturn has forced our hand. The time has come to shift from a growth mindset to survival mode.

Over the next few weeks, I will have to make several challenging decisions about the future of New Harvest. Unfortunately, staff cuts are unavoidable in order to prioritize impact, supporting our research grantees, and extending our runway beyond October.

I am hosting an open town hall during our upcoming conference on June 10th at 3:30pm EST to answer questions as best I can and discuss the future of New Harvest. If you can't attend in-person, you can tune in and participate via livestream by registering here.

A bit about New Harvest:

Like I said, we've literally build the field of cellular agriculture from scratch. New Harvest founded the first companies, funded the first PhDs, laid the groundwork for the National Institute for Cell Ag, and has had our work cited by the world's leading climate scientists in the latest IPCC report. Heck, we coined the term cellular agriculture.

I am posting here because after sending out an emergency email yesterday, I received a flood of responses asking: What about OPP? Is New Harvest getting any EA funding?

New Harvest's first supporters were EA and our founder, Jason Matheny, continues to be active in the EA world. Despite being recommended by Animal Charity Evaluators as a standout charity and our donor base swinging EA, we haven't seen enough support come through to really sustain us.

Here's how you can help:

Every dollar counts to make it through Giving Season. You can donate here.

Attend our town hall on June 10 at 3:30pm EST (taking place on day two of our upcoming conference in Brooklyn). You can tune in and participate virtually by registering here.

If you're interested in making a major gift, get in touch at [email protected].

Send intros and connections. Impact investors far outnumber those willing to philanthropically support an org building the root system and ecosystem that allow companies in the private sector to flourish. If you have suggestions for how to tap into the investment $$ sloshing around in alt protein, get in touch at [email protected].

We’re on the search for new funders and grant opportunities (ideally grants with quick turnaround that we can apply for with an agile and skeleton team). If you have any leads, please let me know (again: [email protected])

I sincerely believe this will be just one short part of the much longer New Harvest story. New Harvest isn't going anywhere, but we have to make rapid and drastic changes to survive.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Global health is important for the epistemic foundations of EA, even for longtermists, published by Owen Cotton-Barratt on June 2, 2022 on The Effective Altruism Forum.

One of the central/foundational claims of EA as seen in the wild is that some ways of doing good are much better than others.

I think this claim isn’t obvious. Actually I think:

It’s a contingent claim about the world as it is today

While there are theoretical reasons to expect the distribution of opportunities to start distributed over several orders, there are also theoretical reasons to expect the best opportunities to be taken systematically in a more-or-less efficient altruistic market

In fact, EA is aiming for a world where we do have an efficient altruistic market, so if EA does very well, the claim will become false!

It’s pretty reasonable to be sceptical of the claim

One of the most natural reference class claims to consider is “some companies are much better buys than others” . while this is true ex post, it’s unclear how true it is ex ante; why shouldn’t we expect something similar for ways of doing good?

So why is it so widely believed in EA? I think a lot of the answer is that we can look at concrete domains like global health where there are good metrics for how much interventions help — and the claim seems empirically to be true there! But this is within a single cause area (we presumably expect some extra variation between cause areas), and good metrics should make it easier for the altruistic market to be efficient. So the appropriate conclusion is something like “well if it’s true even there where we can measure carefully, it’s probably more true in the general case”.

Another foundational claim which is somewhat contingent about the world is “it’s possible to do a lot of good with a relatively small expenditure of resources”. Again, it seems pretty reasonable to be sceptical of the claim. Again, the concrete examples in global health make a particularly good test case, and I think are valuable in informing many people's intuitions about the general situation.

I think this is an important reason why concrete areas like global health should be prominently featured in introductory EA materials, even if we’re coming from a position that thinks they’re not the most effective causes (e.g. because of a longtermist perspective). I think that we should avoid making this (or being seen to make this) a bait-and-switch by being clear that they’re being used as illustrative examples, not because we think they’re the most important areas. Of course many people in EA do think that global health is the most important cause area, and I don’t want to ignore that or pretend it isn’t the case. Perhaps it’s best to introduce global health examples by explaining that some people think it’s an especially important area, but many others think there are more important areas, but still think it’s a particularly good example for understanding some of the fundamentals of how impact is distributed.

Why not just use a more abstract/toy domain for making these points? If I illustrate them with a metaphor about looking for gold, nobody will mistakenly think I'm claiming that literally searching for gold is the best way to do good. I think this is a great tactic for conveying complex points which are ultimately grounded in theory. However, I don’t think it works for claims which are importantly contingent on the world we find ourselves in. For these, I think we want easy-to-assess domains where we can measure and understand what’s going on. And the closer the domain is to the domains where we ultimately want to apply the inferences, the more likely it is to be valid to import them. Global health — centred around helping people, and with a great deal of effort from many parties going into securing good outcomes, with...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Public beliefs vs. Private beliefs, published by Eli Tyre on June 1, 2022 on LessWrong.

A distinction that I get a lot of value out of is the difference between private beliefs and public beliefs.

Public vs. Private

A public belief is a proposition that someone thinks is true, and justifying on the basis of legible info and reasoning. If X is a public belief, then implicit claim is "not only do I think that X is true, I think that any right thinking person who examines the evidence should come to conclude X."

If someone disagrees with you about a public belief, it is prosocial and epistemically virtuous to defend your claim and to debate the matter on a public forum. Public beliefs, if consensus is reached about them, can be added to the sum total of human knowledge, for others to take for granted and then build on.

A private belief is proposition that someone thinks is true based on their own private or illegible info and reasoning. In this case, the implicit claim is "given my own read of the evidence, I happen to think X. But I don't think that the arguments that I've offered are necessarily sufficient to convince a third party. I'm not claiming that you should believe this, I'm merely providing you the true information that I believe it."

If someone disagrees with you about a private belief, there might or might not be a fruitful discussion to be had about , but it is also important to be able to "agree to disagree."

An example

I think that Circling meaningfully develops real skills of introspection, subtle interpersonal sensitivity, and clarity of map-territory distinctions. I further think that Circling is relevant to the Art of Rationality.

There have been some write-ups that describe why I think this is the case, but I don't know that any of them are persuasive. If a person is curious about why I might be interested in Circling, I think this post is a decent overview. But crucially, I don't think that the evidence presented should be sufficient to convince a skeptic.

I would say that I have a private belief that Circling is useful. It is actually my calibrated view, based on my personal experiences and my reasoning about those experiences. But by stating that belief, I am not at all making a social bid that others believe it too.

A special case: cloaks

There's a special case of having a private belief that, at CFAR, we used to refer to as "having a cloak".

If you are pursuing some ambitious project or personal development goals, it can be damaging to tell people about or justify your ambitions. Many ambitious goals are butterfly ideas, that need to be handled gently. Your own sense of what is possible might be fragile, and you need to nurture it.

And, for many people, sharing their ambitions puts them in a mindset of asking themselves to justify whether they're cool enough to succeed, or needing to fend off a kind of social pressure from causal pessimism. All of which is wasted motion.

So it's useful to have a "cloak": an understanding that your plans and hopes can be private beliefs, that are no one's business but yours.

Sometimes that cloak can be keeping what you're working on a secret (as Paul Graham suggests in What You'll Wish You'd Known).

Alternatively, it's useful to have a true(!), but incomplete, description about what you're aiming to do, that gives others a bucket for conceptualizing your actions, while you also have private, more ambitious plans. (Paul Graham also recommends this sort of cloak, in his "tactics" section of Frighteningly Ambitious Startup Ideas.)

A motivating example is Amazon.com. I bet that back in 1999, Jeff Bezos had at least a glimmer of the the long term future of Amazon. But if Jeff Bezos had outright declared, "Amazon's plan is to build an online bookstore, and eventually conquer almost the whole online retail economy (whi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Revisiting "Why Global Poverty", published by Jeff Kaufman on June 1, 2022 on The Effective Altruism Forum.

Seven years ago I gave a talk,

Why

Global Poverty, at the 2015

effective altruism

conference. I concluded that building the EA movement was the highest priority, but that we should be doing some things that are directly valuable in parallel and that global poverty was the best option there.

Reading back over the talk, with the benefit of seven years of hindsight, I'm not happy with my treatment of existential risk. Here's that section of the talk, with thoughts interspersed:

There's a bit of a continuum. At one end we have risks like an asteroid hitting the earth. Cataloging asteroids and comets that might hit the earth at some point is something that people are working on, and actually is reasonably well understood. Because we have a pretty good understanding, and governments have a lot of sensible people, risks like this are reasonably well funded. So this end of the continuum is probably not high impact.

I'm far more skeptical of the "governments have this covered" position than I was in 2015. Some of this is for theoretical reasons (ex: preventing catastrophe benefits people beyond your country) and some of it is from observing governments more (ex: pandemic response).

At the other end we have risks like the development of an artificial intelligence that destroys us through its indifference. Very few people are working on this, there's low funding, and we don't have much understanding of the problem. Neglectedness is a strong heuristic for finding causes where your contribution can go far, and this does seem relatively neglected. The main question for me, though, is how do you know if you're making progress?

First, a brief digression into feedback loops. [...] Back to AI risk.

The problem is we really really don't know how to make good feedback loops here. We can theorize that an AI needs certain properties not to just kill us all, and that in order to have those properties it would be useful to have certain theorems proved, and go work on those theorems. And maybe we have some success at this, and the mathematical community thinks highly of us instead of dismissing our work. But if our reasoning about what math would be useful is off there's no way for us to find out. Everything will still seem like it's going well.

This was primarily criticism of MIRI's approach, and was about a year before Concrete

Problems in AI Safety came out. That paper had an enormous impact on the field, and when I interviewed one of the co-authors a year later I really liked the emphasis on grounding work in empirical feedback loops.

With existential risk we have a continuum from well understood risks that don't need our marginal contribution to poorly understood risks where we don't have a way to find out if our contribution is reducing the risk. Maybe there's a sweet spot in between, where we can make progress and existing funding bodies are blind to the need? Future generations don't get to vote, so it wouldn't surprise me if governments systematically discount their interest. I'm not aware of any good candidates here, but if you'd like to find me after the talk

I'd be interested in hearing about any.

I think this was mostly wrong. In the talk I divided work into what

Owen

Cotton-Barratt calls "Phase 1" and "Phase 2". First you have indirectly valuable work, such as exploring what things might be good to do, evaluating specific options, or building capacity, and then you have work that more concretely makes the world better, such as distributing bednets, detecting outbreaks, or preventing illicit synthesis of hazardous DNA.

While this Phase 1/2 division is still a good one, and I continue to find the arguments against over-investing in "Phase 1" convincing, I didn't apply it we...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paradigms of AI alignment: components and enablers, published by Vika on June 2, 2022 on The AI Alignment Forum.

(Cross-posted from my personal blog. This post is based on an overview talk I gave at UCL EA and Oxford AI society. Thanks to Janos Kramar for detailed feedback on this post and to Rohin Shah for feedback on the talk.)

This is my high-level view of the AI alignment research landscape and the ingredients needed for aligning advanced AI. I would divide alignment research into work on alignment components, focusing on different elements of an aligned system, and alignment enablers, which are research directions that make it easier to get the alignment components right.

Alignment components

Outer alignment

Inner alignment

Alignment enablers

Mechanistic interpretability

Understanding bad incentives

Foundations

You can read in more detail about work going on in these areas in my list of AI safety resources.

Alignment components

The problem of alignment is getting AI systems to do what we want them to do. Let’s consider this from the perspective of different levels of specification of the AI system’s objective, as given in the Specification, Robustness & Assurance taxonomy. We start with the ideal specification, which represents the wishes of the designer – what they have in mind when they build the AI system. Then we have the design specification, which is the objective we actually implement for the AI system, e.g. a reward function. Finally, the revealed specification is the objective we can infer from behavior, e.g. the reward that the system seems to be actually optimizing for. An alignment problem arises when the revealed specification doesn’t match the ideal specification: the system is not doing what we want it to do.

The gaps between these specification levels correspond to different alignment components. We have outer alignment when the design specification matches the ideal specification, e.g. when the reward function perfectly represents the designer’s wishes. We have inner alignment when the revealed specification matches the design specification, e.g. when the agent actually optimizes the specified reward. (Robustness problems also belong in the design-revealed gap, but we expect them to be less of an issue for advanced AI systems, while inner alignment problems remain.)

Now let’s have a look at how we can make each of those components work.

Outer alignment

The most promising class of approaches to outer alignment is scalable oversight. These are proposals for training an aligned AI system by scaling human oversight to domains that are hard to evaluate.

A foundational proposal for scalable oversight is iterated distillation and amplification (IDA), which recursively amplifies human judgment with the assistance of AI. You start with an agent A imitating the judgment of a human H (the distillation step), then use this agent to assist human judgment at the next level (the amplification step) which results in amplified human HA, and so on. This recursive process can in principle scale up human judgment to any domain, as long as the human overseer is able to break down the task to delegate parts of it to AI assistants.

A related proposal is safety via debate, which can be viewed as a way to implement amplification for language models. Here we have two AIs Alice and Bob debating each other to help a human judge decide on a question. The AIs have an incentive to point out flaws in each other’s arguments and make complex arguments understandable to the judge. A key assumption here is that it’s easier to argue for truth than for falsehood, so the truth-telling debater has an advantage.

A recent research direction in the scalable oversight space is ARC‘s Eliciting Latent Knowledge agenda, which is looking for ways to get a model to honestly tell humans what it...