Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link post] Michael Nielsen's "Notes on Existential Risk from Artificial Superintelligence", published by Joel Becker on September 20, 2023 on The Effective Altruism Forum.

Summary

From the piece:

Earlier this year I decided to take a few weeks to figure out what I think about the existential risk from Artificial Superintelligence (ASI xrisk). It turned out to be much more difficult than I thought. After several months of reading, thinking, and talking with people, what follows is a discussion of a few observations arising during this exploration, including:

Three ASI xrisk persuasion paradoxes, which make it intrinsically difficult to present strong evidence either for or against ASI xrisk. The lack of such compelling evidence is part of the reason there is such strong disagreement about ASI xrisk, with people often (understandably) relying instead on prior beliefs, self-interest, and tribal reasoning to decide their opinions.

The alignment dilemma: should someone concerned with xrisk contribute to concrete alignment work, since it's the only way we can hope to build safe systems; or should they refuse to do such work, as contributing to accelerating a bad outcome? Part of a broader discussion of the accelerationist character of much AI alignment work, so capabilities / alignment is a false dichotomy.

The doomsday question: are there recipes for ruin -- simple, easily executed, immensely destructive recipes that could end humanity, or wreak catastrophic world-changing damage?

What bottlenecks are there on ASI speeding up scientific discovery? And, in particular: is it possible for ASI to discover new levels of emergent phenomena, latent in existing theories?

Excerpts

Here are the passages I thought were interesting enough to tweet about:

"So, what's your probability of doom?" I think the concept is badly misleading. The outcomes humanity gets depend on choices we can make. We can make choices that make doom almost inevitable, on a timescale of decades - indeed, we don't need ASI for that, we can likely4 arrange it in other ways (nukes, engineered viruses, .). We can also make choices that make doom extremely unlikely. The trick is to figure out what's likely to lead to flourishing, and to do those things. The term "probability of doom" began frustrating me after starting to routinely hear people at AI companies use it fatalistically, ignoring the fact that their choices can change the outcomes. "Probability of doom" is an example of a conceptual hazard5 - a case where merely using the concept may lead to mistakes in your thinking. Its main use seems to be as marketing: if widely-respected people say forcefully that they have a high or low probability of doom, that may cause other people to stop and consider why.

But I dislike concepts which are good for marketing, but bad for understanding; they foster collective misunderstanding, and are likely to eventually lead to collective errors in action.

With all that said: practical alignment work is extremely accelerationist. If ChatGPT had behaved like Tay, AI would still be getting minor mentions on page 19 of The New York Times. These alignment techniques play a role in AI somewhat like the systems used to control when a nuclear bomb goes off. If such bombs just went off at random, no-one would build nuclear bombs, and there would be no nuclear threat to humanity. Practical alignment work makes today's AI systems far more attractive to customers, far more usable as a platform for building other systems, far more profitable as a target for investors, and far more palatable to governments. The net result is that practical alignment work is accelerationist. There's an extremely thoughtful essay by Paul Christiano, one of the pioneers of both RLHF and AI safety, where he addresses the question of whether he regrets working ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Compilation of Profit for Good Redteaming and Responses, published by Brad West on September 20, 2023 on The Effective Altruism Forum.

TLDR: Document compiling redteaming on Profit for Good available, with public comments invited and appreciated. This thread further includes two argument headings against Profit for Good to be discussed in the comments.

As one might suspect given that I have started a nonprofit to advance it, I believe Profit for Good, the philanthropic use of businesses with charities in vast majority shareholder position, is an immensely powerful tool. To get a sense of the scale of good that Profit for Good could do, see 14:22 of Cargill's TED Talk for what 3.5T could do, which would be about 3.5 years of 10% of global net profits. Here is a good place to start regarding why we think it is promising and here is a longer reading list.

However, throughout time, many people have brought up criticisms and red-teaming. It is important to realize that Profit for Good, especially in initial stages, would likely require the use of philanthropic funds for launching, acquiring, and accelerating businesses that counterfactually could be used to directly fund extremely impactful charities that we know can do good. For this reason, it is critical that all of the potential reasons that Profit for Good might not succeed, not be the best use of funds, and/or have unintended negative consequences be considered. For this reason, I am in the process of aggregating red-teaming from wherever I can find it, organizing such redteaming, according to argument heading, verbatim where possible, and doing the same for the responses to such red-teaming. I am in the process of forming my own syntheses of the criticisms and responses. I am also continuing to compile new arguments and formulations/evidence regarding former arguments.

If you are interested in supplying new arguments under existing headings, your own syntheses of existing materials, or new argument headings, you may feel free to add a comment on the existing document or email me at [email protected] and I will integrate it verbatim into the document (please indicate if you would like such argument to be anonymous for any reason, or I will credit you with the contribution.

I will include in this piece the first two argument headings, and invite you to contribute to such arguments for or against in this thread (I will update the main document accordingly). I intend to do future posts evaluating later argument headings. These headings regard the criticism (1) that Profit for Good businesses will increase costs for consumers and (2) that Profit for Good businesses will be held to a higher standard than normal businesses in ways that critically impair their ability to compete. Entire Redteaming document also here.

Criticism: Profit for Good businesses will increase costs for consumers

Brad's Criticism Summary: A frequent version of this "criticism" is in fact a confusion of the Profit for Good model with "bundling." With "bundling", a normal firm with normal shareholders would build in a charitable donation into a purchase, increasing the product's price commensurately so as not to harm its shareholders. "Bundling" would increase consumer costs, however, it is not the Profit for Good model. The donation is not a "cost", but rather a function of the identity of the shareholder that is entitled to profit. It is not clear why a charity as shareholder would be a higher cost than a normal investor as a shareholder.

There are versions of this criticism that are not products of confusion as to what Profit for Good is. One is that many businesses that are able to provide the lowest prices are multinational firms and those prices are possible due to billions of dollars of capital costs. Essentially, many kinds of ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Protest against Meta's irreversible proliferation (Sept 29, San Francisco), published by Holly Elmore on September 20, 2023 on The Effective Altruism Forum.

Meta's frontier AI models are fundamentally unsafe. Since Meta AI has released the model weights publicly, any safety measures can be removed. Before it releases even more advanced models - which will have more dangerous capabilities - we call on Meta to take responsible release seriously and stop irreversible proliferation. Join us for a peaceful protest at Meta's office in San Francisco at 250 Howard St at 4pm PT.

RSVP on Facebook or through this form.

Let's send a message to Meta:

Stop irreversible proliferation of model weights. Meta's models are not safe if anyone can remove the safety measures.

Take AI risks seriously.

Take responsibility for harms caused by your AIs.

Stop free-riding on the goodwill of the open-source community. Llama models are not and have never been open source, says the Open Source Initiative.

All you need to bring is yourself and a sign, if you want to make your own. I will lead a trip to SF from Berkeley but anyone can join at the location. We will have a sign-making party before the demonstration-- stay tuned for details. We'll go out for drinks afterward

I like the irony.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The possibility of an indefinite AI pause, published by Matthew Barnett on September 19, 2023 on The Effective Altruism Forum.

This post is part of AI Pause Debate Week. Please see this sequence for other posts in the debate.

tl;dr An indefinite AI pause is a somewhat plausible outcome and could be made more likely if EAs actively push for a generic pause. I think an indefinite pause proposal is substantially worse than a brief pause proposal, and would probably be net negative. I recommend that alternative policies with greater effectiveness and fewer downsides should be considered instead.

Broadly speaking, there seem to be two types of moratoriums on technologies: (1) moratoriums that are quickly lifted, and (2) moratoriums that are later codified into law as indefinite bans.

In the first category, we find the voluntary 1974 moratorium on recombinant DNA research, the 2014 moratorium on gain of function research, and the FDA's partial 2013 moratorium on genetic screening.

In the second category, we find the 1958 moratorium on conducting nuclear tests above the ground (later codified in the 1963 Partial Nuclear Test Ban Treaty), and the various moratoriums worldwide on human cloning and germline editing of human genomes. In these cases, it is unclear whether the bans will ever be lifted - unless at some point it becomes infeasible to enforce them.

Overall I'm quite uncertain about the costs and benefits of a brief AI pause. The foreseeable costs of a brief pause, such as the potential for a compute overhang, have been discussed at length by others, and I will not focus my attention on them here. I recommend reading this essay to find a perspective on brief pauses that I'm sympathetic to.

However, I think it's also important to consider whether, conditional on us getting an AI pause at all, we're actually going to get a pause that quickly ends. I currently think there is a considerable chance that society will impose an indefinite de facto ban on AI development, and this scenario seems worth analyzing in closer detail.

Note: in this essay, I am only considering the merits of a potential lengthy moratorium on AI, and I freely admit that there are many meaningful axes on which regulatory policy can vary other than "more" or "less". Many forms of AI regulation may be desirable even if we think a long pause is not a good policy. Nevertheless, it still seems worth discussing the long pause as a concrete proposal of its own.

The possibility of an indefinite pause

Since an "indefinite pause" is vague, let me be more concrete. I currently think there is between a 10% and 50% chance that our society will impose legal restrictions on the development of advanced AI systems that,

Prevent the proliferation of advanced AI for more than 10 years beyond the counterfactual under laissez-faire

Have no fixed, predictable expiration date (without necessarily lasting forever)

Eliezer Yudkowsky, perhaps the most influential person in the AI risk community, has already demanded an "indefinite and worldwide" moratorium on large training runs. This sentiment isn't exactly new. Some effective altruists, such as Toby Ord, have argued that humanity should engage in a "long reflection" before embarking on ambitious and irreversible technological projects, including AGI. William MacAskill suggested that this pause should perhaps last "a million years". Two decades ago, Nick Bostrom considered the ethics of delaying new technologies in a utilitarian framework and concluded a delay of "over 10 million years" may be justified if it reduces existential risk by a single percentage point.

I suspect there are approximately three ways that such a pause could come about. The first possibility is that governments could explicitly write such a pause into law, fearing the development of AI in a broad sense,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Review] Move First, Think Later: Sense and Nonsense in Improving Your Chess, published by Arjun Panickssery on September 19, 2023 on LessWrong.

The author is Dutch International Master Willy Hendriks (IM is the rank below Grandmaster; there are ~4000 IMs and ~2000 GMs). He takes aim at chess instructors like IM Jeremy Silman, whose popular books The Amateur's Mind and How to Reassess Your Chess teach students a structure for thinking about chess positions:

Before you get carried away, let me remind you: DON'T look at individual moves! In fact, never calculate until you understand the basic components (imbalances) of the position.

The Amateur's Mind

Imbalances include a space advantage on one side of the board, or two knights traded for two bishops, or an unsafe opposing king. Silman lists imbalances like these and tells you to use them to formulate plans which only then suggest individual moves.

Hendriks rejects this definitively: "No chess player thinks like this, no one has learned to play chess by thinking like this and even trainers and authors of chess books don't think like this." Instead, the verbal descriptions are mostly retroactive and follow the initial step of identifying good candidate moves from pattern recognition alone. He gives you some reasons to think this:

He gives some examples of positions where the best move is hard to find unless you're familiar with the pattern. In the game below (Oleg Romanishin v Predrag Nikolic, Leningrad 1987) I marked the correct move for black with an arrow. It's hard to find this move unless you've been exposed to similar maneuvers before but easy to fabricate yourself some position justifications ("Yes, I saw this bishop being a bit hampered by my pawns, and he has weakened his light squares . works nicely together with the rook's semi-open file on f8"). It's often difficult to get students to produce the move using verbal descriptions like these as a hint.

He cites Adrian de Groot's research from Thought and choice in chess in which grandmasters and weaker players think out loud about positions. Hendriks argues against the top players displaying extreme verbal clarity. Instead, "de Groot saw no great differences regarding the process of decision-making: the grandmaster calculated not much more or deeper and didn't seem to decide on his move in a fundamentally different way than the lesser player."

He points out that common positional maxims are probably wrong and contradict each other in any case. An example is taken from Silman: "During my private lessons, I [often] . remind my students to follow one of the finest general rules in chess: The best reaction to an attack on the wing is a counterattack on the center." He selects a sample of 110 games with a 17. g4 kingside attack and concludes that at most 10% call for a counterattack in the center.

The book isn't worth actually reading. Several of the chapters are taken from Hendriks's old magazine columns or something and so have basically no relation to the thesis. He repeats himself, rambles, and the text is awkward (translated from Dutch?)

A typical bizarre tangent:

With the attitude in his book, Silman reminds me a bit of the "Uncle Jan" figure in the famous parody by Donner. In the Netherlands there was a popular manual for beginners called Uncle Jan teaches his nephew how to play chess (first published in 1935). In Donner's parody, there suddenly shows up another uncle, Uncle Hein. He seems to be the black sheep of the family (smoking, drinking, getting thrown out of the chess club for not paying his membership fee) but is, not surprisingly, the better player. Uncle Hein interrupts the solid and respectable teaching of Uncle Jan with some concrete lines that show Uncle Jan's dogmatic approach to be incorrect. But then the lesson ends as the nephew's mother thro...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Luck based medicine: angry eldritch sugar gods edition, published by Elizabeth on September 19, 2023 on LessWrong.

Introduction

Epistemic status: everything is stupid. I'm pretty sure I'm directionally right but this post is in large part correcting previous statements of mine, and there's no reason to believe this is the last correction. Even if I am right, metabolism is highly individual and who knows how much of this applies to anyone else.

This is going to get really in the weeds, so let me give you some highlights

1-2 pounds of watermelon/day kills my desire for processed desserts, but it takes several weeks to kick in.

It is probably a microbiome thing. I have no idea if this works for other people. If you test it let me know.

I still eating a fair amount of sugar, including processed sugar in savory food. The effect is only obvious and total with desserts.

This leads to weight loss, although maybe that also requires potatoes? Or the potatoes are a red herring, it just takes a while to kick in?

Boswellia is probably necessary for this to work in me, but that's probably correcting an uncommon underlying defect so this is unlikely to apply widely.

Stevia-sweetened soda creates desire for sugar in me, even though it doesn't affect my blood sugar. This overrides the watermelon effect, even when I'm careful to only drink the soda with food.

My protein shakes + bars also have zero-calorie sweeteners and the watermelon effect survives them. Unclear if it's about the kind of sweetener, amount, or something else.

Covid also makes me crave sugar and this definitely has a physiological basis.

Metabolism is a terrifying eldritch god we can barely hope to appease, much less understand.

Why do I believe these things? Deep breath this is going to take a while. I've separated sections by conclusion for comprehensibility, but the discovery was messy and interconnected and I couldn't abstract that out.

Boswellia

Last October I told my story of luck based medicine, in which a single pill, taken almost at random, radically fixed lifelong digestion issues. Now's as good a time as any to give an update on that.

The two biggest effects last year were doubling my protein consumption, and cratering sugar consumption. I'm still confident Boswelia is necessary for protein digestion, because if I go off it food slowly starts to feel gross and I become unable to digest protein. I'm confident this isn't a placebo because I didn't know Boswelia was the cause at the time, so going off it shouldn't have triggered a change.

As I'll discuss in a later section, Boswelia is not sufficient to cause a decrease in sugar consumption; that primarily comes from consuming heroic amounts of watermelon. The Boswellia might be necessary to enable that much watermelon consumption, by increasing my ability to digest fiber. I haven't had to go off Boswellia since I figured out how it helps me, so I haven't tested its interaction with watermelon.

How does Boswellia affect micronutrient digestion? I have always scored poorly on micronutrient tests. I had a baseline test from June 2022 (shortly after starting Boswellia + watermelon), and saw a huge improvement in October testing (my previous tests are alas too old to be accessible). Unfortunately this did not hold up - my March and June 2023 tests were almost as bad as June 2022. My leading hypotheses are "the tests suck" and "the November tests are the only ones taken after a really long no-processed-dessert period, and sugar is the sin chemical after all". I hate both of these options.

If we use fuzzier standards like energy level, illness, and injury healing, I'm obviously doing much better. Causality is always hard when tracking effects that accumulate over a year. In that time there's been at least one other major intervention that contributed to energy leve...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust, published by Zac Hatfield-Dodds on September 19, 2023 on LessWrong.

I'm delighted that Anthropic has formally committed to our responsible scaling policy. We're also sharing more detail about the Long-Term Benefit Trust, which is our attempt to fine-tune our corporate governance to address the unique challenges and long-term opportunities of transformative AI.

I and a few colleagues will probably also be in the comments, as usual in a personal capacity rather speaking for my employer.

Today, we're publishing our Responsible Scaling Policy (RSP) - a series of technical and organizational protocols that we're adopting to help us manage the risks of developing increasingly capable AI systems.

As AI models become more capable, we believe that they will create major economic and social value, but will also present increasingly severe risks. Our RSP focuses on catastrophic risks - those where an AI model directly causes large scale devastation. Such risks can come from deliberate misuse of models (for example use by terrorists or state actors to create bioweapons) or from models that cause destruction by acting autonomously in ways contrary to the intent of their designers.

Our RSP defines a framework called AI Safety Levels (ASL) for addressing catastrophic risks, modeled loosely after the US government's biosafety level (BSL) standards for handling of dangerous biological materials. The basic idea is to require safety, security, and operational standards appropriate to a model's potential for catastrophic risk, with higher ASL levels requiring increasingly strict demonstrations of safety.

A very abbreviated summary of the ASL system is as follows:

ASL-1 refers to systems which pose no meaningful catastrophic risk, for example a 2018 LLM or an AI system that only plays chess.

ASL-2 refers to systems that show early signs of dangerous capabilities - for example ability to give instructions on how to build bioweapons - but where the information is not yet useful due to insufficient reliability or not providing information that e.g. a search engine couldn't. Current LLMs, including Claude, appear to be ASL-2.

ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities.

ASL-4 and higher (ASL-5+) is not yet defined as it is too far from present systems, but will likely involve qualitative escalations in catastrophic misuse potential and autonomy.

The definition, criteria, and safety measures for each ASL level are described in detail in the main document, but at a high level, ASL-2 measures represent our current safety and security standards and overlap significantly with our recent White House commitments. ASL-3 measures include stricter standards that will require intense research and engineering effort to comply with in time, such as unusually strong security requirements and a commitment not to deploy ASL-3 models if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers (this is in contrast to merely a commitment to perform red-teaming). Our ASL-4 measures aren't yet written (our commitment is to write them before we reach ASL-3), but may require methods of assurance that are unsolved research problems today, such as using interpretability methods to demonstrate mechanistically that a model is unlikely to engage in certain catastrophic behaviors.

We have designed the ASL system to strike a balance between effectively targeting catastrophic risk and incentivising beneficial applications and safety progress. On the one hand, the ASL system implicitly requires us to temporarily pause training of more powerful models if our AI scal...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some reasons why I frequently prefer communicating via text, published by Adam Zerner on September 19, 2023 on LessWrong.

I often prefer communication via text to chatting in person, on the phone, or via a video call. Let's call this latter grouping "oral communication" and the former "textual communication".

This preference applies to various contexts: social, personal, work, intellectual. And I don't think I personally know anyone who has a stronger preference for this than I do, although I suspect that there are at least a handful of people on LessWrong with stronger preferences than mine.

Here are some reasons that stick out to me for why I frequently prefer textual communication to oral communication, ordered very roughly from most to least important.

I find that, with oral communication, especially in groups, you frequently just end up in a "tangent frenzy" instead of discussing any one thing to something that at least vaguely resembles completion.

It's easier/possible to discuss multiple threads in parallel.

Being asynchronous, I, as well as the person(s) I'm talking with, can take my/their time and think before responding. This helps in figuring out your thoughts as well as with expressing them more clearly.

Personally I find both interruptions and inverted interruptions a good amount more unpleasant than others do.

The conversation can last a lot longer. With in person conversations, you usually will need to end them for practical reasons such as "it's 11pm and time to head home". But with textual communication, there's never really these sorts of obstacles.

I think there is something about textual communication that is more meritocratic. It better holds people accountable. If you say something dumb, someone else can easily call it out in a thread, and if you don't respond to that thread, well, it kinda just sits there and makes you look bad. Sort of. On the other hand, in oral communication, you can say something dumb, the conversation moves on, and people don't get a proper chance to call you out for the dumb thing.

It's easy to share the conversation with others.

For various reasons, sometimes it's nice to be able to reference the conversation in the future.

I like being able to quote things. Sometimes quoting something the other person said to make it clear that I am responding to it. Other times quoting something someone who isn't part of the conversation said, but that I think is relevant.

If I have something to say that has a dependency, I can say it, link to the dependency, and then the other person can, asynchronously, read the dependency before reading the main thing.

I can ramble. With oral communication, I feel like it's a bit of an anti-pattern to say things of the form "I think X because of 1, 2 and 3. I also suspect Y, also because of 1 and 2 but also because of 4 and 5. This is probably tangential, but I strongly suspect Z because of 6." It's a bit rambly and doesn't really give the other person or people a chance to butt in and say "actually, I'd like to dispute 2". At least not at the most convenient time.

Sometimes it's helpful to reference an image or graphic.

I wouldn't say that I could be more expressive in written communication, but I would say that I could be differently expressive -- such as saying things parenthetically like this -- in written communication.

Sometime I prefer to say things in a way that sounds "fancy". In oral communication, even amongst rationalists, I often feel awkward about it and try to think of a more informal way of saying the thing. I usually wish I didn't have to do that though.

Well, speaking more precisely, these are a bunch of benefits that I frequently see to textual communication. My preference depends on costs and benefits, on both sides. So what I really mean is that these sorts of benefits frequentl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My life would be much harder without the Community Health Team. I think yours might too., published by Daystar Eld on September 19, 2023 on The Effective Altruism Forum.

I don't have much time to get into this, but I heard rumblings, saw a post, and wrote a comment, and now I'm making a post of my own because this information feels worth spreading now. I will not be going into much more details, for reasons that should be obvious.For those that don't know me, I'm a therapist who has been working in the community for about 5 years now, and has been almost exclusively working with the community since early 2020, though I've been scaling down my therapy practice to focus on other projects like therapist recruitment/supervision, and mental health research. I also do mediation now and then, and in both situations, Community Health has been incredibly helpful.

Another of my major things is teaching at various rationality camps and workshops for highschool age students, as well as some for adults. To say that Community Health has been incredibly valuable here would be an understatement. I often I've spoken at a few EAGs, which is relevant insofar as sometimes things happen there which also requires Community Health interaction. I think it is incredibly easy to undervalue the good CH does, particularly if people don't regularly interact with it or make use of them rather than just having a few anecdata to go off of.I think it is also incredibly easy to be uncharitable toward CH if you don't interact with them regularly and only have a few anecdata to go off of. There's no appropriate comparison for the work they do, but police with ~most of the responsibility but ~none of the power seems apt as a first approximation.

If people have been burned by CH before, I get it. Maybe it seems weird to say that they have ~no power, when clearly they have a lot of social power in particular circumstances. But from my observations, this power is roughly equivalent to what any group of people can accomplish by acting as a whisper network, and the fact that most people don't does not give CH extra power when they try to serve the same purpose with more self-awareness and fairness than most would.

What I can say about my own experiences, however, is that CH often does an amazing job of walking the tightwire between taking accusations seriously without accepting them at face value. I have seen them let people know what they've been accused of, so long as they have not promised anonymity to reporters. I have seen them inform people that anonymity in reporting and investigation/consequence level directly trade off against each other. I have seen them spend many hours investigating stuff to try and reach a fair and balanced conclusion.

There are some situations that take up hundreds of hours of grueling emotional labor by people working in organizations like the ones I've been part of, and sometimes CH only helps a little with that, but other times they help a lot. They also help resolve many issues that could grow into bigger ones by being a mediating force. Basically no one knows about any of the times they do things well because why would they?I get that transparency is good, but so is privacy, and one of the points of having a CH organization is to not turn every event into a massive drama that sucks in thousands of hours from hundreds of people. Sometimes that's appropriate, sometimes it's not, nor wanted by the parties involved.

As for people worried about being talked about or blacklisted from things without their knowledge... again, I get it. But I promise you, whisper networks will not disappear without CH, and from those I've participated in both in this community and outside it, they're not made worse from the existence of CH. Quite the contrary.

So yeah. I need to get back to work, b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic's Responsible Scaling Policy & Long Term Benefit Trust, published by Zac Hatfield-Dodds on September 19, 2023 on The AI Alignment Forum.

I'm delighted that Anthropic has formally committed to our responsible scaling policy. We're also sharing more detail about the Long-Term Benefit Trust, which is our attempt to fine-tune our corporate governance to address the unique challenges and long-term opportunities of transformative AI.

I and a few colleagues will probably also be in the comments, as usual in a personal capacity rather speaking for my employer.

Today, we're publishing our Responsible Scaling Policy (RSP) - a series of technical and organizational protocols that we're adopting to help us manage the risks of developing increasingly capable AI systems.

As AI models become more capable, we believe that they will create major economic and social value, but will also present increasingly severe risks. Our RSP focuses on catastrophic risks - those where an AI model directly causes large scale devastation. Such risks can come from deliberate misuse of models (for example use by terrorists or state actors to create bioweapons) or from models that cause destruction by acting autonomously in ways contrary to the intent of their designers.

Our RSP defines a framework called AI Safety Levels (ASL) for addressing catastrophic risks, modeled loosely after the US government's biosafety level (BSL) standards for handling of dangerous biological materials. The basic idea is to require safety, security, and operational standards appropriate to a model's potential for catastrophic risk, with higher ASL levels requiring increasingly strict demonstrations of safety.

A very abbreviated summary of the ASL system is as follows:

ASL-1 refers to systems which pose no meaningful catastrophic risk, for example a 2018 LLM or an AI system that only plays chess.

ASL-2 refers to systems that show early signs of dangerous capabilities - for example ability to give instructions on how to build bioweapons - but where the information is not yet useful due to insufficient reliability or not providing information that e.g. a search engine couldn't. Current LLMs, including Claude, appear to be ASL-2.

ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities.

ASL-4 and higher (ASL-5+) is not yet defined as it is too far from present systems, but will likely involve qualitative escalations in catastrophic misuse potential and autonomy.

The definition, criteria, and safety measures for each ASL level are described in detail in the main document, but at a high level, ASL-2 measures represent our current safety and security standards and overlap significantly with our recent White House commitments. ASL-3 measures include stricter standards that will require intense research and engineering effort to comply with in time, such as unusually strong security requirements and a commitment not to deploy ASL-3 models if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers (this is in contrast to merely a commitment to perform red-teaming). Our ASL-4 measures aren't yet written (our commitment is to write them before we reach ASL-3), but may require methods of assurance that are unsolved research problems today, such as using interpretability methods to demonstrate mechanistically that a model is unlikely to engage in certain catastrophic behaviors.

We have designed the ASL system to strike a balance between effectively targeting catastrophic risk and incentivising beneficial applications and safety progress. On the one hand, the ASL system implicitly requires us to temporarily pause training of more powerful models i...