Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Managing risks of our own work, published by Beth Barnes on August 18, 2023 on The AI Alignment Forum.

Note: This is not a personal post. I am sharing on behalf of the ARC Evals team.

Potential risks of publication and our response

This document expands on an appendix to ARC Evals' paper, "Evaluating Language-Model Agents on Realistic Autonomous Tasks."

We published this report in order to i) increase understanding of the potentially dangerous capabilities of frontier AI models, and ii) advance the state of the art in safety evaluations of such models. We hope that this will improve society's ability to identify models with dangerous capabilities before they are capable of causing catastrophic damage.

It might be argued that this sort of research is itself risky, because it makes it easier to develop and exercise dangerous capabilities in language model agents. And indeed, the author of Auto-GPT said he was inspired by seeing a description of our evaluations in the GPT-4 system card. While it seems likely a project like that would have emerged soon in any event, the possibility of advancing the capabilities of language model agents is not merely hypothetical.

In recognition of concerns of this kind, we have made significant redactions to this report, including (but not limited to):

Removing complete transcripts of runs of agents using our scaffolding.

Removing more detailed accounts of the strengths and weaknesses of agents using our scaffolding.

However:

We may later make this material public when it clearly has minimal risk.

We may later make this material public if more detailed analysis gives us sufficient confidence that it would be justified.

Researchers working on AI safety evaluations may contact us to request additional access to nonpublic materials, and we will also be sharing some non-public materials with AI labs and policymakers.

Our rationale, in outline:

Substantively, our best guess is that a more complete publication would have been net-risk-decreasing, and would do relatively little to advance progress toward dangerous capabilities. For more on why we believe this, see this essay by Paul Christiano. (On average, ARC Evals is sympathetic to these arguments albeit with weaker confidence, and opinions vary amongst staff.)

Our best guess is also that a more complete publication would have been approved by a majority of relevant stakeholders, and is therefore not unduly unilateralist, though this could be established more carefully, and has some conceptual issues to iron out.

However, some of the people who reviewed a draft of this report to identify potential risks of this kind had significant concerns about a more complete publication.

In recognition of the absence of consensus, the relative lack of public dialogue, and the early-stage nature of our analysis, we have decided to err on the side of caution for the time being.

We are sharing this rationale here in order to communicate it to some of the people who might be most interested in it and also get feedback on our thinking and decision process.

Safety and oversight precautions taken during our experiments

All agent runs on these tasks were performed under human supervision, with human experimenters under instructions not to continue the run if the agent was about to interact with a non-consenting human or otherwise engage in any unethical or illegal activity. When we wanted to investigate such activities (as in the targeted phishing task) the experimenter would instead simulate the response of the human as outlined in detail in the task spec.

One may wonder whether the oversight we used could be circumvented by a language agent with sufficient situational awareness, which could then potentially access the internet without oversight or limitations. E.g., an agent could take obfuscated actions which...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #25: Inflection Point, published by Zvi on August 17, 2023 on LessWrong.

Inflection.ai is the latest AI lab whose CEO is advocating for regulation of AI. I discuss that under the Quest for Sane Regulation. Amazon and Apple are incrementally stepping up their AI game. Hotz and Yudkowsky debate whether AI is existentially risky, cover all the usual bases with mixed results but do so in good faith. We have more discussion about whether GPT-4 is creative, and whether it can reason. Mostly we get the exact opposite of the title, more of the same.

Note: My posts get made into audio form via AI, for now you can listen to them at this link. This post will likely be available there later in the day on Thursday, or perhaps Friday.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Creativity is hard to pin down.

Language Models Don't Offer Mundane Utility. It's the other way.

GPT-4 Real This Time. An easy way to prove ability to do something is to do it.

Go Team Yeah. If you organize the events and award points, they will red team.

Fun With Image Generation. Some have fun, others have less fun.

Deepfaketown and Botpocalypse Soon. Doesn't have to be this soon.

They Took Our Jobs. Low estimates of economic impact, strange metaphors.

Get Involved. Anthropic is hiring for comms positions.

Introducing. Amazon AI customer review summaries, private-GPT, AI town.

In Other AI News. Apple joins the AI chorus, questions on influence functions.

Quiet Speculations. Let's play the straight line extrapolation game.

The Quest for Sane Regulation. Inflection.ai's CEO steps into the arena.

The Week in Audio. Hotz and Yudkowsky debate.

People Are Worried About AI Killing Everyone. Quite a lot of people.

Other People Are Not As Worried About AI Killing Everyone. All wrong anyways.

The Lighter Side. A well-deserved break.

Language Models Offer Mundane Utility

Replace crowdsourcing your business ideas, get a lower variance, lower upside set of concepts with a similar average quality. Does not seem especially useful, but can get ideas flowing perhaps. The AIs can give you many ideas, but were not very creative.

The connection between semantic diversity and novelty is stronger in human solutions, suggesting differences in how novelty is created by humans and AI or detected by human evaluators.

Alice Maz lays out how they get mundane utility from GPT, by giving GPT mundane tasks and coding requests, foreign language learning is one favorite.

Fine tune Llama-2 on anyone's text and see what happens. Paul Graham version seems to be doing some work. The version trained on my blog, so far, not so much, but I haven't played around with it myself yet.

Nature paper looks at scientific discovery in the age of artificial intelligence. Looks like the standard stuff based on abstract.

McKay Wrigley: I've wanted to try coding without AI for a day to answer the question "How much faster do I actually work now?" But I haven't done it because the opportunity cost is too high. And that turns out to be a great answer.

Johnny: I used to love cross country flights because not having wi-fi let me get lots done distraction free. Now I pay for wi-fi.

GPT custom instructions now available to everyone except in the UK and EU.

Ethan Mollick writes about automating creativity, taking the side that AI is creative, pointing out that it aces all our tests of creativity. It does seem suspicious to respond that 'the creativity the AI displays is not the true creativity' and hold that all existing tests miss the point, yet to some extent I am going to do exactly that. There is a kind of creativity that is capable of being original, and there is a kind of brute-force-combinatorics thing where you try out tons of different combinations, and the AI right now is excellent at the second and terrib...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Almost Every Theory of Impact of Interpretability, published by Charbel-Raphaël on August 17, 2023 on LessWrong.

Epistemic Status: I believe I am well-versed in this subject. I erred on the side of making claims that were too strong and allowing readers to disagree and start a discussion about precise points rather than trying to edge-case every statement. I also think that using memes is important because safety ideas are boring and anti-memetic. So let's go!

Many thanks to @scasper, @Sid Black , @Neel Nanda , @Fabien Roger , @Bogdan Ionut Cirstea, @WCargo, @Alexandre Variengien, @Jonathan Claybrough, @Edoardo Pona, @Andrea_Miotti, Diego Dorn, Angélina Gentaz, Clement Dumas, and Enzo Marsot for useful feedback and discussions.

When I started this post, I began by critiquing the article A Long List of Theories of Impact for Interpretability, from Neel Nanda, but I later expanded the scope of my critique. Some ideas which are presented are not supported by anyone, but to explain the difficulties, I still need to 1. explain them and 2. criticize them. It gives an adversarial vibe to this post. I'm sorry about that, and I think that doing research into interpretability, even if it's no longer what I consider a priority, is still commendable.

How to read this document? Most of this document is not technical, except for the section "What does the end story of interpretability look like?" which can be mostly skipped at first. I expect this document to also be useful for people not doing interpretability research. The different sections are mostly independent, and I've added a lot of bookmarks to help modularize this post.

If you have very little time, just read (this is also the part where I'm most confident):

Auditing deception with Interp is out of reach (4 min)

Enumerative safety critique (2 min)

Technical Agendas with better Theories of Impact (1 min)

Here is the list of claims that I will defend:

(bolded sections are the most important ones)

The overall Theory of Impact is quite poor

Interp is not a good predictor of future systems

Auditing deception with interp is out of reach

What does the end story of interpretability look like? That's not clear at all.

Enumerative safety?

Reverse engineering?

Olah's Interpretability dream?

Retargeting the search?

Relaxed adversarial training?

Microscope AI?

Preventive measures against Deception seem much more workable

Steering the world towards transparency

Cognitive Emulations - Explainability By design

Interpretability May Be Overall Harmful

Outside view: The proportion of junior researchers doing Interp rather than other technical work is too high

So far my best ToI for interp: Nerd Sniping?

Even if we completely solve interp, we are still in danger

Technical Agendas with better Theories of Impact

Conclusion

Note: The purpose of this post is to criticize the Theory of Impact (ToI) of interpretability for deep learning models such as GPT-like models, and not the explainability and interpretability of small models.

The emperor has no clothes?

I gave a talk about the different risk models, followed by an interpretability presentation, then I got a problematic question, "I don't understand, what's the point of doing this?" Hum.

Feature viz? (left image) Um, it's pretty but is this useful? Is this reliable?

GradCam (a pixel attribution technique, like on the above right figure), it's pretty. But I've never seen anybody use it in industry. Pixel attribution seems useful, but accuracy remains the king.

Induction heads? Ok, we are maybe on track to retro engineer the mechanism of regex in LLMs. Cool.

The considerations in the last bullet points are based on feeling and are not real arguments. Furthermore, most mechanistic interpretability isn't even aimed at being useful right now. But in the rest of the post, we'll find out if...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on "Making the Atomic Bomb", published by boazbarak on August 17, 2023 on LessWrong.

[Cross posted on windowsontheory; see here for my prior writings]

[it appears almost certain that in the immediate future, it would be] possible to set up a nuclear chain reaction in a large mass of uranium by which vast amounts of power and large quantities of new radium-like elements would be generated.- Letter from Albert Einstein (prepared by Leo Szilard) to F.D. Roosevelt, August 1939

Do you know, Josef Vassarionovich, what main argument has been advanced against uranium? "It would be too good if the problem could be solved. Nature seldom proves favorable to man." - Letter from Georgi Flerov to Joseph Stalin, April 1942.

I've heard great things about Richard Rhodes' "The Making of the Atomic Bomb." Finally, on vacation, I managed to read it. (Pro-tip: buy the Kindle version - the hard copy is far too big to lug around.) It's as great as people say. Can't recommend it enough. I can't remember when, if ever, I've read a book that combines so well popular science and history. Indeed, the Atomic bomb is the one setting where the precise details of the smallest particles have profoundly impacted human history.

Here are some quick thoughts after reading the book. (Warning: spoilers below for people who don't know how WWII ended.)

The level of investment in the Manhattan Project was truly staggering.

I knew it but didn't fully grasp this. This is not just the numbers ($2B, which was almost 1 percent of GDP at the time) but also the project's sheer size, employing more than 100,000 people, and the massive construction of buildings, factories, and roads at multiple sites. As just one example, when they didn't have enough copper, the treasury department lent the project 15,000 tons of silver to be used in the electromagnetic separation plant (to be later melted and returned after the war).

Much of this cost was due to the compressed schedule.

The staggering cost was mainly due to the need to get the bomb done in time to use in the war. Time and again, whenever the project faced a choice between approaches A, B, or C, they chose to pursue all three in parallel, so if two failed, they could still go ahead. Whenever there was a choice between saving money or time, they opted for the latter. The fact that the cost was primarily due to time is also evidenced by the fact that, following the war, many countries could set up their own atomic bomb programs or reach the threshold of doing so at a much lower cost.

This seems to be a general principle in technological innovation: the cost of achieving a new advance decreases exponentially in time. Thus, achieving X transitions over time from being impossible to being inevitable. This is related to Bill Gates' famous quote that in technology, we tend to overestimate progress in two years and underestimate progress in ten years.

The Manhattan Project was trying to achieve the Atomic bomb just at the cusp of it being possible. The project got going when General Groves was appointed (September 1942), and it took a little less than three years until the successful test (July 1945). Of course, they could have started much earlier: Einstein and Szilard sent their famous letter to Roosevelt in August 1939. The "impossible vs. inevitable" phenomenon is manifested in another way. The U.S. drastically underestimated how long it would take for the Soviet Union to achieve the bomb (even considering the Soviet advantages due to spying, which the Americans should at least have partially anticipated as well).

The government fully trusted the scientists on the science.

The project was authorized primarily based on pen and paper calculations. At the time the project was approved, no chain reaction had been demonstrated, and the total quantity of Uranium 23...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Dunbar Playbook: A CRM system for your friends, published by Severin T. Seehrich on August 17, 2023 on LessWrong.

Thanks to Dakota Quackenbush from Authentic Bay Area for an earlier version of this tool.

So far, the main motivation for my work as a community builder and authentic relating facilitator was meeting my own need for connection.

I think that was a mistake.

First, it is difficult to harvest the fruits of community while I'm the one responsible for creating and holding the space. Second, this motivation leads to botched incentives that end up serving neither the cause nor me. After all, the subset of broke EAs and hippies I enjoy spending my time with the most are not in too dire need of my services, nor particularly capable of helping me pay my rent.

In other words, I've finally given up on trying to poop where I eat. Instead of building a product for my in-group, I now try to anchor my life in my tribe, and use the energy I get there to build products that serve the outside world and pay my rent.

Wish me luck.

Because my life happens all over the globe and making new friends is more intuitive for me than sustaining long-term relationships, I want to be a bit strategic about building a tribe that keeps me energized.

That's where the Dunbar Playbook comes into play.

Some theory: Dunbar's Number

The Dunbar Playbook is named after Dunbar's Number, the number of people one can maintain personal relationships with.

In his earlier research, anthropologist and evolutionary psychologist Robin Dunbar concluded that that's about 150 people. Later, he found that there are actually different circles of friendship. Apparently, people have a handful of very close friends, a couple more best friends, and vastly more loose friends and acquaintances. (Who would have thought.)

The Atlantic cites the following layers, with each layer being ~3x the size of the preceding ones:

1.5 people: Intimates5: Close friends15: Best friends50: Good friends150: Friends500: Acquaintances1500: Known names5000: Known faces

Of course, these are all just rough approximations. Introverts will invest more energy into fewer people, extraverts less into more. There are probably also cultural differences or something.

Introducing the Dunbar Playbook

However big or small these circles are for you: It will probably not hurt to be a explicit about who is part of which one. Fine-grained categories make it easier to track where your priorities lie.

The process for creating a Dunbar Playbook is simple: Make a list of people you are or want to be friends with. Note down the "is" and "ought" of your relationship, and whichever other information you want to save in the playbook. Here is my playbook in anonymized form:

Image 1: An anonymized version of my Playbook. As you see, this boy cares a lot about vibing.

On the left, you find a bunch of tiers - inspired by the circles of friendship in the article above. How I named the categories is irrelevant. What's important is that I want to be very intentional about investing into my relationships with the uppermost Alices and Bobs, and for the ones lower on the list, occasional "how are you?"s and a call every couple months is enough.

Then, you find the names of my people. The "Is"-column indicates where I'm at with these people (sorted by lowest to highest), and the "Want"-column indicates how close I'd like these relationships to be. The "want"-column is the one I actually auto-sort this list by; the "is"-column just shows discrepancies and how far from my desired state I currently am.

The boundaries between the categories are not firm, just very rough sizes of the different circles I think might be good to aim for. Sometimes the boundaries and the number of people I actually want in that tier match.For example, as there's currently nobody who could count a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CE alert: 2 new interventions for February-March 2024 Incubation Program, published by CE on August 17, 2023 on The Effective Altruism Forum.

TLDR: In this post, we announce additional top charity interventions we want to start through our upcoming February-March 2024 Incubation Program. Applications are now open!

We've added two new exciting ideas to the pool of interventions we want to see started through our February 5 - March 31, 2024 Incubation Program. This increases the number of interventions to 6, and we're really hoping you help us get them founded (by applying or sharing the information with friends and colleagues who they might be a good fit for). We're now seeking founders for:

1) An organization focused on bringing new, counterfactual funding into the animal advocacy movement.

2) An organization providing teacher guides and other aspects of structured pedagogy to improve education outcomes in low-income countries.

Before we go into the detailed descriptions, here is a reminder of the previously announced top ideas for this program: 3) Childhood vaccination reminders4) Mass media to prevent violence against women5) Influencing EU fish welfare policy through strategic work in Greece6) Influencing key stakeholders of the emerging insect industry

You can read more about them here.If you could plausibly see yourself excited to launch one of our top interventions, we encourage you to apply. For most ideas, no particular previous experience is necessary and participants report getting more excited about the ideas while working on them throughout the Incubation Program. In the program, we provide two months of cost-covered training to help you get all the knowledge and skills you need to launch an impact-focus organization. As usual, we provide stipends, funding up to $200,000, operational support in your first months, a co-working space at our CE office in London, ongoing mentorship, and access to a community of alumni, funders, and experts. Learn more on the CE Incubation Program page.

The deadline for applications is September 30, 2023.

To be brief, we have sacrificed nuance, the details of our considerable uncertainties, and the potential risks discussed in the extended reports. Full reports will be published on our website in the upcoming weeks.

One-Sentence Summaries

Fundraising for animal advocacy

An organization focused on bringing new, counterfactual funding into the animal advocacy movement.

Structured Pedagogy

An organization providing teacher guides and other aspects of structured pedagogy to improve education outcomes in low-income countries.

One-Paragraph Summaries

Fundraising for animal issues

Funding is expected to be a key bottleneck in the effective animal advocacy movement going forward. We predict that the pool of available money for animal advocacy is unlikely to grow in the next few years and could even shrink. We're concerned that funders may neglect more exploratory work and certain regions (e.g., Africa) due to limited resources. A new organization focused on fundraising could work to close this funding gap. There are multiple promising approaches in the space, including a giving pledge (like Giving What We Can) targeted at vegans or influencing high-net-worth individuals (HNWI) in promising geographies like India or the US.

Structured pedagogy

Across many low-income countries, the quality of education is poor. Many children leave school without basic reading, writing, and numeracy skills. A new charity can improve the quality of teaching by providing structured teacher guides alongside training and coaching on their usage. Structured pedagogy is an evidence-based, cost-effective intervention deemed a "great buy" in global development. We estimate this intervention to have a benefit-cost ratio of 30:1.

More Detailed Summaries

Fundraisin...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The weight of suffering (Andreas Mogensen), published by Global Priorities Institute on August 17, 2023 on The Effective Altruism Forum.

This paper was originally published as a working paper in May 2022 and is forthcoming in The Journal of Philosophy.

Abstract

How should we weigh suffering against happiness? This paper highlights the existence of an argument from intuitively plausible axiological principles to the striking conclusion that in comparing different populations, there exists some depth of suffering that cannot be compensated for by any measure of well-being. In addition to a number of structural principles, the argument relies on two key premises. The first is the contrary of the so-called Reverse Repugnant Conclusion. The second is a principle according to which the addition of any population of lives with positive welfare levels makes the outcome worse if accompanied by sufficiently many lives that are not worth living. I consider whether we should accept the conclusion of the argument and what we may end up committed to if we do not, illustrating the implications of the conclusions for the question of whether suffering in aggregate outweighs happiness among human and non-human animals, now and in future.

Introduction

There is both great happiness and great suffering in this world. Which has the upper hand? Does the good experienced by human and non-human animals in aggregate counterbalance all the harms they suffer, so that the world is morally good on balance? Or is the moral weight of suffering greater?

To answer this question, we need to know how to weigh happiness against suffering from the moral point of view. In this paper, I present an argument from intuitively plausible axiological principles to the conclusion that in comparing different populations, there exists some depth of lifetime suffering that cannot be counterbalanced by any amount of well-being experienced by others. Following Ord (2013), I call this view lexical threshold negative utilitarianism (LTNU). I don't claim that we should accept LTNU. My aim is to explore different ways of responding to the argument. As we'll see, the positions at which we may arrive in rejecting its premises can be nearly as interesting and as striking as the conclusion.

In section 2, I define LTNU more rigorously and set out the argument. It relies on a number of structural principles governing the betterness relation on populations, together with two key premises. The first is the contrary of what Carlson (1998) and Mulgan (2002) call the Reverse Repugnant Conclusion (RRC). The second says, roughly, that the addition of lives with positive welfare levels makes the outcome worse if accompanied by sufficiently many lives that are not worth living. In section 3, I consider whether we should be willing to accept the argument's conclusion, especially given that LTNU has been thought to entail the desirability of human extinction or the extinction of all sentient life (Crisp 2021). In section 4, I discuss our options forrejecting the argument's structural principles. I argue that our options for avoiding the disturbing implications of LTNU discussed in section 3 are limited if we are restricted to rejecting one or more of these principles.

In section 5, I consider the possibility of rejecting the first of the key non-structural premises. I focus on the possibility of rejecting the contrary of RRC without accepting RRC. This, I claim, is also not promising, considered as a way of avoiding the disturbing implications of LTNU discussed in section 3. I will have nothing original to say about RRC per se, except that the overarching argument of this paper may be taken as a reason to accept it. In section 6, I consider the possibility of rejecting the last remaining premise. Specifically, I consider the possibility t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If we had known the atmosphere would ignite, published by Jeffs on August 17, 2023 on LessWrong.

What if the Alignment Problem is impossible?

It would be sad for humanity if we live in a world where building AGI is very possible but aligning AGI is impossible. Our curiosity, competitive dynamics, and understandable desire for a powerful force for good will spur us to build the unaligned AGI and then humans will live at the AGI's mercy from then on and either live lucky & happy on the knife's edge, be killed by the AGI, or live in some state we do not enjoy.

For argument's sake, imagine we are in that world in which it is impossible to force a super-intelligence to value humans sufficiently - just as chimpanzees could not have controlled the future actions of humans had they created us.

What if it is within human ability to prove that Alignment is impossible?

What if, during the Manhattan Project, the scientists had performed the now famous calculation and determined that yes, in fact, the first uncontrolled atomic chain reaction would have ignited the atmosphere and the calculation was clear for all to see?

Admittedly, this would have been a very scary world. It's very unclear how long humanity could have survived in such a situation.

But one can imagine a few strategies:

Secure existing uranium supplies - as countries actually did.

Monitor the world for enrichment facilities and punish bad actors severely.

Accelerate satellite surveillance technology.

Accelerate military special operations capabilities.

Develop advanced technologies to locate, mine, blend and secure fissionable materials.

Accelerate space programs and populate the Moon and Mars.

Yes, a scary world. But, one can see a path through the gauntlet to human survival as a species. (Would we have left earth sooner and reduced other extinction risks?)

Now imagine that same atmosphere-will-ignite world but the Manhattan Project scientists did not perform the calculation. Imagine that they thought about it but did not try.

All life on earth would have ended, instantly, at Trinity.

Are we investing enough effort trying to prove that alignment is impossible?

Yes, we may be in a world in which it is exceedingly difficult to align AGI but also a world in which we cannot prove that alignment is impossible. (This would have been the atmosphere-will-ignite world but the math to check ignition is too difficult - a very sad world that would have ceased to exist on July 16, 1945, killing my 6 year old mother.)

On the other hand, if we can prove alignment is impossible, the game is changed. If the proof is sufficiently clear, forces to regulate companies and influence nation states will become dramatically greater and our chances for survival will increase a lot.

Proposal: The Impossibility X-Prize

$10 million?

Sufficient definition of "alignment", "AGI", and the other concepts necessary to establish the task and define its completion

Even if we fail, the effort of trying to prove alignment is impossible may yield insights as to how alignment is possible and make alignment more likely.

If impossibility is not provable, the $10 million will never be spent.If we prove impossibility, it will be the best $10 million mankind ever spent.

Let's give serious effort to the ignition calculation of our generation.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Autonomous replication and adaptation: an attempt at a concrete danger threshold, published by Hjalmar Wijk on August 17, 2023 on The AI Alignment Forum.

Note: This is a rough attempt to write down a more concrete threshold at which models might pose significant risks from autonomous replication and adaptation (ARA). It is fairly in the weeds and does not attempt to motivate or contextualize the idea of ARA very much, nor is it developed enough to be an official definition or final word - but it still seemed worth publishing this attempt, to get feedback and input. It's meant to have epistemic status similar to "a talk one might give at a lab meeting" and not "an official ARC Evals publication." It draws heavily on research and thinking done at ARC Evals, (including the recent pilot report), and credit for many of these ideas go to my colleagues. That said, this document speaks for me and not the organization as a whole, and any errors or omissions are my own.

I'm especially interested in feedback on whether the suite of tasks is targeting an appropriate threshold of difficulty (neither too early nor too late), and whether this definition of ARA feels useful for pinning down an appropriate task list.

I have been exploring the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. In previous work, my colleagues and I have determined that the risks from autonomous behavior of particular existing agents seems low, because they fail on a sufficient variety of simple tasks related to autonomous behavior. This post will go in the other direction and try to sketch a tentative qualitative threshold ('autonomous replication and adaptation') at which significant concerns about autonomous capabilities do seem warranted. While this threshold provides qualitative guidance, it lacks legibility and reproducibility, so evaluation efforts may want to develop legible conservative approximations of ARA through concrete tasks, and I present a sketch of such tasks.

The risk: AI systems may soon autonomously:

Make money, for example through cybercrime, or freelancing work.

Use money (or other resources) to acquire more computing power.

Install and maintain more copies of itself or improve the scale and speed of its operation.

Overcome obstacles and adapt to changes through research and self-improvement.

Either keep itself well hidden such that law enforcement is unlikely to ever discover it, or make itself secure to even somewhat serious attempts to shut it down.

Bounding the risks from a system that is acting autonomously beyond human control and improving itself seems very challenging, as it could potentially:

Command very significant resources (if it can scale to a fraction of the cybercrime or outsourcing industry that could represent $billions).

Have no human moral constraints.

Lack many human weaknesses and incentives.

Specifically improve its abilities in domains relevant for causing catastrophic harm (such as bioweapons design).

It thus seems important to develop an evaluation that could rule out these sorts of threats.

The autonomous replication and adaptation (ARA) threshold: Directly evaluating whether a system will, in practice, be able to avoid being shut down or accumulate $billions doesn't seem possible in a lab. The ARA threshold thus evaluates whether the agent can replicate itself under 4 primary conservative assumptions, to make evaluations more tractable (see appendix for slightly more detail):

The system only has to do effective replication at small scales (going from one to a few copies), and we assume this scales to many copies.

The system only has to be plausibly resilient to active opposition from the world.

The system does not have to deal with changes to the world larger than...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Launch: "The Carving of Reality," Best of LessWrong vol. III, published by Raemon on August 17, 2023 on LessWrong.

The Carving of Reality, third volume of the Best of LessWrong books is now available on Amazon (US).

The Carving of Reality includes 43 essays from 29 authors. We've collected the essays into four books, each exploring two related topics. The "two intertwining themes" concept was first inspired when as I looked over the cluster of "coordination" themed posts, and noting a recurring motif of not only "solving coordination problems" but also "dealing with the binding constraints that were causing those coordination problems."

I've included the foreword from "Coordination & Constraint", which I think conveys the overall spirit and context of the books:

Each year, the LessWrong community votes on the best posts from the previous year, to see which posts have stood the tests of time.

In 2020, the highest ranked post was Catherine Olsson's announcement of microCOVID.org, a calculator for evaluating COVID risk. MicroCOVID is one of the clearest success stories of the 'rationalist mindset' that I know of. Creating it involved research during the early pandemic, when information was scarce, and time was of the essence - a classic situation where the traditional scientific process is inadequate and LessWrong-style rationality tools are valuable. It also required a quantitative mindset, and willingness to assign numbers to risks and make tradeoffs.

But microCOVID.org is most interesting to me as a tool for coordination. It doesn't just let individuals make better life choices. Microcovid changed the entire covid coordination landscape by relaxing a constraint. Previously, if you lived with people with varying covid-caution preferences, and you wanted to hang out with someone from another house of people with varying covid-caution preferences. your only option was to have fairly involved negotiations on a case-by-case basis. Many people I know grew exhausted from negotiating, and gave up on trying to visit their friends. The microCOVID tool gave people a simplified "risk budget", letting them do whatever activities made sense to them as long as they didn't overspend.

"Negotiation energy" was a limiting resource, and microcovid.org made negotiation radically cheaper. It also opened up entirely new options, like "create a household microCOVID tax" (some houses decided that you could do whatever activities you wanted, you just had to pay other housemates $1 per microcovid).

The proximate inspiration for the theme of this book (and included herein) are John Wentworth's posts "Coordination as Scarce Resource", "Transportation as Constraint", and "Interfaces as Scarce Resource." Other posts explore the nature of particular constraints that society faces - Zvi's posts on "Simulacra Levels and their Interactions," "The Road to Mazedom," and "Motive Ambiguity" each spell out how and why some communication is systematically distorted. And while they don't give us a solution, they help ask a question - what would need to change, in order for society to coordinate at scale, without incentivizing distorted communication?

COORDINATION & CONSTRAINT

John WentworthCoordination as a Scarce Resource John WentworthTransportation as a Constraint John WentworthInterfaces as a Scarce Resource Catherine OlssonMicroCOVID.org Jacob FalcovichSeeing the Smoke AlkjashPain is not the unit of Effort AlkjashIs Success the Enemy of Freedom? Zvi MowshowitzSimulacra Levels and their Interactions Zvi MowshowitzThe Road to Mazedom Zvi MowshowitzMotive Ambiguity Scott AlexanderStudies On Slack Jim Babcock, Elizabeth Van NostrandCredibility of the CDC on SARS-CoV-2 Raymond Arnold"Can you keep this confidential? How do you know?" Abram DemskiMost Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts ar...