Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Less Threat-Dependent Bargaining Solutions?? (3/2), published by Diffractor on August 20, 2022 on The AI Alignment Forum.

In the previous two posts, we went over various notions of bargaining. The Nash bargaining solution. The CoCo value. Shapley values. And eventually, we managed to show they were all special cases of each other. The rest of this post will assume you've read the previous two posts and have a good sense for what the CoCo value is doing.

Continuing from the last post, the games which determine the payoff everyone gets (not to be confused with the games that directly entail what actions are taken) are all of the form "everyone splits into two coalitions S and N/S, and both coalitions are trying to maximize "utility of my coalition - utility of the opposite coalition"".

Now, in toy cases involving two hot-dog sellers squabbling over whether to hawk their wares at a beach or an airport, this produces acceptable results. But, in richer environments, it's VERY important to note that adopting "let's go for a CoCo equilibria" as your rule for how to split gains amongst everyone incentivizes everyone to invent increasingly nasty ways to hurt their foes. Not to actually be used, mind you. To affect how negotiations go.

After all, if you invent the Cruciatus curse, then in all those hypothetical games where your coalition faces off against the foe coalition, and everyone's utility is "the utility of my coalition - the utility of the foe coalition"... well, you sure can reduce the utility of the foe coalition by a whole lot! And so, your team gets a much higher score in all those games.

Of course, these minimax games aren't actually played. They just determine everyone's payoffs. And so, you'd end up picking up a whole lot of extra money from everyone else, because you have the Cruciatus curse and everyone's scared of it so they give you money. In the special case of a two-player game, getting access to an option which you don't care about and the foe would pay $1000 to avoid, should let you demand a 500$ payment from the foe as a "please don't hurt me" payment.

But Which Desiderata Fails?

Step 1 in figuring out how to get an outcome which is Not That is to look at the list of nice properties which the CoCo solution uniquely fulfills, and figure out which one to break.

As it turns out from picking through the paper, the assumption that must be broken is the axiom of "gaining access to more actions shouldn't lead to you getting less value". As a quick intuitive way of seeing why it should fail, consider the game of Chicken. If you physically can't swerve (because your car started off not having a steering wheel), and are locked into always going straight through no fault of your own, then any sensible opponent will swerve and you will get good payoffs. Adding in the new option of being able to swerve means the opponent will start demanding that you swerve sometimes, lowering your score.

As a rough intuition for how the "the CoCo value is the only way to fulfill these axioms" proof works, it is reasoning as follows:

"Hm, let's say I only had access to my minimax action, that maximized minb(U1(a,b)−U2(a,b)). It makes me relatively much better off than my foe. Any sensible foe going up against this threat would simply press a button that'd just give us both the CoCo value, instead of playing any other action in response. By the axiom of "adding more options can't hurt me", I can add in all my other actions and not get less money. And then by the axiom of "adding redundant actions doesn't affect anything", I can take away the foe's CoCo value button and nothing changes. And so, in this game against the foe, I have to get a value equal or greater to my CoCo value. But the foe can run through this same reasoning from their side, and so in this game, we must both get th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Call off the EAs": Too Much Advertising?, published by tae on August 19, 2022 on The Effective Altruism Forum.

Today, my non-EA friend asked me "Hey, could you call off the EAs?". He heard an ad for What We Owe The Future in a podcast yesterday. "Nice," he thought; "I've heard of Will MacAskill." Then he heard another ad in a different podcast. And then he watched three YouTube videos in a row, each from different creators, that again included ads. Now he is genuinely annoyed.

This exact pattern showed up in the Carrick Flynn campaign. So many EA funds went into advertising that people got frustrated by how many ads they received.

There is such a thing as too much money in advertising! Oregon voters came to believe that Carrick was funded by bottomless money to be a crypto shill. And now smart people, who watch YouTube channels and listen to podcasts that are EA-adjacent, will rightfully get suspicious about who is putting this much money into promoting... a book about philosophy?Hope this anecdotal experience informs how we publicize/market/advertise in the future!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 3rd wave of EA is coming - what does it mean for you?, published by Jakob on August 19, 2022 on The Effective Altruism Forum.

In this post I'll first argue that we may look back at 2021-2022 as a time when EA entered a new phase (a "third wave" or "adulthood" could be appropriate terms). To be more specific, this phase shift would entail a sustained increase in money moved by the EA movement (say, by 2-5x or so? I haven't dug into the data enough to make a proper forecast), as well as the attention that people who don't identify as effective altruists pay to the movement.

We are already seeing some of this acceleration (with e.g., Open Philanthropy and GiveWell scaling up significantly in 2021, good growth for CEA in 2021, the FTX Future Fund launching in 2022 and attracting significant attention for donations to congressional candidate Carrick Flynn, record-breaking numbers of EA Global participants in 2022, and the recent publication of Will MacAskill's book What We Owe the Future, with associated media attention), so I don't expect this to be a controversial claim.

Then, I'll list some implications for 1) people who identify as (aspiring) effective altruists, 2) leaders in EA-aligned organizations, and 3) people with influence strategy and cultural norms at the movement level (a "movement leader"). In short, many effective altruists can take this moment of positive momentum to raise their ambitions; however, it is also a precarious moment of where the risk of values drift is elevated, so movement leaders should invest more than usually in steering the EA movement in the right direction.

Many of these thoughts have already been expressed more eloquently elsewhere; please consider this post my 5 cents chiming in - and for some, perhaps it can serve as a convenient summary.

A short history of EA

First, one may (somewhat simplified) consider the history of EA in roughly two waves: one may trace the origins of the movement to the first wave ("the infancy" of the movement), which I'll limit to pre-2011, and the second wave (the "youth phase" of the movement), which I'll consider to have lasted around a decade. I'll give some more details below, but since I only learned of EA in 2014 myself, I'm not the best to give a detailed account of the early days. The following paragraphs are a summary and an interpretation of the events described by CEA, Wikipedia, and various Forum articles on the history of EA, and so may be skipped by some readers (though they form some of the context for the argument further down).

During the first wave, organizations such as GiveWell (2007), Giving What We Can (2009) and 80000hours (2011) were launched. Peter Singer published his book The Life You Can Save (2009), and the rationalist community grew up around the Overcoming Bias/LessWrong blogs (2006). Individuals within these various communities and organizations sometimes interacted with each other, but there was no explicit unifying umbrella.

2011 marked a pivotal year, with the founding of the Centre for Effective Altruism (and the invention of the term effective altruist), and the launch of GiveWell labs, which later became Open Philanthropy. This is why I've used this as the pivotal year from wave 1 to wave 2.

During the following decades, the amount of funding in the EA movement increased substantially, and many new organizations and projects were started. Some projects took names that explicitly showed a link to the EA movement, such as EA Funds, EA Global (which started as the EA summit in 2013), and the EA Foundation, and local effective altruism groups all over the world. Others, Like Longview Philanthropy, Effective Giving, Charity Entrepreneurship, endorsed many of the same values and ideas, and were started by people affiliated with the movement, but did not put the link expl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does moral progress consist of?, published by jasoncrawford on August 19, 2022 on LessWrong.

There’s something that strikes me as odd about the way I hear “moral progress” discussed in the EA/longtermist community in general and in What We Owe the Future in particular.

The issues that are generally discussed under this heading are things like: animal welfare, existential risk, “coordination” mechanisms, and so forth.

However, when I look out at the world, I see a lot of problems rarely discussed among EAs. Most of the world is still deeply religious; in some communities and even whole nations, this pervades the culture and can even create oppression. Much of the world still lives under authoritarian or even totalitarian governments. Much of the world lives in countries that lack the institutional capacity to support even rudimentary economic development, leaving their people without electricity, clean water, etc.

And the conversation around the world’s problems leaves even more to be desired. A lot of discussion of environmental issues is lacking in scientific or technical understanding, treating nature like a god and industry like a sin—there’s even now a literal “degrowth” movement. Many young people in the US are now attracted to socialism, an ideology that should have been left behind in the 20th century. Others support Trump, an authoritarian demagogue with no qualifications to leadership. Both sides seem eager to tear down the institutions of liberalism, a “burn it all down” mentality. And all around me I see people more concerned with tribal affiliations than with truth-seeking.

So when I think about what moral progress the world needs, I mostly think it needs a defense of Enlightenment ideas such as reason and liberalism, so that these ideas can become the foundation for addressing the real problems and threats we face.

I think this might be highly relevant even to someone solely concerned with existential risk. For instance, if we want to make sure that an extinction-level weapon doesn’t get in the hands of a radical terrorist group, it would be good if there were fewer fundamentalist ideologies in the world, and no nation-states that sponsor them. More prosaically, if we want to have a good response to pandemics, it would be good to have competent leadership instead of the opposite (my understanding is that the US covid response could have been much better if we had just followed the pre-existing pandemic response plan). If we want to make sure civilization deals with climate change, it would be good to have a world that believed in technological solutions rather than being locked in a battle over “degrowth.” Etc.

Looking at it another way, we could think about two dimensions of moral progress, analogous to two dimensions of economic progress: pushing forward the frontier, vs. distribution of a best-practice standard. Zero-to-one progress vs. one-to-N progress. EA folks are very focused on pushing forward the moral frontier, breaking new moral ground—but I’m very worried about, well, let’s call it “moral inequality”: simple best practices like “allow freedom of speech,” “give women equality,” or even “use reason and science” are nowhere near universal.

These kind of concerns are what drew me to “progress studies” in the first place (before that term even existed). I see progress studies first and foremost as an intellectual defense of progress as such, and ultimately of the Enlightenment ideas that underlie it.

But I never hear EA folks talk about these kinds of issues, and these ideas don’t seem to resonate with the community when I bring them up. I’m still left wondering, what is the disconnect here?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing the Existential Risks Introductory Course (ERIC), published by Nandini Shiralkar on August 19, 2022 on The Effective Altruism Forum.

Crossposted to LessWrong.

Introduction

We (the Cambridge Existential Risks Initiative) ran an Existential Risks Introductory Course (ERIC) in the first quarter of 2022, aiming to introduce the field of existential risks, without being explicitly associated with any particular philosophy. We expect the programme to be most useful to people who are new to this field, and we hypothesised that we may be able to reach a different target audience by not explicitly branding it as EA.

The full curriculum we used for the programme, along with exercises and organisation spotlights, can be found here. This was primarily designed by Callum McDougall, with some inputs from the rest of the CERI team.

If you are interested in joining the next iteration of the course in Winter 2022 (either as a participant or as a facilitator), please fill out this interest form.

This post contains an overview of the course, which is followed by an abbreviated version of the syllabus for the ease of gathering feedback. The weekly summaries may also be helpful for community builders looking for summaries of any of the core readings from our syllabus.

We welcome any feedback on the content, exercises or anything else pertaining to the course, either here publicly on the Forum, or you can also reach out to us privately if you prefer that.

Course overview

The course consists of 8 weeks of reading (split into core and applied). Some weeks also include exercises, which participants are encouraged to complete and discuss in the session. Each week, participants will meet for 1.5 hour sessions where they will discuss the material and exercises with a facilitator.

The topics for each week are as follows:

Week 1: Introduction to Existential RisksProvides an introduction to x-risks, why they might be both highly important and neglected, and introduces some important terminology.

Week 2: Natural & Anthropogenic RisksDiscusses natural risks, and risks from nuclear war and climate change.

Week 3: Biosecurity, And How To Think About Future Risks Discusses risks from engineered pandemics, as well as a broader look at future risks in general and how we can reason about them and prepare for them.

Week 4: Unaligned Artificial IntelligenceDiscusses risks from unaligned AI, and provides a brief overview of the different approaches that are being taken to try and solve the problem.

Week 5: Dystopias, Lock-in & Unknown UnknownsConcludes the discussion of specific risks by discussing some more neglected risks. Also includes a discussion of the “unknown unknowns” problem, and how we can categorise and assess probabilities of risks.

Week 6: Forecasting & Decision-makingMoves away from specific risks, and discusses broad strategies that can help mitigate a variety of risks, with a focus on improving forecasting and decision-making (both at the institutional and individual level).

Week 7: Different Frameworks for Existential RiskFurther explores some alternative frameworks for x-risks than those found in The Precipice, e.g. FHI’s origin/scaling/endgame model, and the “Democratising Risk” paper.

Week 8: Next StepsConcludes the fellowship with a lookback on the key themes in the material, and a discussion of how the fellows plan to put what they’ve learned into action (e.g. in their future careers).

Abbreviated curriculum (Core readings)

Week 1: Introduction to Existential Risks

The first group of core materials here outlines the key ideas of Toby Ord’s book The Precipice, that we may be living in a uniquely important and dangerous time thanks to the threat of existential risks.

What are the most important moral problems of our time? (10 mins.)

The Precipice: Introduction, Chapters 1 & 2...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to do theoretical research, a personal perspective, published by Mark Xu on August 19, 2022 on The Effective Altruism Forum.

"Where do new [algorithms] come from? I keep reading about someone who invented [an algorithm] to do something-or-other but there's no mention of how."

A shrug of robed shoulders. "Where do new books come from, Mr. Potter? Those who read many books sometimes become able to write them in turn. How? No one knows."

"There are books on how to write -"

"Reading them will not make you a famous playwright. After all such advice is accounted for, what remains is mystery. The invention of new [algorithms] is a similar mystery of purer form."

- Harry Potter and the Methods of Rationality

Most of the content in this document came out of extensive conversations with Paul Christiano.

(This document describes one way of thinking about how to do one particular type of research. There are other ways to productively do this kind of research, and other productive kinds of research. Consider this document peppered with phrases like “from my perspective”, “I think”, “my sense is”, etc.)

A lot of people have a vague mental picture of what empirical research looks like, which consists of exploring data, articulating hypotheses about the data, and running experiments that potentially falsify the hypothesis. I think people lack a similarly mechanistic picture of what theoretical research looks like, which results in them not knowing how to do theory, being skeptical of the possibility of theoretical progress, etc. I do think the difficulty of getting high-quality real-world feedback makes theoretical research more difficult than empirical research, but I think it’s possible to get enough real-world feedback when doing theory that you can still expect to make steady progress.

Currently, I think many people think of theory as “someone sits in a room and has a brilliant insight to solve the problem.” Instead, I think a more accurate picture is very similar to the picture one has for empirical research: the theorist explores some data, gradually builds an intuitive sense of what’s going on, articulates hypotheses that capture their intuition, and falsifies their hypotheses by testing them against data, all the while iteratively building up their understanding.

The key difference between the empirical researcher and the theoretical researcher is while the empirical researcher can build intuition and falsify hypothesis by considering real-world data, the theorist, although ultimately be grounded in the real world, must build intuition and falsify hypothesis by considering thought experiments, simple toy examples, computations, etc. Even mathematics, the purest of intellectual pursuits, roughly follows this process of iteration. Terrance Tao:

[A]ctual solutions to a major problem tend to be arrived at by a process more like the following (often involving several mathematicians over a period of years or decades, with many of the intermediate steps described here being significant publishable papers in their own right):

Isolate a toy model case x of major problem X.

Solve model case x using method A.

Try using method A to solve the full problem X.

This does not succeed, but method A can be extended to handle a few more model cases of X, such as x’ and x”.

Eventually, it is realised that method A relies crucially on a property P being true; this property is known for x, x’, and x”, thus explaining the current progress so far.

Conjecture that P is true for all instances of problem X.

Discover a family of counterexamples y, y’, y”, . to this conjecture. This shows that either method A has to be adapted to avoid reliance on P, or that a new method is needed.

Take the simplest counterexample y in this family, and try to prove X for this special case. Meanwhile, try to see wheth...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to do theoretical research, a personal perspective, published by Mark Xu on August 19, 2022 on The AI Alignment Forum.

"Where do new [algorithms] come from? I keep reading about someone who invented [an algorithm] to do something-or-other but there's no mention of how."

A shrug of robed shoulders. "Where do new books come from, Mr. Potter? Those who read many books sometimes become able to write them in turn. How? No one knows."

"There are books on how to write -"

"Reading them will not make you a famous playwright. After all such advice is accounted for, what remains is mystery. The invention of new [algorithms] is a similar mystery of purer form."

- Harry Potter and the Methods of Rationality

Most of the content in this document came out of extensive conversations with Paul Christiano.

(This document describes one way of thinking about how to do one particular type of research. There are other ways to productively do this kind of research, and other productive kinds of research. Consider this document peppered with phrases like “from my perspective”, “I think”, “my sense is”, etc.)

A lot of people have a vague mental picture of what empirical research looks like, which consists of exploring data, articulating hypotheses about the data, and running experiments that potentially falsify the hypothesis. I think people lack a similarly mechanistic picture of what theoretical research looks like, which results in them not knowing how to do theory, being skeptical of the possibility of theoretical progress, etc. I do think the difficulty of getting high-quality real-world feedback makes theoretical research more difficult than empirical research, but I think it’s possible to get enough real-world feedback when doing theory that you can still expect to make steady progress.

Currently, I think many people think of theory as “someone sits in a room and has a brilliant insight to solve the problem.” Instead, I think a more accurate picture is very similar to the picture one has for empirical research: the theorist explores some data, gradually builds an intuitive sense of what’s going on, articulates hypotheses that capture their intuition, and falsifies their hypotheses by testing them against data, all the while iteratively building up their understanding.

The key difference between the empirical researcher and the theoretical researcher is while the empirical researcher can build intuition and falsify hypothesis by considering real-world data, the theorist, although ultimately be grounded in the real world, must build intuition and falsify hypothesis by considering thought experiments, simple toy examples, computations, etc. Even mathematics, the purest of intellectual pursuits, roughly follows this process of iteration. Terence Tao:

[A]ctual solutions to a major problem tend to be arrived at by a process more like the following (often involving several mathematicians over a period of years or decades, with many of the intermediate steps described here being significant publishable papers in their own right):

Isolate a toy model case x of major problem X.

Solve model case x using method A.

Try using method A to solve the full problem X.

This does not succeed, but method A can be extended to handle a few more model cases of X, such as x’ and x”.

Eventually, it is realised that method A relies crucially on a property P being true; this property is known for x, x’, and x”, thus explaining the current progress so far.

Conjecture that P is true for all instances of problem X.

Discover a family of counterexamples y, y’, y”, . to this conjecture. This shows that either method A has to be adapted to avoid reliance on P, or that a new method is needed.

Take the simplest counterexample y in this family, and try to prove X for this special case. Meanwhile, try to see whether meth...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Cause Exploration Prizes] More Animal Advocacy R&D, published by Open Philanthropy on August 19, 2022 on The Effective Altruism Forum.

This anonymous essay was submitted to Open Philanthropy's Cause Exploration Prizes contest and posted to the Forum with the author's permission.

If you're seeing this in summer 2022, we'll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!

Summary

Of the approximate $200 million in funding aimed at improving the lives of farmed animals, only about 3% is spent on research and measurement and evaluation. This seems remarkably low, given how little the animal advocacy movement knows about which interventions are actually effective in changing behaviours, influencing policy or affecting animal product production. My very naive BOTEC puts the leverage factor of improved research at about 12x, meaning I believe $2 million of animal advocacy R&D could influence $23 million (2% of the next five years of funding).

Funding intervention research early-on can improve the trajectory of the animal advocacy movement, and counterfactually improve the number of animals we help over the long-term. Not only does it provide a large value of information, I believe it is fairly tractable. For example, one can:

Increase the pipeline of talented researchers by funding summer research fellowships

Support existing academic research groups by academic network building, providing administrative support and hiring, as done by BERI.

Release requests for proposals on large-scale and neglected advocacy methods.

Incubate a new charity to support animal advocacy charities in conducting measurement and evaluation on their interventions. This organisation could also start conducting highly empirical cost-effectiveness analyses of various interventions and charities a la Givewell.

Despite this, I have some large uncertainties on the cost-effectiveness of the last animal dollar, the popularity of the Food System Research Fund amongst researchers, and how easy it is to influence funding.

Importance

There are approximately 70 billion land animals and up to 2.3 trillion wild fish killed for food each year. Most land animals are chickens, who experience high levels of suffering throughout their lives whether they’re an egg-laying hen or a broiler used for meat. Some estimates believe that 74% of all farmed land animals are in factory farms, with this number being even higher for farmed fish. In addition, the number of animals killed each year for food is growing, driven primarily by an increase in chicken numbers and increasing wealth across the world.

The farmed animal welfare (FAW) movement is trying to reduce the suffering of farmed animals, and support interventions that either improve the welfare of farmed animals or reduce the number of animals being warmed. Estimates by Farmed Animal Funders (FAF) puts the farmed animal movement at a size of approximately $200 million per year of funding, which has been steadily growing for the past few years. Despite $160 million of this being spent in the UK, Western Europe and the US, meat consumption for these three countries and regions still seems to be stable or still rising. However, there has been considerable success in corporate welfare reforms, with over 2,400 commitments since 2010.

This juxtaposition in relative success in improving welfare but failing to affect total meat and dairy consumption potentially points to some particular difficulties in improving farmed animal welfare. Specifically, we have very little evidence on most animal interventions, specifically around ones that affect consumer preferences or non-corporate welfare institutional asks, e.g. the impact of media, different messaging strategies, humane education or grassroots move...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Epistemic Artefacts of (conceptual) AI alignment research, published by Nora Ammann on August 19, 2022 on The AI Alignment Forum.

Tl;dr

In this post, I describe four types of insights - what I will call Epistemic Artefacts - that we may hope to acquire through (conceptual) AI alignment research. I provide examples and briefly discuss how they relate to each other and what role they play on the path to solving the AI alignment problem. The hope is to add some useful vocabulary and reflective clarity when thinking about what it may look like to contribute to solving AI alignment.

Four Types of Epistemic Artefacts

Insofar as we expect conceptual AI alignment research to be helpful, what sorts of insights (here: “epistemic artefacts”) do we hope to gain?

In short, I suggest the following taxonomy of potential epistemic artefacts:

Map-making (de-confusion, gears-level models, etc.)

Characterising risk scenarios

Characterising target behaviour

Developing alignment proposals

(1) Map-making (i.e. conceptual de-confusion, gears-level understanding of relevant phenomena, etc.)

First, research can aim to develop a gears-level understanding of phenomena that appear critical for properly understanding the problem as well as for formulating solutions to AI alignment (e.g. intelligence, agency, values/preferences/intents, self-awareness, power-seeking, etc.). Turns out, it’s hard to think clearly about AI alignment without having a good understanding of and “good vocabulary” for phenomena that lie at the heart of the problem. In other words, the goal of "map-making" is to dissolve conceptual bottlenecks holding back progress in AI alignment research at a the moment.

Figuratively speaking, this is where we are trying to draw more accurate maps that help us better navigate the territory.

Some examples of work on this type of epistemic artefact include Agency: What it is and why it matters, Embedded Agency, What is bounded rationality?, The ground of optimization, Game Theory, Mathematical Theory of Communication, Functional Decision Theory and Infra-Bayesianism—among many others.

(2) Identifying and specifying risk scenarios

We can further seek to identify (new) civilizational risk scenarios brought about by advanced AI and to better understand the mechanisms leading to risk scenarios.

Figuratively speaking, this is where we try to identify and describe the monsters hiding in the territory, so we can circumvent them when navigating the territory.

Why does a better understanding of risk scenarios represent useful progress towards AI alignment? In principle, one way of guaranteeing a safe future is by identifying every way things could go wrong and finding ways to defend against each of them. (We could call this a “via negativa” approach to AI alignment.) I am not actually advocating for adopting this approach literally, but it still provides a good intuition for why understanding the range of risk scenarios and their drivers/mechanisms is useful. (More thoughts on this later, in "How these artefacts relate to each other".)

In identifying and understanding risk scenarios, like in many other epistemic undertakings, we should seek to apply a diverse set of epistemic perspectives on how the world works in order to gain a more accurate, nuanced, and robust understanding of risks and failure stories and avoid falling prey to blind spots.

Some examples of work on this type of epistemic artefact include What failure looks like, What Multipolar Failure Looks Like, The Parable of Predict-O-Matic, The Causes of Power-seeking and Instrumental Convergence, Risks from Learned Optimization, Thoughts on Human Models, Paperclip Maximizer, and Distinguishing AI takeover scenarios—among many others.

(3) Characterising target behaviour

Thirdly, we want to identify and characterize, from within the space ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discovering Agents, published by zac kenton on August 18, 2022 on LessWrong.

Work done with Ramana Kumar, Sebastian Farquhar (Oxford), Jonathan Richens, Matt MacDermott (Imperial) and Tom Everitt.

Our DeepMind Alignment team researches ways to avoid AGI systems that knowingly act against the wishes of their designers. We’re particularly concerned about agents which may be pursuing a goal that is not what their designers want.

These types of safety concerns motivate developing a formal theory of agents to facilitate our understanding of their properties and avoid designs that pose a safety risk. Causal influence diagrams (CIDs) aim to be a unified theory of how design decisions create incentives that shape agent behaviour to illuminate potential risks before an agent is trained and inspire better agent designs with more appealing alignment properties.

Our new paper, Discovering Agents, introduces new ways of tackling these issues, including:

The first formal causal definition of agents, roughly: Agents are systems that would adapt their policy if their actions influenced the world in a different way

An algorithm for discovering agents from empirical data

A translation between causal models and CIDs

Resolving earlier confusions from incorrect causal modelling of agents

Combined, these results provide an extra layer of assurance that a modelling mistake hasn’t been made, which means that CIDs can be used to analyse an agent’s incentives and safety properties with greater confidence.

Example: modelling a mouse as an agent

To help illustrate our method, consider the following example consisting of a world containing three squares, with a mouse starting in the middle square choosing to go left or right, getting to its next position and then potentially getting some cheese. The floor is icy, so the mouse might slip. Sometimes the cheese is on the right, but sometimes on the left.

This can be represented by the following CID:

The intuition that the mouse would choose a different behaviour for different environment settings (iciness, cheese distribution) can be captured by a mechanised causal graph (a variant of mechanised causal game graph), which for each (object-level) variable, also includes a mechanism variable that governs how the variable depends on its parents. Crucially, we allow for links between mechanism variables.

This graph contains additional mechanism nodes in black, representing the mouse's policy and the iciness and cheese distribution.

Edges between mechanisms represent direct causal influence. The blue edges are special terminal edges – roughly, mechanism edges ~A ~B that would still be there, even if the object-level variable A was altered so that it had no outgoing edges.

In the example above, since U has no children, its mechanism edge must be terminal. But the mechanism edge ~X ~D is not terminal, because if we cut X off from its child Uthen the mouse will no longer adapt its decision (because its position won’t affect whether it gets the cheese).

Causal definition of agents

We build on Dennet’s intentional stance – that agents are systems whose outputs are moved by reasons. The reason that an agent chooses a particular action is that it expects it to lead to a certain desirable outcome. Such systems would act differently if they knew that the world worked differently, which suggests the following informal characterisation of agents:

Agents are systems that would adapt their policy if their actions influenced the world in a different way.

The mouse in the example above is an agent because it will adapt its policy if it knows that the ice has become more slippery, or if the cheese is more likely on the left. In contrast, the output of non-agentic systems might accidentally be optimal for producing a certain outcome, but these do not typically adapt. F...