Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transcript of a Twitter Discussion on EA from June 2022, published by Zvi on June 6, 2022 on LessWrong.

Recently on Twitter, in response to seeing a contest announcement asking for criticism of EA, I offered some criticism of that contest’s announcement.

That sparked a bunch of discussion about central concepts in Effective Altruism. Those discussions ended up including Dustin Moskovitz, who showed an excellent willingness to engage and make clear how his models worked. The whole thing seems valuable enough to preserve in a form that one can navigate, hence this post.

This compiles what I consider the most important and interesting parts of that discussion into post form, so it can be more easily seen and referenced, including in the medium-to-long term.

There are a lot of offshoots and threads involved, so I’m using some editorial discretion to organize and filter.

To create as even-handed and useful a resource as possible, I am intentionally not going to interject commentary into the conversation here beyond the bare minimum.

As usual, I use screenshots for most tweets to guard against potential future deletions or suspensions, with links to key points in the threads.

(As Kevin says, I did indeed mean should there.)

At this point there are two important threads that follow, and one additional reply of note.

Thread one, which got a bit tangled at the beginning but makes sense as one thread:

Thread two, which took place the next day and went in a different direction.

Link here to Ben’s post, GiveWell and the problem of partial funding.

Link to GiveWell blog post on giving now versus later.

Dustin’s “NO WE ARE FAILING” point seemed important so I highlighted it.

There was also a reply from Eliezer.

And this on pandemics in particular.

Sarah asked about the general failure to convince Dustin’s friends.

These two notes branch off of Ben’s comment that covers-all-of-EA didn’t make sense.

Ben also disagreed with the math that there was lots of opportunity, linking to his post A Drowning Child is Hard to Find.

This thread responds to Dustin’s claim that you need to know details about the upgrade to the laptop further up the main thread, I found it worthwhile but did not include it directly for reasons of length.

This came in response to Dustin’s challenge on whether info was 10x better.

After the main part of thread two, there was a different discussion about pressures perhaps being placed on students to be performative, which I found interesting but am not including for length.

This response to the original Tweet is worth noting as well.

Again, thanks to everyone involved and sorry if I missed your contribution.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Estimated $4 – $7 Billion in Matching Gift Funds Goes Unclaimed Per Year, published by Tyler Smith on June 5, 2022 on The Effective Altruism Forum.

There are billions of free donation dollars that are missed every year through employees that don’t take advantage of their matching benefits. I work at a Fortune 500 company in the U.S. that offers 50% matching for up to $2000 worth of donations ($1000 free dollars) every year. They even allow me to donate to the Long-Term Future EA Fund. There are 50,000 employees at my company and I’m confident most of them do not take advantage of this benefit.

In individual cases you can give money to your coworkers to donate. I’ve already spoken to one of my friends at work who is fine doing this. They may even be able to benefit from a tax write off (I’m not sure the legality of this). If you can individually convince a few friends at work to do this, that is a couple thousand extra dollars a year that can be donated.

Larger scale we could create an organization that takes advantage of these missed opportunities. For example, there are already companies that front you money so that you can take advantage of your employee stock purchase program. Why couldn’t there be a similar organization that allows individuals to fully take advantage of their donation matching benefits. Further, our organization could even pay people to donate. If a company is offering a 50% match up to $2000 it would even make sense to incentivize individuals with an additional $100 (or more) if they would just take the steps of receiving money from us and donating it to an EA approved organization.

To clarify, if a person was working at the company I do and didn’t previously donate we could send them $2100. They could keep $100 and donate the other $2000 to an EA approved organization. That would give the organization an additional $900 that they otherwise would not have received.

An estimated $4 – $7 billion in matching gift funds goes unclaimed per year.

65% of Fortune 500 companies offer matching gift programs.

Over 26 million individuals work for companies with matching gift programs.

An estimated $2 – $3 billion is donated through matching gift programs annually.

Over the last three years, the percentage of Russell 1000 companies publicly disclosing that they offer employee matching gifts grew by 11.8%.

Some questions and caveats that I have:

How would we confirm that they donated the money?

What is the legality of doing this?

Companies will start to decrease their matching benefits before we could reach the $4 - $7 billion limit.

If the company finds out that you are individually doing this with your colleagues it could hurt your reputation.

Has this idea already been thought of?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New cause area: Violence against women and girls, published by Akhil on June 6, 2022 on The Effective Altruism Forum.

Summary

Globally, one third of women and girls experience violence during their lifetime

Violence against women and girls (VAWG) has significant health, economic and social effects for survivors, perpetrators and society more broadly.

There is moderate and growing evidence that interventions that seek to prevent VAWG from occurring are cost-effective

VAWG as a cause area has moderate absorbency for more funding, with a number of potentially promising avenues for funding

There are several key uncertainties that were identified in this report, concerning the scalability of interventions in this area, how interventions would be received by communities and governments, and the comparative and additional benefit of directly addressing VAWG versus its risk factors.

Overall, this report recommends that VAWG be considered as a new cause area.

Disclaimer: This report represents a shallow dive into this cause area; approximately 20 hours was spent on this report. With more time, several of the key uncertainties presented through the article might be addressed, and more time would be spent analysing potential opportunities for new organisations and funders in this space. However, I would anticipate that this is unlikely to significantly change how promising VAWG is as a cause area

How big a problem is violence against women and girls?

Nearly one third of women and girls aged 15 years of age or older have experienced either physical or sexual intimate partner violence (IPV) or non-partner sexual violence globally, with 13% (10–16%) experiencing it in 2018 alone (Sardinha et al 2022). This figure does not include a number of areas, including sexual harassment, female genital mutilation, trafficking in women or cyber-harassment (UN Women). It is also likely that this figure, due to chronic under-reporting, underestimates the true burden of VAWG,

The Global Burden of Disease, which includes information on IPV but not VAWG more generally, reports it as the 19th leading burden of disease globally- it is responsible for 8.5 million DALYs and 68 500 deaths annually. In several countries, violence against women is in top 3-5 leading causes of death for young women aged between 15 and 29 (Mendoza et al 2018). In addition to the direct harms of VAWG, it is a significant risk factor for other conditions- VAWG is responsible for 11% of the DALY burden of depressive disorders and 14% of the DALY burden of HIV in women (Healthdata.org).

Globally, the rates of VAWG are both alarmingly high and have slightly increased over the last 30 years, despite gains in other areas of women’s health, such as maternal care (Think Global Health). Further, there are certain countries and regions of the world (e.g. several Asian countries) that have seen significant increases in the rates of VAWG over the last two decides (Borumandnia et al 2020). The geographical distribution is shown in the figure below, which takes data from the 2018 WHO Global Database and which was published in The Lancet earlier in 2022 (Sardinha et al 2022)

What effects do violence against women and girls have?

Health costs of violence against women

The health consequences of VAWG are significant; they can be immediate and acute, long-lasting and chronic, and/or fatal. Some of the most common health consequences of VAWG are summarised in the table below.

PhysicalSexual and reproductiveMentalBehavioural

-Acute or immediate physical injuries, including bruises, lacerations, burns, bites, and fractures

- more serious injuries leading to long term disability, including injuries to head, eyes, ears, chest and abdomen

-Gastrointestinal conditions

- chronic pain conditions

-death, including femicide

-unintended/unwanted pregnan...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reading the ethicists 2: Hunting for AI alignment papers, published by Charlie Steiner on June 6, 2022 on The AI Alignment Forum.

Introduction

I'm back, reading more papers in ethics (but now also in philosophy and wherever the citations lead me). Unlike last time when I surveyed one journal's approach to AI in general, this time I'm searching specifically for interesting papers that bear on aligning AGI, preferably written by people I've never heard of before. Are they going to be any good? shrug

To this end, I skimmed the titles and sometimes abstracts of the last 5 years of papers in a pretty big chunk of the AI ethics space, namely:

Ethics and Information Technology

Minds and Machines

AI and Ethics

Philosophy & Technology

Science and Engineering Ethics

AI & Society

IEEE Transactions on Philosophy and Society

And a few bonus miscellanea

From the set of all papers that even had a remote chance at being relevant (~10 per journal per year), I read more deeply and am relaying to you in this post all the ones that were somewhat on-topic and nontrivial (~0.5 per journal per year). By "nontrivial" I mean that I didn't include papers that just say "the alignment problem exists" - I certainly do not mean that I set a high bar for quality. Then I did some additional searching into what else those authors had published, who they worked with, etc.

What were all the other papers about, the ones that didn't match my criteria? All sorts of things! Whether robots are responsible for their actions, how important privacy is, how to encourage and learn from non-Western robotics paradigms, the ethics of playing MMORPGs, how to make your AI ignore protected characteristics, the impact of bow-hunting technology on middle stone age society, and so on and so forth.

The bulk of this post is a big barely-ordered list of the papers. For each paper I'll give the title, authors, author affiliations, journal and date. Each paper will get a summary and maybe a recommendation. I'll also bring up interesting author affiliations and related papers.

Whereas last post was more a "I did this so you don't have to" kind of deal, I think this post will be more fun if you explore more deeply, reading the papers that catch your interest. (Let me know if you have any trouble finding certain pdfs - most are linked on google scholar.) If you find this post to be too long, don't go through it all in one sitting - I sure didn't!

Papers

The possibility of deliberate norm-adherence in AI, Danielle Swanepoel (U. Johannesburg, SolBridge International School of Business), Ethics and Information Technology, 2021.

I binned most papers talking about artificial moral agents ("AMAs") for being dull and anthropocentric. I decided to include this paper because it's better than most, and its stab at drawing the line between "not moral agent" and "moral agent" is also a good line between "just needs to be reliable" and "actually needs to be value aligned." Recommended if you like people who really like Kant.

Human-aligned artificial intelligence is a multiobjective problem, P. Vamplew (LessWrong), R. Dazeley, C. Foale, S. Firmin and J. Mummery (Federation University Australia), Ethics and Information Technology, 2018.

They argue that if you design an AI's motivations by giving it multiple objective functions that only get aggregated near the point of decision-making, you can do things like using a nonlinear aggregation function as a form of reduced-impact AI, or throwing away outliers for increased robustness. Recommended if you haven't thought about this idea yet and want something to skim while you turn on your brain (also see Peter Vamplew's more technical papers if you want more details).

This has been cited a healthy amount, and by some interesting-looking papers, including:

MORAL: Aligning AI with Human Norms through ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on impostor syndrome, published by Lizka on June 6, 2022 on The Effective Altruism Forum.

This post contains the following things:

A description of my experience with impostor syndrome

One of my favorite ways of fighting back: get a ton of honest feedback

A list of some other things that help me

The bulldozer strategy: Ignore the impostor syndrome and do the things

Seriously think through the worst-case scenario (within the realm of plausibility)

View looking silly as charity unto others

Ask for sanity checks

Talk to a friend or mentor

Physical health

Consider getting professional help

Remind yourself that comparing yourself to other people is not necessary and that you might be caught in a false framework

Assorted other things

Ending notes and some resources

Notes:

I have no relevant expertise or credentials. I am discussing impostor phenomenon-y things somewhat indiscriminately, so some of this might be more relevant to being generally anxious.

Context: I wrote this over half a year ago — before I started working on the Forum, before Luisa’s awesome post on a similar subject came out — and then sat on the post without publishing it. I’m finally posting it, but I haven’t significantly edited this since Luisa’s post, so some of the content might be redundant. Both our posts discuss what we’ve found helpful in combatting impostor syndrome. Luisa’s post focuses more on the real harms of impostor syndrome on your career and impact specifically.

What if someone told you:

I think I’m lazy, stupid, self-absorbed, and useless. Everyone around me is incredibly impressive; they have tons of ideas on how to improve the world, they’re brilliant— they can create systems that model complex structures, they have so much energy, they’re selfless and ambitious and awesome. I don’t know what I’m doing in the same place as these other people. I don’t know how I ended up here, and I’m just waiting for them to realize that I don’t fit.

But you know that the person saying this speaks three languages, graduated from a great university with two majors and a good GPA, got a competitive summer internship and did reasonably well in it, and quickly got hired into a respectable role.

You might say to this person, “You’ve got impostor syndrome!”

You’d probably be right. Moreover, the person you’d be saying that to is me, and I know I’ve got it. I also know that the whole situation is ridiculous.

Suffering from impostor syndrome is like having the awful feeling that something is on your face and that this something will reveal the dark truth about your soul to the world just as soon as the world sees it. This feeling does not allow itself to be wiped away with rational arguments. A friend coming up to you and telling you that you have nothing on your face just makes you feel like they’re lying to you, or that they haven’t seen it yet because they’re looking in the wrong place. The feeling is pervasive and has an answer to everything. (See the appendix-comment for how this feeling actually works.)

So what’s happening? Why doesn’t someone telling me you’re actually quite smart— you’ve just got impostor syndrome fix all my issues?

I'm new at my job, and I try to ask for lots of feedback. Every time I try to evaluate my work, I almost burst from anxiety. Here's what happens in my head, where two pretty distinct voices develop.

[Part 1]

Pragmatic Voice: Well, I’ll estimate how well I did: what went wrong, what went well, where I helped, and the like.

A model of the situation forms; I messed up in two places and delayed one task, but that seems pretty much as expected. I also did a cool new thing that I'm excited about and finished a big project.

Pragmatic Voice concludes: All in all, a pretty good week.

[Part 2]

Alarm bells start ringing in my head. Voice 2 wakes up.

Panic Voice: The words "pret...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some ideas for follow-up projects to Redwood Research’s recent paper, published by JanBrauner on June 6, 2022 on The AI Alignment Forum.

Disclaimer: I originally wrote this list for myself and then decided it might be worth sharing. Very unpolished, not worth reading for most people. Some (many?) of these ideas were suggested in the paper already, I don’t make any claims of novelty.

I am super excited by Redwood Research’s new paper, and especially by the problem of “high-stakes reliability”.

Others have explained much better why the problem of “high-stakes reliability” is a worthy one (LINK, LINK), so I’ll keep my opinion. In short, I think:

Solving “high-stakes reliability” is probably essential for building safe AGI.

You can do useful empirical research on it now already.

High-stakes reliability would be super useful for today's systems and applications; for systems that make high-stakes decisions where a single instance of a certain bad behaviour would be terrible (e.g. confidently but incorrectly classifying a medical image)

I think there is little existing work on exactly this problem, but I might be wrong about this. There is definitely a lot of work on related problems.

Here is an extremely unpolished of ideas and directions for follow-up research that I had after reading the paper, in no particular order. Some (many?) of these ideas were suggested in the paper already, I don’t make any claims of novelty.

How does this methodology fail?

Find (ideally systematic) ways to break their classifier (I tried a bit, it is actually harder than expected :-) )

How exactly are capabilities impacted by the filtering?

From Paul Christiano:

Avoiding failures where the model learns to be conservative only on “sneaky-looking prompts designed to trick me.”

Improve the classifier or its training

use uncertainty in the classifier

(if necessary via expensive ensembles)

something something out-of-distribution detection

experiment with prompting

Do a project on high-stakes reliability that does not use human oversight, in order to focus more on the ML side of things

Example project: Get a model to solve easy programming questions. The model is allowed to not submit any code to a question, and it's fine if the model doesn't submit code to most questions. However, if the model submits code, the code should solve the test cases in 99.9X% of cases, for any input (from a natural distribution).

If you work with tasks that require human oversight (like "does this snippet entail injuries?"), you learn many things, e.g. how to enable humans to find adversarial inputs. But there are also parts of high-stakes reliability that don't require you to work with humans. Getting a model to almost never do bad behaviour X, on any (natural/non-gibberish?) input, while not losing much average performance, is a challenging project even if we can automatically evaluate model outputs.

Do a project on high-stakes reliability that uses real-world high-stakes tasks

I think this could be useful for field-building, making high-stakes reliability a subfield of mainstream ML robustness

E.g. in medical image classification:

The classifier has the option to say "I don't know" (IDK)

It's fine if the classifier says "IDK" pretty often, e.g. 50% of the time. We'll just have a human look at these examples.

But whenever the classifier does predict a class other than "IDK", I require extremely high accuracy (let's say >99.95%).

This accuracy must be high for samples from any part of the "natural" distribution (e.g. medical images from various hospitals/scanners/populations).

Ideally, it would even be robust to adversarial attacks, but that is probably too hard

Alternatively, if the above is too challenging, we could change the setting such that the classifier only has to choose between "healthy" and "IDK".

Seems like su...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four Concerns Regarding Longtermism, published by Pat Andriola on June 6, 2022 on The Effective Altruism Forum.

I'm submitting the below as part of the Red Teaming contest. Any prize money won will be donated to the non-profit I founded (the Rikers Debate Project).

I think it makes a bit of sense to start with who I am for this post, because I'm hoping to provide a bit of a curious outsider's perspective on EA in the hopes of offering a somewhat unique perspective.

I consider myself “EA adjacent.” I have a number of good friends (Josh Morrison, Jay Shooster, Alex Silverstein) who are varying parts of the movement (then again, because I’m only adjacent I have no idea if any of you will know these names). I’m a rationalist by nature, did high-level college debate with decent success, and was part of the Moneyball-era baseball analytics movement (so I’m good with the intersection of numbers and logic). I think all of those qualities give me a proclivity toward EA. These days I’m a rather boring complex commercial litigator who does a good amount of pro bono work. My greatest civic contribution is founding (with Josh Morrison and others) the Rikers Debate Project, which I consider to be a rather typical, not especially EA-y non-profit.I’m writing this because I think I’ve absorbed enough about EA via osmosis to provide a halfway intelligible critique.

To state my lack of bona fides: I sometimes read EA stuff accidentally, but neither typically or intentionally. I had to make an account for this forum to post this. I've never heard Will MacAskill speak before, but I have had multiple people tell me about him. I read Peter Singer before I associated him with EA. I am generally sympathetic to and supportive of the positions of the EA movement. I mean, who can oppose ambitious people coming together to provide efficacious good for the world? It sounds wonderful. In both theory and in practice, I care about the movement (from a distance) and hope it succeeds.The devil is in the details, as always. I provide the following critique basically in part because I feel guilty that others have done far more to try to help the world than I have, and the least I can do is provide my thoughts to help the movement get better. I caution that the below is based on an outsider's second-hand perspective. Therefore, it may/will get details wrong.

But my hope is that the spirit is right.I have heard that there is a focus in EA lately on longterm problems. You can read a bit about longtermism in Simon Bazelon’s post here. Intuitively, longtermism makes sense. EA, as a collective movement, has a finite amount of resources in the present day, yet it is uniquely (literally) temporally positioned. Therefore, as a % of all resources from hereon out, the current amount of resources should be used to take advantage of the unique temporal position to maximize future utility returns. To invert the hypo a bit—if the movement could go back in time, then using the money on anything else but, say, ending slavery or stopping the Holocaust (or, to be more EA about it, handing out vaccines/cures for the Bubonic plague) would be not just foolhardy, but ethically disastrous. Therefore, we must focus on fat-tail future risks that threaten future life.I like this thought a lot. I think it makes sense.

I write to provide four discrete (but at times overlapping) concerns/cautions w/r/t longtermism. At most, the implication of these criticisms is that there may be a current overcorrection toward longtermism that should be somewhat corrected back in the direction of the prior distribution. This doesn’t mean we stop focusing on longterm risks (far from it), but simply that we recalibrate the risk utility curve, and potentially allocate more resources toward present-ish causes. At the least, I think these criticisms sho...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New cooperation mechanism - quadratic funding without a matching pool, published by Filip Sondej on June 5, 2022 on The Effective Altruism Forum.

cross-posted to ethresear.ch and LessWrong

Motivation

Quadratic funding[1] is a powerful mechanism for resolving some collective action problems. But it has a major limitation - it relies on some third party, that provides a matching pool of funds.

In the most dangerous collective action problems, we don't have such third party helping us from above. Those situations already involve the most powerful actors, so we can't expect someone more powerful to resolve the conflict, like a galactic mom.

Some examples:

global superpowers trying to coordinate to fight climate change

AI organisations coordinating to pay AI alignment tax (more info here)

for example by funding safety research

or creating some large dataset together, that's useful for alignment

or funding methods which are thought to be safer, like STEM AI or tool AI

in general, escaping inadequate equilibria (see this post for many great examples)

and most importantly, conflict between transformative AI systems or their owners[2]

Solution

One thing we can try in this situation, is to create a smart contract where each party says "I'll pay more if others pay more". This way, if you decide to increase your contribution by 1$, it causes the pot to grow by more than 1$, because your dollar caused other agents to contribute some more. This leverage, in some situations can be enough to make someone pay, because the value they get out of the bigger pot is higher than what they have to pay.

Some properties that it would be nice to have in such a system are:

continuity - every increase in your payment causes an increase in others' payments

known payment limit - you won't have to pay more than some limit you chose

everyone is incentivised to contribute something - just like in quadratic funding, small contributions get a high leverage (it can get arbitrarily high, as you'll see later) - so even if you're only willing to pay if you get >100x leverage, there is always some contribution size that gives you such a high leverage

A very simple system that has these properties is given by those equations:

h=∑i√paymenti(h) paymenti(h)=limitiπ2arctan(h∗saturation_speedi)

paymenti(h) is the amount that i'th agent has to pay

limiti is the i'th agent payment limit

saturation_speedi tells how quickly i'th agent's limit will be approached as new agents make contributions (the choice of this parameter is underspecified for now, and is discussed in Parameter choice section)

given all those parameters, we find h that satisfies the two equations above

It turns out, this system has a pretty graphical representation:

Each quarter-circle represents one agent's contribution. Area of a quarter-circle is the payment limit - the maximum amount this agent can pay. The yellow areas are what they currently pay in this particular situation. The squares on the right have the same areas as the respective sectors. So the height of the tower of squares represents h - the sum of square roots of payments. The distance of a quarter-circle's center to the right corner is 1saturation_speed - for small saturation_speed, the quarter-circle is put further to the left and you can see that they saturate more slowly.

The animation shows the procedure for finding the solution to those two equations. We start with some arbitrary h, then compute the payments (yellow sectors), then compute h, recompute payments, recompute h, and so on, until we converge on the stable solution.

On the next animation, you see what happens when someone new joins the smart contract. Their contribution increases h, which makes others pay more. (Here the procedure of finding the solutions is ommited, and just the final solutions are shown).

Here you ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Terminate deliberation based on resilience, not certainty, published by Gregory Lewis on June 5, 2022 on The Effective Altruism Forum.

BLUF:

“We should ponder until it no longer feels right to ponder, and then to choose one of the acts it feels most right to choose. If pondering comes at a cost, we should ponder only if it seems we will be able to separate better options from worse options quickly enough to warrant the pondering” - Trammell

Introduction

Many choices are both highly uncertain, highly consequential, and difficult to develop a highly-confident impression of which option is best. In EA-land, the archetypal example is career choice, but similar dilemmas are common in corporate decision-making (e.g. grant-making, organisational strategy) and life generally (e.g. “Should I marry Alice?” “Should I have children with Bob?”).

Thinking carefully about these choices is wise, and my impression is people tend to err in the direction of too little contemplation - stumbling over important thresholds that set the stage for the rest of their lives. One of the (perhaps the main) key messages of effective altruism is our altruistic efforts err in a similar direction. Pace more cynical explanations, I think (e.g.) the typical charitable donor (or typical person) is “using reason to try and do good”. Yet they use reason too little and decide too soon whilst the ‘returns to (further) reason’ remain high, and so typically fall far short of what they could have accomplished.

One can still have too much of a good thing: ‘reasonableness’ ‘prudence’ or even ‘caution’ can be wise, but ‘indecisiveness’ less so. I suspect many can recall occasions of “analysis paralysis”, or even when they suspect prolonged fretting worsened the quality of the decision they finally made. I think folks in EA-land tend to err in this opposite direction, and I find myself giving similar sorts of counsel on these themes to those who (perhaps unwisely) seek my advice. So I write.

Certainty, resilience, and value of (accessible) information

It is widely appreciated that we can be more certain (or confident) of some things than others: I can be essentially certain I have two eyes, whilst guessing with little confidence whether it will rain tomorrow. We can often (and should even oftener) use numbers and probability to express different levels of certainty - e.g. P(I have two eyes) ~ 1 (- 10^-(lots)); P(rain tomorrow) ~ 0.5.

Less widely appreciated is the idea that our beliefs can vary not only in their certainty but how much we expect this certainty to change. This ‘second-order certainty’ sometimes goes under the heading ‘credal (¬)fragility’ or credal resilience. These can come apart, especially in cases of uncertainty. I might think the chances of ‘The next coin I flip will land heads’ and ‘Rain tomorrow’ are 50/50, but I’d be much more surprised if my confidence in the former changed to 90% than the latter: coins tend fair, whilst real weather forecasts I consult often surprise my own guesswork, and prove much more reliable. For coins, I have resilient uncertainty; for the weather, non-resilient uncertainty.

Credal resilience is - roughly - a forecast of the range or spread of our future credences. So when applied to ourselves, it is partly a measure of what resources we will access to improve our guesswork. My great-great-great-grandfather, sans access to good meteorology, probably had much more resilient uncertainty about tomorrow’s weather. However the ‘resource’ need not be more information, sometimes it is simply more ‘thinking time’: a ‘snap judgement’ I make on a complex matter may also be one I expect to shift markedly if I take time to think it through more carefully, even if I only spend that time trying to better weigh information I already possess.

Thus credal resilience is one part of va...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why agents are powerful, published by Daniel Kokotajlo on June 6, 2022 on The AI Alignment Forum.

[Written for Blog Post Day. Not super happy with it, it’s too rambly and long, but I’m glad it exists.]

Here are some questions I think this theory of agency can answer:

What are agents?

Why should we expect AI agents to be useful for various important tasks?

Why should we think agentic mesa-optimizers may arise for some data+reward-signals that weren’t designed explicitly to produce them?

Why should we think humans+AI tools will eventually be outcompeted by AI agents?

Why should we expect, on priors, mildly superhuman AI agents to be powerful enough to take over the world, if they wanted to?

What are agents?

Earlier in “Agents as P2B chain reactions” I defined agents as successful P2B feedback loops—a learner algorithm and a planner algorithm hooked up together, the planner plans to plan better and thus outputs actions that result in getting more useful data into the learner, more resources for the planner to plan with. then the process repeats. Like how “fire” is a successful feedback loop in which heat+plasma+fuel come together to produce more heat+plasma, and envelop more fuel, if any is nearby.

Then I asked: Is the planner algorithm strictly necessary here? Could you get something similar to a P2B feedback loop, but built around something other than a planner? I explored this question indirectly in “Gradations of Agency.” The answer seems to be “sorta.” You can have e.g. a Level 3 system that doesn’t do any planning and yet behaves remarkably similar to a P2B feedback loop in many contexts, due to imitating the success of others. Such a system would tend to underperform P2B in contexts where there aren’t good examples to imitate.We could stick to the original definition and say: Agents are P2B feedback loops, so, level 4 and above.

But I think it’s probably better to be a bit more galaxy brain and say: the key thing is convergent instrumental resource feedback loops and that thinking about P2B is a stepping stone which helps us see why such phenomena exist and why the really big ones tend to have planners sitting in the middle directing them, and the really really big ones tend to have level 6+ algorithms sitting in the middle directing them.

By analogy, heat+plasma+wood+oxygenl chain reactions aren’t the key thing. The key thing is chain reactions that convert some sort of fuel into thermal and kinetic energy plus more of the chain reaction. Heat+plasma+wood+oxygen is a particularly important one, due to the ease of creating it and the abundance of fuel for it on Earth, but there’s also e.g. baking soda + vinegar and neutrons+uranium. And also self-replicating nanobots + pretty much anything.

Similarly: The important thing for understanding agency is understanding that the world contains various self-sustaining chain reactions that feed off instrumental resources like data, money, political power, etc. and spread to acquire more such resources. Level 4 P2B loops are a particularly important, powerful, and widespread instance of this type, analogous to fire, but there are less powerful/important things (such as Level 3, Level 2.) and also more powerful/important things (Level 6 and above?).

Why should we expect AI agents to be useful for various important tasks?

Why should we think agentic mesa-optimizers may arise for some data+reward-signals that weren’t designed explicitly to produce them?

Both questions have the same answer. Agency works; it’s powerful and general. Recalling this graph and the conjectures from last post, I claim that as you crank up the “how fast does this thing learn, how high a score does it get, in how large and diverse a range of environments?” dial, you ascend the hierarchy until you get agents. You can get powerful non-agents but only by d...