Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't Over-Optimize Things, published by Owen Cotton-Barratt on June 16, 2022 on The Effective Altruism Forum.

or Optimizing Optimization

The definition of optimize is:

to make something as good as possible

It's hard to argue with that. It's no coincidence that a lot of us have something of an optimization mindset.

But sometimes trying to optimize can lead to worse outcomes (because we don't fully understand what to aim for). It's worth understanding how this happens. We can try to avoid it by a combination of thinking more what to aim for, and (sometimes) simply optimizing less hard.

Reflection on purpose vs optimizing for that purpose

What does the activity of making something as good as possible look like in practice? I think often there are two stages:

Reflection on the purpose — thinking about what the point of the thing at hand is, and identifying what counts as "good" in context

Optimizing for that purpose — identifying the option(s) which do best at the identified purpose

Both of these stages are important parts of optimization in the general sense. But I think it's optimization-for-a-given-purpose that feels like optimization. When I say "over-optimization" I mean doing too much optimization for a given purpose.

What goes wrong when you over-optimize

Consider this exaggerated example:

Alice is a busy executive. She needs to get from one important meeting to another in a nearby city; she's definitely going to be late to the second meeting. She asks her assistant Bob to sort things out. "What should I be optimizing for?", Bob asks. "Just get me there as fast as possible", Alice replies, imagining that Bob will work out whether a taxi or train is faster.

Bob is on this. Eager to prove himself an excellent assistant, he first looks into a taxi (about 90 minutes) and a train (about 60 minutes plus 10 minutes travel at each end — but there's a 20 minute wait for the right train). So the taxi looks better.

But wait. Surely he can do better than 90 minutes? OK, so the journey is too short for a private jet to make sense, but what about a helicopter? Yep, 15 minutes to get to a helipad, plus 45 minutes flight time, and it can land on the hotel roof! Even adding in 5 minutes for embarking/disembarking, this is 25 minutes faster.

Or ... was he assuming that the drivers were sticking to the speed limit? Yeah, if he make the right phone calls he can find someone who can drive door to door in 60 minutes.

Can he get the helicopter to be faster than that? Yeah, the driver can speed to the helipad, and bring it down to 57 minutes. Or what if he doesn't have it take off from a helipad? He just needs to find the closest possible bit of land and pay the owners to allow it to land there (or pay security people to temporarily clear the land even if they don't have permission to land). Surely that will come in under 55 minutes. Actually, if he's not concerned about proper airfields, he can revisit the option of a private jet ... just clear the street outside and use that as a runway, then have a skydiving instructor jump with Alice to land on the roof of the hotel ...

What's going wrong here? It isn't just that Bob is wasting time doing too much optimization, but that his solutions are getting worse as he does more optimization. This is because he has an imperfect understanding of the purpose. Goodhart's law is biting, hard.

It's also the case that Bob has a bunch of other implicit knowledge baked into how he starts to search for options. He first thinks of taking a taxi or the train. These are unusually good options overall among possible ways to get from one city to the other; they're salient to him because they're common, and they're common because they're often good choices. Too much optimization is liable to throw out the value of this implicit knowledge.

So there are two way...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A transparency and interpretability tech tree, published by Evan Hubinger on June 16, 2022 on The AI Alignment Forum.

Thanks to Chris Olah, Neel Nanda, Kate Woolverton, Richard Ngo, Buck Shlegeris, Daniel Kokotajlo, Kyle McDonell, Laria Reynolds, Eliezer Yudkowksy, Mark Xu, and James Lucassen for useful comments, conversations, and feedback that informed this post.

The more I have thought about AI safety over the years, the more I have gotten to the point where the only worlds I can imagine myself actually feeling good about humanity’s chances are ones in which we have powerful transparency and interpretability tools that lend us insight into what our models are doing as we are training them.[1] Fundamentally, that’s because if we don’t have the feedback loop of being able to directly observe how the internal structure of our models changes based on how we train them, we have to essentially get that structure right on the first try—and I’m very skeptical of humanity’s ability to get almost anything right on the first try, if only just because there are bound to be unknown unknowns that are very difficult to predict in advance.

Certainly, there are other things that I think are likely to be necessary for humanity to succeed as well—e.g. convincing leading actors to actually use such transparency techniques, having a clear training goal that we can use our transparency tools to enforce, etc.—but I currently feel that transparency is the least replaceable necessary condition and yet the one least likely to be solved by default.

Nevertheless, I do think that it is a tractable problem to get to the point where transparency and interpretability is reliably able to give us the sort of insight into our models that I think is necessary for humanity to be in a good spot. I think many people who encounter transparency and interpretability, however, have a hard time envisioning what it might look like to actually get from where we are right now to where we need to be. Having such a vision is important both for enabling us to better figure out how to make that vision into reality and also for helping us tell how far along we are at any point—and thus enabling us to identify at what point we’ve reached a level of transparency and interpretability that we can trust it to reliably solve different sorts of alignment problems.

The goal of this post, therefore, is to attempt to lay out such a vision by providing a “tech tree” of transparency and interpretability problems, with each successive problem tackling harder and harder parts of what I see as the core difficulties. This will only be my tech tree, in terms of the relative difficulties, dependencies, and orderings that I expect as we make transparency and interpretability progress—I could, and probably will, be wrong in various ways, and I’d encourage others to try to build their own tech trees to represent their pictures of progress as well.

Some important distinctions

Before I get into the actual tech tree, however, I want to go over a couple of distinctions that I’ll be leaning on between different types of transparency and interpretability.

First is my transparency trichotomy, which lays out three different ways via which we might be able to get transparent models:[2]

Inspection transparency: use transparency tools to understand M via inspecting the trained model.

Training[-enforced] transparency: incentivize M to be as transparent as possible as part of the training process.

Architectural transparency: structure M's architecture such that it is inherently more transparent.

For the purposes of this post, I’m mostly going to be ignoring architectural transparency—that’s because, while it’s definitely something I think we should do, I don’t expect it to be able to get us to the sort of transparency I think we’ll eventually need....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humans are very reliable agents, published by alyssavance on June 16, 2022 on LessWrong.

Over the last few years, deep-learning-based AI has progressed extremely rapidly in fields like natural language processing and image generation. However, self-driving cars seem stuck in perpetual beta mode, and aggressive predictions there have repeatedly been disappointing. Google's self-driving project started four years before AlexNet kicked off the deep learning revolution, and it still isn't deployed at large scale, thirteen years later. Why are these fields getting such different results?

Right now, I think the biggest answer is that ML benchmarks judge models by average-case performance, while self-driving cars (and many other applications) require matching human worst-case performance. The best current image models have ~99.9% reliability on MNIST, an easy handwriting recognition task. Redwood Research, which is exploring worst-case performance in the context of AI alignment, got reliability rates around 99.997% for their text generation models.

By comparison, human drivers are ridiculously reliable. The US has around one traffic fatality per 100 million miles driven; if a human driver makes 100 decisions per mile, that gets you a worst-case reliability of ~1:10,000,000,000 or ~99.999999999%. That's around five orders of magnitude better than a very good deep learning model, and you get that even in an open environment, where data isn't pre-filtered and there are sometimes random mechanical failures. Matching that bar is hard! I'm sure future AI will get there, but each additional "nine" of reliability is typically another unit of engineering effort. (Note that current self-driving systems use a mix of different models embedded in a larger framework, not one model trained end-to-end like GPT-3.)

(The numbers here are only rough Fermi estimates. I'm sure one could nitpick them by going into pre-pandemic vs. post-pandemic crash rates, laws in the US vs. other countries, what percentage of crashes are drunk drivers, do drunk drivers count, how often would a really bad decision be fatal, etc. But I'm confident that whichever way you do the math, you'll still find that humans are many orders of magnitude more reliable.)

Other types of accidents are similarly rare. Eg. pre-pandemic, there were around 40 million commercial flights per year, but only a handful of fatal crashes. If each flight involves 100 chances for the pilot to crash the plane by screwing up, then that would get you a reliability rate around 1:1,000,000,000, or ~99.99999999%.

Even obviously dangerous activities can have very low critical failure rates. For example, shooting is a popular hobby in the US; the US market buys around 10 billion rounds of ammunition per year. There are around 500 accidental gun deaths per year, so shooting a gun has a reliability rate against accidental death of ~1:20,000,000, or 99.999995%. In a military context, the accidental death rate was around ten per year against ~1 billion rounds fired, for a reliability rate of ~99.9999999%. Deaths by fire are very rare compared to how often humans use candles, stoves, and so on; New York subway deaths are rare compared to several billion annual rides; out of hundreds of millions of hikers, only a tiny percentage fall off of cliffs; and so forth.

The 2016 AI Impacts survey asked hundreds of AI researchers when they thought AI would be capable of doing certain tasks, playing poker, proving theorems and so on. Some tasks have been solved or have a solution "in sight", but right now, we're nowhere close to an AI that can replace human surgeons; robot-assisted surgeries still have manual control by human operators. Cosmetic surgeries on healthy patients have a fatality rate around 1:300,000, even before excluding unpredictable problems like bl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are too many young, highly-engaged longtermist EAs doing movement-building?, published by Anonymous EA on June 16, 2022 on The Effective Altruism Forum.

I feel some nervousness about whether a too-high fraction of younger longtermist EAs are focused on movement-building rather than skilling up to do/testing fit for object-level work:

One college-aged community builder I spoke with noted that a high percentage (~50%) of friends her age whom she considered highly-engaged longtermists were working on EA movement building. She noted that there are strong selection effects here, since she herself is a longtermist movement builder and movement builders are likely to be especially visible, so this was probably an overestimate.

My vague sense of younger longtermists is that they tend to want to work directly on either AI risk or movement building, but many consider AI risk too hard and end up focusing on movement building instead.

But I myself am not college-aged or recently-graduated, and my worry is mostly based on conversations with a small number of younger EAs rather than any systematic inquiry. I’d love to get a better sense of the distribution here — particularly from folks who have a read on what’s happening at universities with active EA groups, such as younger longtermists and perhaps CEA’s groups team. I’d be delighted to learn that my worries are unfounded, and I think there’s a good chance they are.

So, let’s hear it:

What share of young, highly-engaged longtermists are doing movement-building?

Is that share too high, too low, or about right?

My own thoughts on question #2:

If the 50 percent number is representative, that seems intuitively too high. The EA Leaders Forum Survey suggested that ~11 percent of EA resources should go toward “building the EA community and related communities.” I could imagine shading this upward for college-aged longtermists given their comparative advantages at community building (proximity/relatability to other students + lack of work experience + inability to do a full-time job), and of course it’s possible for community building to be a route to skilling up in direct work (e.g. running an AI safety bootcamp might be a good way to strengthen one’s command of the material). But my intuition is that at least two thirds of such people should be mostly focused on assessing fit for (and ideally beginning to contribute to) object-level work.

I worry some EAs might feel pressure to prioritize doing EA movement building in college even when it’s not a good fit/intrinsically motivating. I’d love to hear whether this rings true for anyone reading this post, or if I’m just imagining this.

In general, younger EAs might be strongly motivated to have an impact as soon as possible, rather than taking a long view of impact over the course of their careers. This could lead to focusing on movement building, where impact often feels very tangible (you can point to people you've introduced to EA; you can feel responsible if they go on to do cool EA-inspired things) even if it's actually quite indirect.

There's also tons of funding and support/mentorship available to young, highly-capable EAs who want to do movement building. This is largely a good thing, of course. Most young people will not have alternatives that would allow them to found a new organization with a large budget so early in their careers; they might otherwise have to specialize more in a particular cause (perhaps including going to grad school), get some work experience, and prove themselves in a line of work, before being trusted with significant amounts of funding and responsibility. Additionally, running your own big project can be a good learning opportunity and is perhaps easier to get started on alone relative to e.g. research. So I imagine entrepreneurship in movement building l...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Active Shooter Drills, published by Zvi on June 16, 2022 on LessWrong.

Epistemic Status: You really, really should know this already. Reference post.

We have a $2.7 billion industry devoted to causing the traumatizing of children.

They refer to it as ‘active shooter drill training.’

I was initially motivated to write this when I saw a claim that the bipartisan gun deal allocated $5 billion in additional funding for causing childhood trauma. That turned out not to be the case (summaries of the deal from NPR, from CNN and from WaPo don’t mention it and it seems too big to have been overlooked if it was real), but these shooter drills are still traumatizing our children on a regular basis, which seems worth pointing out as another prime example of both our civilization losing its mind and also the existence of extremely low-hanging fruit.

As in, we could simply not traumatize our kids all the time for no reason. Crazy, I know.

For those who don’t know, an ‘active shooter drill’ is where they take young children who are forced by law to be at a given location each day, and periodically teach have them practice hiding and being shot at by a mass shooter.

If your response to that idea is ‘what, what, that sounds horrible and terrifying and we should absolutely positively not do that’ then you seem like a normal human to me.

If your response to that idea is ‘yes this will help protect our children’ then I do not understand you.

Yet these drills are required. Since school is also required, this is something the state is forcing on children. Children are being forced to report periodically for trauma.

This is so much worse than wasted time. If I am told that one of my children is going to have such a drill, I will keep my child home from school that day, this seems like what a parent who cares about their child would do.

If the school does not consider this an acceptable solution, I will pull them out of that school entirely. It’s not only a one-time thing, they think this is an important thing that they practice continuously.

All the replies are like this, of parents reporting children coming back from drills terrified. I have tried to argue against Gain of Function research and against constructing an unfriendly AGI, and had to talk various players out of some truly epically bad deck choices, but I have never, ever seen anything like as clear a case of something you absolutely positively and obviously should not be doing.

This Washington Post article has a variety of reactions to drills. Some of them are about how shameful it is that we have people shooting up schools in the first place, but mostly it’s about how these drills primary effect is clearly to traumatize students.

Yet this has ballooned into an industry, due to our demand to do something:

I do not need a study to say that this ‘may’ do more harm than good, and neither do you.

Once again, the problem of actual shootings is terrible but the magnitude is simply not that large. It (once again) bears keeping in mind the magnitude of the problem:

The database documents when a gun is brandished, a gun is fired, or a bullet hits K-12 school property for any reason, regardless of the number of victims, time of day, or day of week. There have been 1,322 individual shootings since the 1970s, resulting in 426 deaths and 1,225 injuries.

Or, alternatively, there is one live shooting incident in a given school every 6,000 years, with an average of less than one death out of an average school size of slightly more than 500 students.

Even if active shooter drills were 100% effective at preventing deaths from school shootings, as opposed to being useless or (more likely) highly actively counterproductive even on that front, it would still be completely criminally insane to subject kids to this kind of trauma.

The Study

For...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Refer the Cooperative AI Foundation’s New COO, Receive $5000, published by Lewis Hammond on June 16, 2022 on The Effective Altruism Forum.

TL;DR: The Cooperative AI Foundation (CAIF) is a new AI safety organisation and we're hiring for a Chief Operating Officer (COO). You can suggest people that you think might be a good fit for the role using this form. If you're the first to suggest the person we eventually hire, we'll send you $5000.

This post was inspired by conversations with Richard Parr and Cate Hall (though I didn't consult them about the post, and they may not endorse it). Thanks to Anne le Roux and Jesse Clifton for reading a previous draft. Any mistakes are my own.

Background

The Cooperative AI Foundation (CAIF, pronounced “safe”) is a new organisation supporting research on the cooperative intelligence of advanced AI systems. We believe that many of the most important problems facing humanity are problems of cooperation, and that AI will be increasingly important when it comes to solving (or exacerbating) such problems. In short, we’re an A(G)I safety research foundation seeking to build the nascent field of Cooperative AI.

CAIF is supported by an initial endowment of $15 million and some of the leading thinkers in AI safety and AI governance, but is currently lacking operational capacity. We’re expanding our team and our top priority is to hire a Chief Operating Officer – a role that will be critical for the scaling and smooth running of the foundation, both now and in the years to come. We believe that this marks an exciting opportunity to have a particularly large impact on the growth of CAIF, the field, and thus on the benefits to humanity that it prioritises.

How You Can Help

Do you know someone who might be a good fit for this role? Submit their name (and yours) via this form. CAIF will reach out to the people we think are promising, and if you were the first to suggest the person we eventually hire, we'll send you a referral bonus of $5000. The details required from a referral can be found by looking at the form, with the following conditions:

Referrals must be made by 3 July 2022 23:59 UTC

You can't refer yourself (though if you're interested in the role, please apply!)

Please don't directly post names or personal details in the comments below

We'll only send you the bonus if the person you suggest (and we hire) isn't someone we'd already considered

The person you refer doesn't need to already be part of the EA community or be knowledgable about AI safety

If you've already suggested a name to us (i.e., before this was posted), we'll still send you the bonus

If you have any questions about the referral scheme, please comment below

Finally, we're also looking for new ways of advertising the role. If you have suggestions, please post them in the comments below. If we use your suggested method (and we weren't already planning to), we'll send you a smaller bonus of $250. Feel free to list all your suggestions in a single comment – we'll send you a bonus for each one that we use.

Why We Posted This

Arguably, the most critical factor in how successful an organisation is (given sufficient funding, at least) is the quality of the people working there. This is especially true for us as a new, small organisation with ambitious plans for growth, and for a role as important as the COO. Because of this, we are strongly prioritising hiring an excellent person for this role.

The problem is that finding excellent people is hard (especially for operations roles). Many excellent people that might consider moving to this role are not actively looking for one, and may not already be in our immediate network. This means that referrals are critical for finding the best person for the job, hence this post.

$5000 may seem like a lot for simply sending us someone's name, b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Perils of optimizing in social contexts, published by Owen Cotton-Barratt on June 16, 2022 on The Effective Altruism Forum.

As a special case of not over-optimizing things, I want to point out that an optimizing approach can have a bunch of subtle negative effects in social contexts. The slogan is something like "optimizers make bad friends" ... people don't like being treated as a means rather than an end, and if they get the vibe from social interactions that you're trying to steer them into something, then they may react badly.

Here I'm referring to things like:

"How do I optimize this conversation to get X to say yes?"

"How do I persuade Y of this particular fact?"

"How can I seem particularly cool/impressive to these people?",

... where it's not clear that the other side are willing collaborators in your social objectives.

So I don't mean to include optimizing for things like:

"How can I give the clearest explanation of this thing, so people aren't confused about what I'm saying?"

"How can I help Z to feel comfortable and supported?"

I think that people (especially smart people) are often pretty good at getting a vibe that someone is trying to steer them into something (even from relatively little data). When people do notice, it's often correct for them to treat you as adversarial and penalize you for this. This is for two reasons:

A Bayesian/epistemic reason: people who are optimizing for particular outcomes selectively share information which pushes towards those outcomes. So if you think someone is doing [a lot of] this optimization your best estimate of the true strength of the position they're pushing should be [much] lower than if you think they're optimizing less (given otherwise the same observations).

Toy example: if Alice and Bob are students each taking 12 subjects, and you randomly find out Alice's chemistry grade and it's a B+, and you hear Bob bragging that his history grade is an A-, you might guess that Alice is overall a stronger student, since it's decently likely that Bob chose his best grade to brag about.

An incentives reason: we want to shape the social incentive landscape so that people aren't rewarded for trying to manipulate us. If we just do the Bayesian response, it will still often be correct for people to invest some in trying to manipulate (in theory they know more about how much information to reveal to leave you with the best impression after Bayesian updating).

I don't think the extra penalty for incentive shaping needs to be too big, but I do think it should be nonzero.

Actually their update should be larger to compensate for the times that they didn't notice what was happening; this is essentially the same argument as given in Sections III and IV of the (excellent) Integrity for Consequentialists

Correspondingly, I think that we should be especially cautious of optimizing in social contexts to try to get particular responses out of people whom we hope will be our friends/allies. (This is importantly relevant for community-building work.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Breaking Down Goal-Directed Behaviour, published by Oliver Sourbut on June 16, 2022 on The AI Alignment Forum.

When we speak about entities 'wanting' things, or having 'goal-directed behaviour', what do we mean?

Because most of the actors that we (my human readers and I) attentively interact frequently with are (presumably) computationally similar (to each other and to ourselves), it is easy for our abstractions to conflate phenomena which are in fact different[1], or to couple together impressions of phenomena which are in fact separable[2]. On the occasions that we attentively observe dissimilar actors[3], we are often-enough 'in the wild' (i.e. looking in their 'habitat') that conflating - anthropomorphising - is sufficiently usefully predictive to get by[4], so these poor abstractions are insufficiently challenged.

Further, most of the people who in fact attentively observe a particular class of human-dissimilar actors (enough to perceive and understand the non-conflations) are focused domain experts about that particular class, and talk mainly to other domain experts about them[5]. There are few venues in which it is useful to have unambiguous terminology here. Thus, the language we use to communicate our abstractions about goal-directed behaviour is prone to conflation and confusion even when some people have some of the right abstractions[6].

Here I aim to take steps to break down 'goal-directed behaviour' into a conceptual framework of computational abstractions for which I offer tentative terminology, and which helps me to better understand and describe analogies and disanalogies between various goal-directed systems. The overarching motivation is to better understand goal-directed behaviour, in the sense of being able to better predict its (especially counterfactual and off-distribution) implications, its arisal, and other properties. Hopefully it is clear why I consider this worthwhile.

In order to ground this discussion, I refer to a reasonably diverse menagerie of candidate goal-directed systems, including natural and artificial systems at various levels of organisation. Contemplation of this diverse collection was responsible for the ideation and refinement of the ideas and gives some confidence in the appropriateness of the abstractions.

A collection of a few 'agents' drawn from the menagerie

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: BBC Future longform article on Progress Studies (including connection between progress and risk), published by Garrison on June 16, 2022 on The Effective Altruism Forum.

I wrote a 4k word feature on Progress Studies for BBC Future that just went up. (Tweet here). I explore:

the stagnation hypothesis

the origins of progress studies

what PS believes

will economic growth make us happier

frontier vs. catch-up growth, and how PS’s focus on the former reveals its biases

progress and existential risk

the future of the community

I think of PS as like EA circa ~2012. They have billionaire support and are quickly professionalizing. The community seems very interested in learning from EA and has responded to critiques that they are not focused enough on x-risk.

But they are different in important ways. Compared to EA, I think PS is:

more entrepreneurial, with a strong bias to action

less academic

more American, tech-y, and rooted in the Bay Area

less defined - PS is not nearly as rigorous or philosophically oriented

less demanding - PS doesn't really ask much, if anything, from its followers. I think this may be a huge force multiplier for the community, as it will better appeal to wealthy tech people, but probably makes any individual member less effective

more neoliberal and libertarian (though, compared to these groups PS is quicker to recognize market failures and call for govt intervention)

more speciesist - the focus is just on human progress, and tech has clearly been net-bad for farmed animals IMO

more growth-oriented

For an interesting look at what the intersection of PS and EA looks like, check out the Institute for Progress, a new think tank funded by PS co-founder Patrick Collison, Open Phil, SBF, and others.

Timeline

March 2017 - Roots of Progress blog starts

July 2018 - Stripe Press launches

July 2019 - “We Need a New Science of Progress” essay published in Atlantic

August 2019 - PS Slack channel launches

Oct 2019 - Roots of Progress becomes a nonprofit

Aug 2020 - Works in Progress online magazine starts

January 2022 - Institute for Progress think tank launches

Feb 2022 - Works in Progress acquired by Stripe Press

April 2022 - Progress Forum launches (sponsored by Roots of Progress)

May 2022 - The Atlantic Progress series launches

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform, published by NunoSempere on June 16, 2022 on The Effective Altruism Forum.

Epistemic status: Dwelling on the negatives.

Summary

From 2013 to 2021, Open Philanthropy donated $200M to criminal justice reform. My best guess is that, from a utilitarian perspective, this was likely suboptimal. In particular, I am fairly sure that it was possible to realize sooner that the area was unpromising and act on that earlier on.

In this post, I first present the background for Open Philanthropy's grants on criminal justice reform, and the abstract case for considering it a priority. I then estimate that criminal justice grants were distinctly worse than other grants in the global health and development portfolio, such as those to GiveDirectly or AMF.

I speculate about why Open Philanthropy donated to criminal justice in the first place, and why it continued donating. I end up uncertain about to what extent this was a sincere play based on considerations around the value of information and learning, and to what extent it was determined by other factors, such as the idiosyncratic preferences of Open Philanthropy's funders, human fallibility and slowness, paying too much to avoid social awkwardness, “worldview diversification” being an imperfect framework imperfectly applied, or it being tricky to maintain a balance between conventional morality and expected utility maximization. In short, I started out being skeptical that a utilitarian, left alone, spontaneously starts exploring criminal justice reform in the US as a cause area, and to some degree I still think that upon further investigation, though I still have significant uncertainty.

I then outline my updates about Open Philanthropy. Personally, I updated downwards on Open Philanthropy’s decision speed, rationality and degree of openness, from an initially very high starting point. I also provide a shallow analysis of Open Philanthropy’s worldview diversification strategy and suggest that they move to a model where regular rebalancing roughly equalizes the marginal expected values for the grants in each cause area. Open Philanthropy is doing that for its global health and development portfolio anyways.

Lastly, I brainstorm some mechanisms which could have accelerated and improved Open Philanthropy's decision-making and suggest red teams and monetary bets or prediction markets as potential avenues of investigation.

Throughout this piece, my focus is aimed at thinking clearly and expressing myself clearly. I understand that this might come across as impolite or unduly harsh. However, I think that providing uncertain and perhaps flawed criticism is still worth it, in expectation. I would like to note that I still respect Open Philanthropy and think that it’s one of the best philanthropic organizations around.

Open Philanthropy staff reviewed this post prior to publication.

Index

Background information

What is the case for Criminal Justice Reform?

What is the cost-effectiveness of criminal justice grants?

Why did Open Philanthropy donate to criminal justice in the first place?

Why did Philanthropy keep donating to criminal justice?

What conclusions can we reach from this?

Systems that could have optimized Open Philanthropy’s impact

Conclusion

Background information

From 2013 to 2021, Open Philanthropy distributed $199,574,123 to criminal justice reform [0]. In 2015, they hired Chloe Cockburn as a program officer, following a “stretch goal” for the year. They elaborated on their method and reasoning on The Process of Hiring our First Cause-Specific Program Officer.

In that blog post, they described their expansion into the criminal justice reform space as substantially a “bet on Chloe”. Overall, the post was very positive about Chloe (more on red teams below). ...