Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rethink Priorities is hiring fellows, research assistants, and researchers in several cause areas, published by abrahamrowe on March 21, 2022 on The Effective Altruism Forum.

We are also hiring operations staff — find out more here.

We are having two webinars on these roles

AI Governance & Strategy and Longtermism on April 5th, 2022. Find out more here.

Animal Welfare and Global Health & Development on April 5th, 2022. Find out more here.

Summary

TLDR: Rethink Priorities is hiring for lots of research positions. Apply here. You can apply for as many of these roles (and our operations roles) as you’d like to!

Rethink Priorities is expanding our research teams in several of the cause areas we work in. Specifically, we are hiring:

Fellows — early-career researchers interested in temporarily exploring EA research

Research Assistants — early-career researchers interested in aiding ongoing research projects and researchers

Researchers — researchers at any career level interested in conducting and leading research projects

(We say “early-career” to mean that people with little experience with research and/or with the relevant topic area could still be excellent fits for these roles. But these roles are of course also open to applicants with more experience!)

We’re hiring fellows, research assistants, and/or researchers for several teams right now:

AI Governance and Strategy

General Longtermism (other than AI)

Animal Welfare

Global Health and Development

To assist our Executive and Operations teams

Research roles at RP are a great way to explore a specific topic alongside others with varied backgrounds and interests across the EA landscape. Right now, we are especially focused on cultivating early-career researchers to help with talent bottlenecks in the EA research space, so we have an especially large number of open positions for three- to five-month Research Fellows and permanent Research Assistants. However, we expect to have other permanent research positions open in the near future.

Research Assistants are a new permanent position at RP. Research Assistants will conduct a variety of tasks to support particular teams and researchers on their agendas. Depending on the team, they may also have opportunities to explore their own research interests. What these roles will look like differs between teams, so we encourage you to check out the relevant job descriptions for more details!

Topics that our teams are focused on right now include:

AI Governance and Strategy: What AI development & deployment scenarios will occur, and what actions grantmakers, policymakers, and others should take to reduce extreme risks in those scenarios

General Longtermism: Facilitating faster and better creation of “longtermist megaprojects”, and improving clarity about which “intermediate goals” longtermists should pursue

Global Health and Development: Climate change and identifying global health interventions that are cost competitive with GiveWell’s top charities

Animal Welfare: The cost effectiveness of farmed animal welfare interventions, invertebrate sentience, and potential wild animal welfare interventions

Why RP?

You get to work on interesting topics, and directly inform how millions of dollars are donated or used in the EA community.

You can use up to 10% of your time for professional development, learning, or projects you want to pursue outside your assigned tasks.

We are entirely remote, and can hire people in most countries.

We are a highly flexible employer — most of our positions are open to candidates at any amount of hours above 20 per week, and staff set their own schedules.

We offer competitive salaries and comprehensive benefits, including parental leave and sabbaticals for employees with a few years of tenure.

Our work culture is also great:

We have an incredibly low turnover ra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effectiveness of a theory-informed documentary to reduce consumption of meat and animal products: three randomized controlled experiments, published by Jacob Peacock on March 21, 2022 on The Effective Altruism Forum.

Link to read the full article

Recommendations

Evidence does not support the use of informational documentaries.

Interventions that change intended eating habits may not result in actual change in eating habits. In our study, even very large changes in intention did not change actual behavior.

Future studies should ensure that participants are unaware of the study's purpose. They can do this with blinding.

Key Findings

Tested a 20-minute documentary "Good For Us" that highlights the environmental, human health and animal welfare harms of eating meat and other animal products.

In a randomized controlled experiment, compared the documentary to a control video (a generic motivational speech). Participants were from the general population of the United States.

Followed up 12 days later with a survey that was described as a different study. This helped to "blind" participants to the purpose of the study when collecting data, reducing potential bias.

Our first study found the documentary had no effect on a variety of different outcomes.

Found no reduction in animal product consumption. The average change we measured in one study was less than a 1-ounce reduction in animal product consumption per week, with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.

Found no change in moral valuation of animals ("speciesism").

Found no meaningful increases in interest in animal activism or in perceived importance of environmental sustainability, animal welfare, or eating a healthful diet.

Our second study was deliberately designed less rigorously, to resemble previous studies that measured intended behavior. Immediately after the documentary, asked viewers if they planned to eat more or less animal products next week. Many viewers planned to eat less animal products in the week after seeing the documentary.

Documentary made viewers 242% more likely to intend to reduce meat consumption than participants who viewed the control video. Critically, our first study suggested that these intentions do not actually translate to reductions in consumption.

A previous meta-analysis suggested that comparable interventions make people about 22% more likely to intend to reduce their meat consumption. So our documentary was likely very convincing relative to other interventions, but still not effective at reducing consumption in our more rigorously designed study.

Our third study tried to make the documentary more effective. Added a pledge, goal-setting exercises, and reminder email; and participants were people interested in nutrition research. The documentary still was not effective by any measurement. Also looked at just people who attended at least a 2-year college and identified as Democrats, but still no effect.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Who is protecting animals in the long-term future?, published by alene on March 21, 2022 on The Effective Altruism Forum.

Dear Amazing EA Forum Users,

Thank you for dedicating yourselves to doing good.

Today I read an article in Watt Poultry that stoked an old fear. It's a fear that comes up for me—and I think for some other animal-focused EAs—when thinking about the longterm future. My fear is: Will the longterm future mean expanded factory farming, which goes on forever? (And therefore produces exponentially more suffering than factory farming currently produces?) Who is looking out for animals in the longterm future?

The article was titled Chickens grown on Mars?. It says:

With the research done by the NC State Nuggets on Mars program, we could one day be growing chickens on Mars . . . .

[The teachers] will learn about . . . the unique challenges of raising chickens on Mars. . . .

At the end of the program, the participants will be tasked with putting together a unit for their class focused on developing ideas on ways to raise chickens on mars.

Emma, Cottrell, Chickens grown on Mars?, Watt Poultry (Mar. 11, 2022),.

The Nuggets on Mars program is apparently being paid for by the U.S. federal government's United States Department of Agriculture (USDA). And it sounds like the program has support from an ag industry group called the North Carolina Farm Bureau Federation.

From this article, it seems that some in the U.S. agriculture industry and the U.S. government hope that a future of humans on other planets will include factory farming on other planets!

That's exactly my fear.

The plans described in this article differ from what I've often heard longtermist EAs say, when I bring up my recurring fear about animal cruelty in the longterm future. I generally hear longtermist EAs reassure me that, (1) 'If we expand to living on other planets, we won't bring factory farming with us, because it would be too cumbersome to raise animals in space,' (2) 'In the far longterm future, all minds will be digital, so we will no longer have to worry about animal cruelty,' and (3) 'Over time, people will become more empathetic, so future humanity will treat animals better.'

I hope that the longtermist EAs are right, and that the meat industry is wrong. But, in my experience, the meat industry is pretty good at getting things done, even if those things seem wasteful or cumbersome. (For example, farming animals for food, even on Earth, is already far less efficient than farming plants. And yet the number of animals farmed expands every year.) And the meat industry and the U.S. government seem to WANT the future to involve high-tech ways to factory farm even more animals. So that scares me.

I acknowledge that this news article is focused on a far shorter term future than longtermists think about. The article seems to hint that people could start living on Mars within the lifetime of children who are alive today. But to someone, like me, who is uneducated on such topics, the practice of factory farming in space seems like a practice that, once started, could simply continue and expand. Think, for example, of what happened with factory farming on Earth. My mind says, 'Once factory farms get to Mars, perhaps it will be too late to prevent a future of constantly expanding space factory farms?'

I know essentially all EAs care about all sentient minds, including animal minds. And I know that the EAs focused on longtermism care about animal welfare, to the extent there will be animals in the longterm future.

My fear, though, is that, from my vantage point, longtermism and factory farming seem like two separate EA cause areas that are rarely discussed together. (This may be totally wrong, and I apologize if it is.) My decade in animal advocacy has taught me that farmed animals almost always get...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ukraine Post #5: Bits of Information, published by Zvi on March 21, 2022 on LessWrong.

Rather than attempt a synthesis this time around, I’m going to experiment with the opposite. Over the course of the last three weeks, I kept adding to a long list of sources and interesting things the same way I do for Covid-19 posts, but without an overall update type of post there was nowhere for them to go.

Thus, while this in no way pretends to be complete or to provide a proper synthesis, this is the article form of that giant link bank and associated thoughts, organized in logical fashion.

When going through the list, many of the links from February were no longer relevant, as they concerned things like whether we would give aid, or whether we would impose sanctions and how many and how Russia would react, or whether firms would divest, and other things like that. Then there was a phase shift when it became clear we would sanction and provide aid and the military situation got bogged down, after which new info mostly stayed more relevant.

If I had to summarize the overall picture, I’d say something like this, note that even the summary contains a ton of stuff because there’s a ton of stuff to cover.

Russia’s military campaign has culminated.

It is going to be very difficult or impossible for Russia to progress further.

Russia has huge, potentially fatal, logistical problems.

Russia is out of deployable resources, seems unable to meaningfully further mobilize.

Sanctions are hurting but a quick total collapse will not happen.

Ukraine will still find it difficult for now to make progress rolling back Russia’s gains, but that slowly should change as Russia’s army degrades further.

Russia is a deeply dysfunctional mafioso state.

Russians are largely buying into not only the invasion but future ones too.

Russia does a good job preventing rebellion, but at cost of everything else.

Western approach to solving the problems of Russian Oil and Gas are non-zero but mostly not serious or of sufficient magnitude or physical-world-orientation.

The coming food crisis is mostly not being addressed at all.

If we did want to solve such issues, expensive but realistic solutions exist.

A lot of very large ‘pure wins’ also exist that we are not using either.

Western coalition’s core has become much stronger and more united.

Our game theory seems aggressive and less than ideal, but much better than that which the public would favor, which would be kind of totally nuts.

Escalation risks definitely exist on both sides, remains unclear what Putin will do if he realizes how badly he is losing and we’ve given NATO countries a green light to send troops into Ukraine (but won’t do it ourselves.)

No one takes nuclear safety or issues seriously, so they take nothing seriously.

West is creating a very big ‘penalty for being late’ problem, where any deviation from our agenda, or in some cases even from a very left-wing agenda, results in massive punishments.

This causes those who cannot accept the totality of the system out of the system, weakening its position and strengthening the opposition.

China is trying to be in opposition without provoking the response, so far this is working, but internal propaganda there seems very pro-Putin and anti-USA.

Those in opposition then tend to both cooperate with each other and to converge on a set of models, beliefs and rhetoric that includes many quite false and/or awful things, an anti-pattern demonstrating opposition.

Peace talks may or may not be ‘fake’ on either side especially the Russian one.

If they are real there are three issues: Territory, demilitarization and ‘denazification.’

Any peace soon likely involves some territorial concessions, unclear if a possible deal exists here yet.

Demilitarization will potentially be a Sweden/Austria model. Includes a no-NATO ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A few notes on managing highly stressful news and situations, published by Ben Williamson on March 20, 2022 on The Effective Altruism Forum.

Following multiple recent discussions here on the Forum about managing stressful news and current affairs, particularly in light of the ongoing war in Ukraine, we have put together a short summary of useful advice.

This post includes suggestions made by other members of other EA community, advice from licenced therapists, and insights based on the authors’ work on an EA mindfulness-based programme and the Effective Self-Help initiative. Our aim here is to provide a quick, practical guide to better managing your response to difficult news and situations. While the discussion here is framed in the context of the current Ukraine crisis, we believe that many of these conclusions are useful for the anxiety produced by other prevalent EA topics like existential risks.

In addition to this post, we are collaborating with EA Anywhere to host an online workshop open to all on Wednesday 23rd March (6pm, UTC) where we explore these themes and ideas in greater depth.

It’s okay to feel stressed

As a starting point, it is important to acknowledge to yourself that it is both understandable and reasonable to feel stressed about the current situation in Ukraine.

We believe it is quite likely that people in the EA community may be particularly prone to high levels of stress, anxiety, and fear from events like those currently unfolding. This seems plausible given EA’s:

Greater focus on helping others and trying to improve the world (something that is jeopardised by the current situation)

Greater than average awareness of how the current situation could become a lot more serious (through wider greater power conflict and/or nuclear warfare)

Familiarity with situations where there are clear positive actions we can take (with the situation in Ukraine a lot tricker to impact in contrast)

For anyone feeling particularly highly stressed or anxious about current events, we encourage you not to feel bad for feeling like this. We all have different levels of resiliency and different emotional responses to bad news and so must accept that we may react in ways that seem better or worse than others. Given this, we should avoid comparing ourselves to the responses of others.

Similarly, there is likely a large variation in what solutions are most useful for different people and so we encourage you to experiment with different approaches to managing your stress. It is more than okay to begin by simply taking a break if current events feel paralysing; only by taking care of ourselves can we sustainably help others.

In this way, one useful approach can be to actively practice self-compassion. Try to accept difficult emotions rather than trying to fix them, being curious about emotions and sensations you are feeling instead of looking to change them. Many different self-compassion exercises exist, but we recommend loving-kindness meditation, simple diaphragmatic breathing exercises, and self-compassion breaks as a starting point.

While being highly stressed can be very understandable, it isn’t helpful for anyone involved. It does nothing to help people directly suffering while reducing your wellbeing and productivity, impacting your ability to make a meaningful impact in your work (or donations) on this situation, or potential similar/ worse conflicts in future.

In this light, it can be valuable to aim at practicing compassion rather than empathic distress. In other words, aim to feel for the suffering of others while retaining an awareness of your separateness from their experience and circumstances.

Practical steps for minimising avoidable stress

Reduce/ streamline your news consumption

While it is easy to feel an overwhelming sense of importance to stay up-to-date...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Landscape Analysis of Institutional Improvement Opportunities, published by IanDavidMoss on March 21, 2022 on The Effective Altruism Forum.

Summary

The Effective Institutions Project is a global working group that seeks out and incubates high-impact strategies to improve institutional decision-making around the world. Over the past several months, we conducted a landscape analysis to identify which of the world’s institutions (e.g., prominent government agencies, corporations, NGOs, etc.) offer the highest-leverage opportunities for philanthropists and professionals seeking to improve world outcomes over the next 10-100 years.

Methodology

To build a short list of important institutions to prioritize for deeper analysis, we combined existing lists of the world’s top private foundations, employers, universities, etc; assessed which institutional affiliations appeared most often among the members of global commissions convened over the past five years; and solicited nominations of important institutions from 30 global experts representing a range of fields, geographies, and professional networks.

Next, we conducted research and developed 3-5-page profiles on 41 institutions. Each profile covered the institution’s organizational structure, expected or hypothetical impact on people’s lives in both typical and extreme scenarios, future trajectory, and capacity for change.

Finally, we built a quantitative model to gauge the potential benefits of engaging with each of these 41 organizations by way of a hypothetical large, well-designed philanthropic investment. The model provided a rough estimate for the impact that an investment of this nature might have on two outcomes of interest: 1) global wellbeing-adjusted life-years over the next 10 years and 2) reducing the probability of existential risks over the next 100 years. The analysis takes into account the feasibility of influencing each institution and the likelihood that equivalent improvements would be implemented in the time frame anyway.

Key Findings

Our analysis suggests that opportunities to drive better world outcomes by improving institutional decision-making are relatively concentrated among a small number of organizations. This finding holds across both of our outcomes of interest, and the top-ranked institutions turned out to be largely the same in each analysis.

Our list of top institutions features heavy representation from tech companies and US government bodies, six of which ranked among the top ten in both our 10-year and 100-year analyses:

Amazon

DeepMind

Meta (formerly Facebook)

OpenAI

The Executive Office of the US President

The US National Security Council.

Additional top-ranked institutions from a near-term perspective included Google and the World Health Organization. Additional top-ranked institutions from a long-term perspective included Alphabet, the European Union, and the US Congress. Finally, we assessed that Chinese citizens and organizations may have opportunities to improve world outcomes via engagement with the Politburo of the Chinese Communist Party and China’s State Council on certain non-politicized issues.

Our extremely rough, please-don’t-take-these-numbers-literally estimate is that an investment of $100 million a year focused on well-designed strategies in this arena would be worth the equivalent of saving 4-30 million lives (20-200x as cost-effective as GiveWell’s top charities) over the course of a decade, or eliminate as much as one-tenth of total existential risk from all sources over the course of a century.

Conclusions and Next Steps

This article is a first attempt to answer a research question of vast complexity conducted with very limited resources (about 250-300 hours of effort in total across all parties). All conclusions put forward here should therefore be regarded as t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against cash benchmarking for global development RCTs, published by Rory Fenton on March 21, 2022 on The Effective Altruism Forum.

Should you fund an expensive program to help people or just send them cash? Using a randomised controlled trial (RCT) to directly compare international development programs to cash transfers was the Cool New Thing a few years back, with Vox calling it a “radical” idea that could be a “sea change in the way that we think about funding development". I spent 2 years on an RCT of a program that wanted to be radical and sea-changey, so naturally we considered using a cash comparison arm.

We didn’t use one. Throughout the process my mind was changed from “cash comparison arms are awesome” to “cash comparison arms probably rarely make much sense”. This change of perspective surprised me, so I wanted to think out loud here about why this happened.

The program I was evaluating

Asset transfer programs, such as giving people goats or fertiliser, have been found to work well at reducing poverty, but they also cost a lot, like over $1,000 per household. They also often have high staff costs from training and are difficult to massively scale. We wanted to try a super cheap asset transfer program that cost about $100 per household with minimal staffing, to see if we could still achieve meaningful impacts. We designed this as an RCT with 2,000 households in rural Tanzania and we'll hopefully have a paper out this year. We gave treatment households a bundle of goods including maize fertiliser, seed, chicks, mosquito bed nets, and a load of other things.

Why someone might suggest a cash arm

Much smarter people than me are in favour of directly comparing development programs to cash. There are arguments for it here and here. My favourite argument comes from a financial markets analogy: imagine you are considering investing in a fund. “We consistently make more money than we lose” is. good to hear. But much better would be, “We consistently beat the market”. That’s the role that a cash arm plays: rather than just check if a program is better than doing nothing at all (comparing to a control), we index it against a simple intervention that we know works well: cash. The fairest, most direct way to do this is to simply add an extra arm to your RCT, comparing treatment, control, and cash arms.

Why we wanted a cash arm at first

We really wanted to be compared to cash. Being able to say “give us $100 and we’ll do as much good as a $300 cash transfer” would be a powerful donor pitch. We were pretty confident that our program was better than cash, too. We bought our products at large scale on global markets, which meant our money went way further than our recipients’ could and we had access to quality products they couldn’t buy locally.

We even ran a small cash trial with 40 households. They all spent the cash well (mostly on home repairs) but no one seemed able to find investment opportunities as good as the ones in our asset bundle. When we eventually told the cash arm participants that we had given other households assets of the same value, most said they would have preferred the assets, “We don’t have good products to buy here”. We had also originally planned to work in 2 countries but ended up working in just 1, freeing up enough budget to pay for cash.

How my mind changed on cash

In short: Different programs will have impacts over different horizons, so the timing of when you collect your impact measurements will heavily skew whether cash or your program looks better.

In long:

Cash impact hits roughly immediately after distribution as households start to spend it

Our program’s impact took much longer to hit:

Our program included chicks that wouldn’t lay eggs or be eaten until they were 6 months old

We also gave maize inputs that would generate income only at h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Searching for outliers, published by benkuhn on March 21, 2022 on LessWrong.

Shortly after I started blogging, because I was a college student and had nothing better to do, I set a goal to write every week. I started in September 2013 and wrote around 150 posts between then and when I started working at Wave. (At that point I stopped having nothing better to do, so my blogging frequency tanked.)

The outcomes of these 150 posts were extremely skewed:

Two made it big on the Hacker News frontpage (What happened to all the non-programmers and Readability, hackability, and abstraction).

After seeing the second post on HN, Dan Luu subscribed to my blog and started submitting lots of my posts. This caused an additional ~5 posts to get decent traction, which resulted in my first wave of subscribers that I didn’t know personally, and provided a lot of motivation to keep writing. Dan and I also eventually became good friends.

The other ~95% of posts were completely forgettable.

This is a pretty typical spread of outcomes for blog writers: a few smash hits and a lot of clunkers. Eight years later, I’ve built a good enough intuition for what posts will resonate with people that I can now mostly avoid writing complete clunkers, but even so, my few best recent posts (In defense of blub studies and You don’t need to work on hard problems) have been much more successful than the others, both in terms of being shared widely, and getting feedback like “this really influenced how I think.”1

This type of statistical distribution of outcomes is called heavy-tailed, because outcomes in the “tail” (i.e. ones that are far better than the typical outcome) have a relatively high chance of occurring, making the tail “heavy.” When I write blog posts, each post is a sample from the heavy-tailed distribution of blog post outcomes.

You can most easily see the difference between a heavy-tailed and light-tailed distribution on a plot. Here’s a plot comparing a heavy-tailed and light-tailed distribution with identical means and standard deviations, chosen to be similar to the distribution of household income in the US (median = $60,000; p99 = $600,000):

Heavy-tailed distributions are really unintuitive to most people, since all the “action” happens in the tiny fraction of samples that are outliers. But lots of important things in life are outlier-driven, like jobs, employees, or relationships, and of course the most important thing of all, blog posts.

Because heavy-tailed distributions are unintuitive, people often make serious mistakes when trying to sample from them:

They don’t draw enough samples

They underestimate how good of an outcome it’s possible to get

They find it hard to tell whether they’re following a strategy that will eventually work or not, so they get incredibly demoralized.

If you’re aware of when you’re working on something that involves sampling from a heavy-tailed distribution, you can avoid those mistakes and end up with much better outcomes.

As a rule of thumb, a heavy-tailed distribution is one where the top few percent of outcomes are a large multiple of the typical or median outcome. A classic example would be Vilfredo Pareto’s finding that about 80% of Italy’s land was owned by 20% of the population. It turns out that this happens across many other domains, too—a phenomenon called the Pareto principle or the 80-20 rule. In contrast, distributions that don’t follow an 80-20 or similar rule are called “light-tailed.” Some examples:

Income is heavy-tailed: the median person globally lives on $2,500 a year, while the top 1% live on $45,000, almost 20× more.

Height is light-tailed: the tallest people are only a few feet taller than average. If height followed the same distribution as income, Elon Musk, who made $121b in 2021, would be about 85,000 km tall, or about ¼ of the dist...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Civilization as Self-Restraint, published by Duncan Sabien on March 21, 2022 on LessWrong.

The following is the first post in the Civilization and Cooperation sequence, with the goal of laying out a coherent, explicit, and actionable model of what humans are doing when they form societies and adhere to their rules. Each essay is intended to communicate something like a single step of a proof—they should stand alone, but are much more interesting in conjunction.

As a child (not raised within a church but surrounded by friends who were), I was always struck by the phrase thou shalt not covet in the last two Old Testament commandments.

Coveting (it seemed to me) was an entirely internal action, one which could at least in theory have no impact on the external world—if you had strong moral boundaries against theft, adultery, and (I guess) whining, then there seemed to be no reason why the act of coveting itself would be a problem. There were certainly things which I coveted which literally no one else on Earth was aware of, so ... ?

Eventually, eight-year-old me generated a pet theory: once in the Christian heaven, people can do anything they want, and thus it was not enough to simply block bad behavior. Heaven is a walled garden, and it's meant to be at least in part a wish-fulfillment paradise—thus, the only people you can safely allow inside are those who not only reliably act well, but also robustly desire only virtuous things. If someone was holding off only because of external prohibitions, they'd wreak havoc once they got into the wish-fulfillment zone.

(This theory was further reinforced by my vague understanding that swearing/cursing was also Forbidden, along with things like "being angry for petty or trivial reasons.")

Another way to express the above theory is something like "external restraints are less reliable than self-restraint." Anyone who has ever been responsible for preventing a toddler from gleefully offing itself will likely agree—there is a limit to what can be accomplished by cleaning and childproofing and making decrees. The job becomes vastly easier once you can recruit the toddler's own motivations, and convince (or bribe) it to not want dangerous things in the first place.

(This is what's behind the questionable-but-not-entirely-outlandish practice of letting toddlers touch the hot stove, once. Usually, goes that theory of parenting, once is all they need.)

No complex system of rules and boundaries can work via entirely external imposition (at least, not until we're surrounded by autonomous surveillance/enforcement drones at all times). There are just too many ways to do something wrong, too many times when you're unobserved and can "get away with it," too many edge cases and loopholes in any explicit framework. Catching and punishing transgressions is too slow and lossy; people have to (more or less) want to adhere to the rules, because of their own values and principles.

This leads us to the central thesis of this first essay.

Consider a range between autonomy and civility, represented below as a gradient between red and white in accordance with the fun-to-play-with MTG color system:

(For the rest of this sequence, I'm going to use the terms "autonomy" and "civility" in a moderately nonstandard way, as the least-wrong handles for the concepts I want to communicate. Therefore, if you quote a passage that includes one of them, please also include this explanation or a summary of it.)

We can define autonomy as something like "the freedom to choose among any of the options that are permitted by physical law." In other words, if you can do it in the sense that it's technically possible, then you can do it in the permissive sense as well.

(Total autonomy is something like savagery or anarchy; I've chosen to avoid those terms to avoid motte-bailey-...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wargaming AGI Development, published by ryan b on March 19, 2022 on LessWrong.

I think wargames would be a useful tool for exploring AGI development scenarios. More specifically, I mean wargames of the kind used to train military officers in strategy. I think this format would help explore things like timelines and takeoff scenarios, in a way that is both concrete and transparent.

The thought cropped up because I have been popping into the AMA/Discussion post for Late 2021 MIRI Conversations over the course of the last week. Over the course of the MIRI conversations, and again in the AMA, I saw a lot of the tactic of offering or requesting concrete situations to test or illustrate some point. Wargaming feels to me like a good way to generate concrete situations. I think we could answer this question from So8res:

What's a story, full of implausibly concrete details but nevertheless a member of some largish plausible-to-you cluster of possible outcomes, in which things go well? (Paying particular attention to how early AGI systems are deployed and to what purposes, or how catastrophic deployments are otherwise forestalled.)

Over the same period I was reading the MIRI Conversations I was reading/re-reading a series of posts on wargaming at War on the Rocks, which I was linked to from the comments in a book-notes post about military innovation.

The oldest article of the lot focuses on the usefulness of wargames for teaching and learning; the original motivation for the author was how poorly they thought teaching a course on Thucydides went. As a highlight, I would like to point out a case where they considered a decision that had historically poor outcomes, but gained awareness of good motivations:

Remarkably, four of the five Athenian teams actually attacked Syracuse on Sicily’s east coast! As they were all aware that such a course had led to an Athenian disaster 2,500 years before, I queried them about their decision. Their replies were the same: Each had noted that the Persians were stirring, which meant there was a growing threat to Athens’ supply of wheat from the Black Sea. As there was an abundance of wheat near Syracuse, each Athenian team decided to secure it as a second food source (and simultaneously deny it to Sparta and its allies) in the event the wheat from the Black Sea was lost to them. Along the way, two of the teams secured Pylos so as to raise helot revolts that would damage the Spartan breadbasket. Two of the teams also ended revolts in Corcyra, which secured that island’s fleet for Athenian purposes, and had the practical effect of blockading Corinth. So, it turns out there were a number of good strategic reasons for Athens to attack Syracuse. Who knew?

Certainly not any War College graduate over the past few decades.

The article also references benefits from playing the game Diplomacy, which has been mentioned around here for similar purposes for a long time.

There was also a recent-history overview of wargaming in the US Defense arena, which links to a bunch of examples and extolls wargaming in a competitive format to develop the US Defense talent pool. I included it mostly for the links to real-world examples, which include but are not limited to: games about specific, current strategic problems; games about integrating AI into warfare; some historical cases from the tactical to the strategic level.

The part that persuaded me we could directly lift from wargames to explore AGI development is the use of wargames for exploring future wars. The important thing is the inclusion of capabilities which don't exist yet, reasoning about which is notoriously hard. A highlight here is the kind of stuff people learn from the games:

First, all games are competitive and involve teams fighting other teams. There is a big difference between fighting an algorithm...