Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Man in the Arena, published by Richard Ngo on June 26, 2023 on LessWrong.

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds. Who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”

Today Kurt’s out strafing, which his stream loves, but which takes a lot of effort. Girl on the left, red jacket, half a block away. He focuses his eyes on her and his viZor immediately starts filling with insults, pick-up lines, nonsensical keyboard-mashing, and whatever else the schizophrenic hivemind of his viewers feels like generating. As the votes pour in the lines are shuffling around too fast for him to read, but that’s okay, she’s still twenty meters away. Ten meters, and it’s stabilized, one of them has clicked into top place; five meters, and he’s figured out just the right intonation. “Hey bitch”, then a pause—gotta get the pause just right, so she has enough time to realize he’s talking to her and look up, but not quite enough to process that anyone who’s calling her a bitch in the middle of the street is not someone she wants to be listening to—“You got a license to be this ugly in public?”

Boom! Perfect timing—he actually manages to get the shocked little o of her open mouth on camera, before she ducks her head away and hurries past him. The next line is popping up in his viZor, and he almost yells it out after her, but when you land a good first hit it’s easy to ruin it with a subpar follow-up. Patience is what separates the best from the rest, he always tells people. So with a swipe of his fingers he replays the clip to his stream instead. “See?” he says. “For the new subs: that’s what it looks like when you’re really fucking good at what you do.” Were those tears in her eyes? Doesn’t matter, let’s roll with it. He subvocalizes a command and his viZor enhances that section, zooming in and adding a slight sheen to the corners of her eyes. It’s a trick he figured out a while back—the livestream is only HD not ultraHD, so as long as you go back and edit before uploading the full video, you can get away with all sorts of stuff.

He gloats for another block, then starts looking for another target. There’s a big guy in the distance, but they’re tricky, you never want to take the chance that they get physical. A waiter standing outside a cafe, taking a couple’s order—oh, perfect. “Go to town”, he says, and his followers do their thing. The first few lines are terrible, but he slows down a bit, and eventually someone comes up with a banger about the three of them being served as his three-course meal. Along with the line, he does a little dance and then a hip-thrust in the woman’s direction. And the stream. Goes. Fucking. Wild.

He’s so busy celebrating that one that he doesn’t even notice the girl with a viZor of her own until she’s only half a block in front of him. Shit. He’d dodge her if he could, but it’s too late, everyone can tell he’s seen her now. Shit. This one could be hard: she might be strafing too, or at least have a proximity sensor up to warn her that she’s about to get streamed. If she starts hitting him, his viewers aren’t gonna be quick enough to generate comebacks; he’s on his own. That’s okay, though. That’s why they pay him the big bucks. He quickly pulls up a couple of his own lines that he’s prepped for this type of situation. Nothing jumps out about her appearance: short, dark ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI-Plans.com - a contributable compendium, published by Iknownothing on June 25, 2023 on LessWrong.

Hello, we’re working on .

Ideas behind the site:

Right now, alignment plans are spread all over the place and it’s difficult for a layperson, or someone unfamiliar with the field to get an idea of what is the current plan for making AGI or even the models we have right now, safe. And the problems with said plans.

Having a place where all AI Alignment plans and criticism of said plans can be added and seen in an easy to read way is helpful. It offers possibilities such as; seeing the most common problems with plans, which kinds of plans have the least problems, pushing for regulations against the most poor plans etc.

Judging the quality of a plan is hard and has a lot of ways to go wrong. Judging the quality of a criticism might be less complicated and perhaps has less ways it can go wrong.

Which is why I believe this site is useful

Hasn’t this already been done? aisafety.info, aiideas.com, etc

aisafety.info is excellent, and the folks there are building a conversational agent for AI Safety, which could be incredibly useful. However it’s not very simple to use to learn about specific alignment plans yet and is more of a general purpose place to learn about ai, ai-risk, ai-safety, etc.

The purpose of ai-plans.com is to be an easy to read platform that sorts the good plans from the bad and shows the problems with each one.

I believe there was a site called ai-ideas.com or something mentioned to me? But that wasn’t working the last time I checked and there’s been no change, as far as I know.

Aims

Stage 1)

A contributable compendium with most, if not all, of the plans for alignment, and criticisms

Estimated time left for this to be done: 1 week - adding ~5 plans a day now

What’s left to do:

Functionality for responding to criticisms

The idea for this is that a plan’s author can select a criticism/criticisms that they think they have a solution to, then submit a new version of the plan, with the criticism(s) quoted in the new plan.Any criticisms they didn’t select, will be automatically added to the new version of the plan (the idea being that any unselected criticisms are ones that they don’t have solutions to, so should still apply).

Adding more plans and criticisms

A filter for spam and duplicates

Creating a template/guide on how to post plans

Improve the UX - font, colours, design, etc.

What we’re missing for this stage:

A lawyer to make sure we’re complying with GDPR and help make a cookies notice

Moderators to help filter spam and make sure plans are submitted correctly

What would be helpful, but not essential for this stage:

More copywriters to speed up the process of adding alignment plans

More red-teamers/quality testers

Stage 2)

In addition to everything from Stage 1, there is now a scoring system for criticisms and a ranking system for plans- where plans are ranked from top to bottom based on the total scores of their criticisms.

The scoring system is a very essential part of the site.

It's aim is to give the most points to the most accurate criticisms and use that to rank plans from the least total criticism points (at the top) to the most criticism points(at the bottom).

Users will be able to upvote or downvote criticisms.

Users will also have a ‘karma’ that will affect how weighted their votes on criticisms are- somewhat similar to the LessWrong and AlignmentForum system- though, we're considering having karma past a certain 'age' become spendable, rather than add to the weight of the users vote, to avoid to first vote problem of LessWrong.

Users who accumulate more points on their criticisms will have a higher karma.

Plans will have a ‘bounty’ inversely proportional to the total number of criticism points they have.

I think it’s important to get this right the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longtermism and alternative proteins, published by BruceF on June 27, 2023 on The Effective Altruism Forum.

I spoke at EA Global London 2023 about longtermism and alternative proteins. Here’s the basic argument:

1) Meat production is a significant contributor to climate change, other environmental harms (pretty much all of them), food insecurity, antibiotic resistance, and pandemic risk - causing significant and immediate harm to billions of people.

2) All of these harms are likely to double in adverse impact (or more) by 2050 unless alternative proteins succeed.

3) Their X risk level is sufficiently high (Ord chart) that they warrant attention from longtermists. Especially for longtermists in policy or philanthropy, adding alt proteins to the portfolio impactful and tractable interventions that you support can allow you to do even more good in the world (a lot of it fairly immediate).

In the talk, I cite this report from the Center for Strategic & International Studies’ director of global food security & director of climate and energy, as well as a report from ClimateWorks Foundation & the Global Methane Hub (1-pager w/r/t the points I made in the talk here).

Below are the recording and transcript - comments welcomed.

Here's a link to the slides from this talk.

Introduction

The observation is that we have been making meat in the same way for 12,000 years. Food is a technology. Making meat is a technology. The way we do it now is extraordinarily inefficient and comes with significant external costs that do indeed jeopardize our long-term future. This is Johan Rockström after the EAT-Lancet Commission called on the world to eat 90 percent less meat back in 2018 and 2019. He said, "Humanity now poses a threat to the stability of the planet. This requires nothing less than a new global agricultural revolution." That's what I'm going to be talking about, and I'm going to situate it in terms of effective altruism.

There are five parts to the talk. The first one is that meat production has risen inexorably for many decades, and there is no sign of that growth slowing. The second is that our only strategy for changing this trajectory is support for alternative proteins - there's not a tractable plan B. The third point is that alternative proteins address multiple risks to long-term flourishing and they should be a priority for longtermists. I'm not going to try to convince you. They should be the priority - they're on par with AI risk or bioengineered pandemics. But I am going to try to convince you that, unless you are working for an organization that is focused on one thing, you should add alternative proteins to your portfolio if you are focused on longtermism. Fourth, I want to give you a sense of how GFI thinks about prioritization so that what we're doing as we expand is the highest marginal possible impact. Then we'll have some time for a discussion which Sim will lead us through.

Meat Production has risen by 300% since 1961.

The first observation is that, since 1961, global meat production has risen 300 percent.

In China, it has skyrocketed by 1,500 percent. It's 15 times up since 1961, and meat production and consumption is going to continue to rise through 2050.

There have been 11 peer-review articles looking at what meat production and consumption are going to look like in 2050. The lowest production is 61 percent more. One of the predictions is 3.4 times as much, so 340 percent more. Most of the predictions hover at about double. Most of that growth is not in developed economies. Developed economies have leveled off. They're going up a little bit. Most of that growth is in developing economies and in Asia.

The world doesn't have tractable solutions to this. Bill Gates when he released How to Avoid a Climate Disaster, on his book tour was talking about how the cl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 60+ Possible Futures, published by Stuckwork on June 26, 2023 on LessWrong.

Introduction

I have compiled a list of possible future scenarios. I hope this list is useful in two ways:

As a way to make your own thinking about the future more explicit; how much probability mass do you put on each possible future?

As a menu option to choose from; which of these futures do we want to make more likely?

This list is just a brainstorm, and I encourage readers to write any missing but probable futures in the comments. I will add any scenarios that do not substantially overlap with existing items and which I subjectively estimate as having at least a 0.01% probability of happening to the list (with attribution).

I have divided the possible futures into the following categories:

Futures without AGI, because we prevent building it

Futures without AGI, because we go extinct in another way

Futures without AGI, because we take a different path

Futures without AGI, because of strange factors

Futures with AGI, in which we die

Futures with AGI, in which we survive, and things are somewhat normal

Futures with AGI, in which we survive, but we're very different humans

Futures with AGI, in which we survive, and the universe gets optimized

Futures without AGI

Because we prevent building it

Successful Treaty: Humanity figures out that building AGI would be super-dangerous. After a long negotiation between world leaders, they succeed in agreeing on a FLOP quotum, which is far under the limit for potentially dangerous AGI. This policy is strongly enforced and prevents any individual or organization from developing AGI.

Surveillance: A world government is instantiated that recognizes that AGI would be dangerous. They ban AGI research and install an Orwellian surveillance machine that records and analyses every keystroke, voice command, and research meeting. This successfully prevents AGI from being created.

Regulation: Humanity enforces strong regulations on AI, mostly in order to combat non-existential risks such as discrimination, fairness, and loss of jobs. This makes R&D in AI unprofitable, as the resulting models cannot be deployed for any real-world use.

Humanity grows up: Humanity makes epistemic, technological, political and moral progress and learns how to defeat Moloch and cooperate at a planetary scale. We decide collectively that building AGI would be bad and is something we just don't do. Consequently, no one works on developing AGI.

Catastrophic risk tax: Economists find a way to fix capitalism by pricing in externalities, for example by using prediction markets to estimate impact. Catastrophic risk is priced as a huge externality. Working on AGI is so expensive that it isn't economically viable for anyone to work on it.

Once-but-never-again AI: Humanity develops powerful but not superintelligent AI. The consequences of this AI are catastrophic, but at least some humans survive and are able to turn it off. Humanity takes action to make sure that AI gets never developed again.

Terrorists: A terrorist group blows up all major actors in creating AGI in a series of terrorist attacks over multiple decades. This instills fear in researchers interested in AGI, preventing it from ever being built.

Pivotal act by humans: A group of people discover and execute a pivotal act that makes it impossible for humanity to create AGI afterward.

Pivotal act by cyborgs: A group of people artificially enhance their intelligence, such that they are intelligent enough to discover and execute a pivotal act that makes it impossible for humanity to create AGI afterward.

Pivotal act by narrow AI: Humanity builds a narrow AI with the task of discovering and executing a pivotal act that makes it impossible for humanity to create AGI afterward.

Because we go extinct in another way

Destruction by humanity: Human...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Field Building vs. EA CB, published by kuhanj on June 27, 2023 on The Effective Altruism Forum.

Summary

As part of the EA Strategy fortnight, I am sharing a reflection on my experience doing AI safety movement building over the last year, and why I am more excited about more efforts in the space compared to EA movement-building. This is mostly due to the relative success of AI safety groups compared to EA groups at universities with both (e.g. read about Harvard and MIT updates from this past year here). I expect many of the takeaways to extend beyond the university context. The main reasons AI safety field building seems more impactful are:

Experimental data from universities with substantial effort put into EA and AI safety groups: Higher engagement overall, and from individuals with relevant expertise, interests, and skills

Stronger object-level focus encourages skill and knowledge accumulation, offers better career capital, and lends itself to engagement from more knowledgeable and senior individuals (including graduate students and professors).

Impartial/future-focused altruism not being a crux for many for working on AI safety

Recent developments increasing the salience of potential risks from transformative AI, and decreasing the appeal of the EA community/ideas.

I also discuss some hesitations and counterarguments, of which the large decrease in neglectedness of existential risk from AI is most salient (and which I have not reflected too much on the implications of yet, though I still agree with the high-level takes this post argues for).

Context/Why I am writing about this

I helped set up and run the Cambridge Boston Alignment Initiative (CBAI) and the MIT AI Alignment group this past year. I also helped out with Harvard’s AI Safety team programming, along with some broader university AI safety programming (e.g. a retreat, two MLAB-inspired bootcamps, and a 3-week research program on AI strategy). Before this, I ran the Stanford Existential Risks Initiative and effective altruism student group and have supported many other university student groups.

Why AI Safety Field Building over EA Community Building

From my experiences over the past few months, it seems that AI safety field building is generally more impactful than EA movement building for people able to do either well, especially at the university level (under the assumption that reducing AI x-risk is probably the most effective way to do good, which I assume in this article). Here are some reasons for this:

AI-alignment-branded outreach is empirically attracting many more students with relevant skill sets and expertise than EA-branded outreach at universities.

Anecdotal evidence: At MIT, we received ~5x the number of applications for AI safety programming compared to EA programming, despite similar levels of outreach last year. This ratio was even higher when just considering applicants with relevant backgrounds and accomplishments. Around two dozen winners and top performers of international competitions (math/CS/science olympiads, research competitions) and students with significant research experience engaged with AI alignment programming, but very few engaged with EA programming.

This phenomenon at MIT has also roughly been matched at Harvard, Stanford, Cambridge, and I’d guess several other universities (though I think the relevant ratios are slightly lower than at MIT).

It makes sense that things marketed with a specific cause area (e.g. AI rather than EA) are more likely to attract individuals highly skilled, experienced, and interested in topics relevant to the cause area.

Effective cause-area specific direct work and movement building still involves the learning, understanding, and application of many important principles and concepts in EA:

Prioritization/Optimization are relevant,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Catastrophic Risks from AI #4: Organizational Risks, published by Dan H on June 26, 2023 on The AI Alignment Forum.

This is the fourth post in a sequence of posts giving an overview of catastrophic AI risks.

4 Organizational Risks

In January 1986, tens of millions of people tuned in to watch the launch of the Challenger Space Shuttle. Approximately 73 seconds after liftoff, the shuttle exploded, resulting in the deaths of everyone on board. Though tragic enough on its own, one of its crew members was a school teacher named Sharon Christa McAuliffe. McAuliffe was selected from over 10,000 applicants for the NASA Teacher in Space Project and was scheduled to become the first teacher to fly in space. As a result, millions of those watching were schoolchildren. NASA had the best scientists and engineers in the world, and if there was ever a mission NASA didn't want to go wrong, it was this one [70].

The Challenger disaster, alongside other catastrophes, serves as a chilling reminder that even with the best expertise and intentions, accidents can still occur. As we progress in developing advanced AI systems, it is crucial to remember that these systems are not immune to catastrophic accidents. An essential factor in preventing accidents and maintaining low levels of risk lies in the organizations responsible for these technologies. In this section, we discuss how organizational safety plays a critical role in the safety of AI systems. First, we discuss how even without competitive pressures or malicious actors, accidents can happen—in fact, they are inevitable. We then discuss how improving organizational factors can reduce the likelihood of AI catastrophes.

Catastrophes occur even when competitive pressures are low. Even in the absence of competitive pressures or malicious actors, factors like human error or unforeseen circumstances can still bring about catastrophe. The Challenger disaster illustrates that organizational negligence can lead to loss of life, even when there is no urgent need to compete or outperform rivals. By January 1986, the space race between the US and USSR had largely diminished, yet the tragic event still happened due to errors in judgment and insufficient safety precautions. Similarly, the Chernobyl nuclear disaster in April 1986 highlights how catastrophic accidents can occur in the absence of external pressures. As a state-run project without the pressures of international competition, the disaster happened when a safety test involving the reactor's cooling system was mishandled by an inadequately prepared night shift crew.

This led to an unstable reactor core, causing explosions and the release of radioactive particles that contaminated large swathes of Europe [71]. Seven years earlier, America came close to experiencing its own Chernobyl when, in March 1979, a partial meltdown occurred at the Three Mile Island nuclear power plant. Though less catastrophic than Chernobyl, both events highlight how even with extensive safety measures in place and few outside influences, catastrophic accidents can still occur. Another example of a costly lesson on organizational safety came just one month after the accident at Three Mile Island. In April 1979, spores of Bacillus anthracis—or simply "anthrax," as it is commonly known—were accidentally released from a Soviet military research facility in the city of Sverdlovsk. This led to an outbreak of anthrax that resulted in at least 66 confirmed deaths [72].

Investigations into the incident revealed that the cause of the release was a procedural failure and poor maintenance of the facility's biosecurity systems, despite being operated by the state and not subjected to significant competitive pressures.

The unsettling reality is that AI is far less understood and AI industry standards are far less stringent th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An update on the Spanish-speaking EA community, published by Jaime Sevilla on June 26, 2023 on The Effective Altruism Forum.

CoI warning: I am friends with many (most really!) of the people who I talk about. I direct Riesgos Catastróficos Globales, one of the organisations I talk about. I also still feel anger around the unfair situation that my partner, Sandra Malagón, had to experience. I do not speak in representation of the community or its members, but in personal capacity as a fairly involved member. I haven’t been following the community developments very closely since January, so it is likely I have missed stuff!

Last year I wrote a post celebrating the Spanish-speaking EA community and its achievements. A lot has happened since then, so I am writing here to organise my thoughts and provide some transparency on the state of the community.

My perception of the Spanish-speaking community has changed a lot since. Back then it felt like an integrated whole, with direction and momentum. Now it feels more like an ecosystem of independent projects, some with a lot of momentum and others languishing.

A major motivator of this is that we no longer have appointed coordinators who are steering the community. Sandra Malagón, who worked full-time as community coordinator, stepped down following a series of incidents where she felt pressured to work together with a community member she didn’t feel comfortable with . She is now working instead on virtual programs for Spanish Speakers. The other part-time coordinator, Laura González, stepped down to focus on other projects.

The effects of this can be seen in the activity stats of our shared workspace. Sandra left her position in early February, which matches the stop of activity growth:

This is hard to separate from other causes like the FTX collapse and sex scandals in the community, so take it with a pinch of salt.

Incidentally, if you are part of the Spanish-speaking community, I would like to apologise for the lack of communication about the status of the coordination team. This situation was and still is very hard to navigate.

Projects

The lack of direction does not mean that nothing has happened over the last few months. Here are a few notable projects and developments that have happened since I last posted.

The EA México fellowship program

The fellowship program received +80 participants from all around the globe who stayed over different periods between November 2022 and January 2023.

This program was, in my opinion, competently run. The organisation team, including Sandra Malagón, Hugo Ikta and Miguel Alvarado, showcased incredible attention to detail and problem solving skills. The fellowship was welcoming and diverse, and it is impressive that no major incidents resulted, given the scale of the program.

The impact of the program was, however, curtailed by a series of factors. First, they had to deal with the FTX scandal and sex scandals in the international EA community. Inhabiting an EA co-living space at that time was not fun. Second, Sandra renounced her position as community coordinator at the end of the fellowship, and the EA Mexico coordinators had distanced themselves from community building. The original vision of the fellowship, as I understood it, was to act as a catalyst for a community in CDMX, which could not be pursued given that nobody was in a position to do so.

This doesn’t mean that the fellowship was in vain. As a participant, I got a lot of value from connecting to other professionals. This resulted in a lead for a hire for Epoch, and I feel the time I spent in CDMX was very productive. My informal impression from talking to others is similar. I’ve been told a retrospective will be released soon-ish, look out for that!

EAGx LatAm

The first ever major Effective Altruism conference in Latin Ameri...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The fraught voyage of aligned novelty, published by Tsvi Benson-Tilsen on June 26, 2023 on The AI Alignment Forum.

[Metadata: crossposted from. First completed March 11, 2023.]

A voyage of novelty is fraught. If a mind takes a voyage of novelty, an observer is hard pressed to understand what the growing mind is thinking or what effects the growing mind will have on the world.

Fraughtness

If a mind grows beyond human activity then it thinks in a language you don't speak, using skills you don't possess, derived from sources you don't share. The mind incorporates new ideas, invalidating your predictions and specifications for how the mind affects the world.

"Fraught" is cognate with "freight": fraughtness is ladenness. "Fraught" is an intensive prefix "fra-" (cognate with English "for-", "fro", Ancient Greek "πρό-") followed by a descendant of Proto-Germanic "aiganą", meaning "to possess". From that etymon comes English "owe", "ought", and "own", and German "eigen". To be fraught is to be laden by intensely possessing.

Extended table of contents:

Understanding: If a mind takes a far-ranging voyage of novelty, it's difficult for an observer to understand what the mind thinks, knows, and does.

Neurath's ship: Novelty involves change. Some of that change deeply changes how the mind operates and how the mind connects its capabilities to the effects it has on the world. Previous understanding of the mind, and in particular previously correct models of the mind's effects, might not stay valid across these changes.

Alienness: A mind that gains a lot of structure not possessed by humans is alien to humans. It has structure (content, generators, contexts) not shared by humans. So it's hard for humans to understand the mind.

Inexplicitness: Structure in the mind is often unavailable for relation within the mind. Such structure tends to also be unavailable for relation with an observer.

Noncomprehensiveness: Creativity, inexplicitness, provisionality, noncartesianness, and integration tend to make a mind and its elements nonencompassable. The mind and its elements can't easily be understood in a way that implies conclusions that will be valid across a very wide range of possible future contexts.

Agency: Agency comes along with novelty. It comes along unannounced.

Pressure towards agency on a voyage of novelty: If a mind takes a far-ranging voyage of novelty, then it likely also has large effects on the world.

The murkiness of values: The existing concepts of values and agency don't explain what determines the direction of the effects that a mind has on the world or how an observer could specify that direction.

Understanding

If a mind takes a far-ranging voyage of novelty, it's difficult for an observer to understand what the mind thinks, knows, and does.

Neurath's ship

Novelty involves change. Some of that change deeply changes how the mind operates and how the mind connects its capabilities to the effects it has on the world. Previous understanding of the mind, and in particular previously correct models of the mind's effects, might not stay valid across these changes.

Creativity implies an influx of novel elements.

Some mental elements are essentially provisional: they're never complete, and there's always a further growth of the mind that will call on them to change, grow, or refactor. To continue to understand the role played by essentially provisional elements, as a mind grows, we may be required to grow in ways analogous to how the mind is growing.

Some novelty is diasystemic: it touches on many preexisting elements, and doesn't fit into preexisting elements in a way neatly analogous to how the preexisting elements fit in with each other. Diasystemic novelty may be necessary to create very capable minds, but more difficult to notice, appreciate as important, understand, and ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Safety Culture for AI" is important, but isn't going to be easy, published by Davidmanheim on June 26, 2023 on LessWrong.

This is a linkpost (to the EA forum version of this post, which is) for a new preprint, entitled "Building a Culture of Safety for AI: Perspectives and Challenges," and a brief explanation of the central points. Comments on the ideas in the post are welcome, but much of the content which clarifies the below is in the full manuscript.Safety culture in AI is going to be critical for many of the other promising initiatives for AI safety.

If people don't care about safety, most safety measures turn into box-ticking. Companies that don't care avoid regulation, or render it useless. That's what happens when fraudulent companies are audited, or when car companies cheat on emissions tests.

If people do care about safety, then audits, standards, and various risk-analysis tools can help get them there.

Culture can transform industries, and norms about trying to be safe can be really powerful as a way to notice and discourage bad actors.

However, there are lots of challenges to making such a culture.

Safety culture usually requires agreement about the risks. We don't have that in AI generally.

Culture depends on the operational environment.

When people have risks reinforced by always being exposed to them, or personally being affected by failures, they pay more attention. In AI, most risks are rare, occur in the future, and/or affect others more than the people responsible.

Most safety cultures are built around routines such as checklists and exercises that deal with current risks. Most AI risks aren't directly amenable to these approaches, so we can't reinforce culture with routines.

Cultures are hard to change once they get started.

AI gets cultural norms from academia, where few consider risks from their work, and there are norms of openness, and from the startup world, where companies generally want to "move fast and break things."

AI companies aren't prioritizing safety over profits - unlike airlines, nuclear power operators, or hospitals, where there is a clear understanding that safety is a critical need, and everything will stop if there is a safety problem.

Companies aren't hiring people who care about safety culture. But people build culture, and even if management wants to prioritize safety, lots of people who don't care won't add up to organizations that do care.

We need something other than routinization to reinforce safety culture.

Thankfully, there are some promising approaches, especially on the last point. These include identifying future risks proactively via various risk analysis methods, red-teaming, and audits. But as noted above, audits are most useful once safety culture is prioritized - though there is some promise in the near-term for audits to make lack of safety common knowledge.

Next steps include building the repertoire of tools that will reduce risks and can be used to routinize and inculcate safety culture in the industry, and getting real buy-in from industry leaders for prioritizing safety.

Thanks to Jonas Schuett, Shaun Ee, Simeon Campos, Tom David, Joseph Rogero, Sebastian Lodemann, and Yonaton Cale for helpful suggestions on the manuscript.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decision-making and decentralisation in EA, published by William MacAskill on June 26, 2023 on The Effective Altruism Forum.

This post is a slightly belated contribution to the Strategy Fortnight. It represents my personal takes only; I’m not speaking on behalf of any organisation I’m involved with. For some context on how I’m now thinking about talking in public, I’ve made a shortform post here. Thanks to the many people who provided comments on a draft of this post.

Intro and Overview

How does decision-making in EA work? How should it work? In particular: to what extent is decision-making in EA centralised, and to what extent should it be centralised?

These are the questions I’m going to address in this post. In what follows, I’ll use “EA” to refer to the actual set of people, practices and institutions in the EA movement, rather than EA as an idea.

My broad view is that EA as a whole is currently in the worst of both worlds with respect to centralisation. We get the downsides of appearing (to some) like a single entity without the benefits of tight coordination and clear decision-making structures that centralised entities have.

It’s hard to know whether the right response to this is to become more centralised or less. In this post, I’m mainly hoping just to start a discussion of this issue, as it’s one that impacts a wide number of decisions in EA. At a high level, though, I currently think that the balance of considerations tends to push in favour of decentralisation relative to where we are now.

But centralisation isn’t a single spectrum, and we can break it down into sub-components. I’ll talk about this in more depth later in the post, but here are some ways in which I think EA should become more decentralised:

Perception: At the very least, wider perception should reflect reality on how (de)centralised EA is. That means:

Core organisations and people should communicate clearly (and repeatedly) about their roles and what they do and do not take ownership for. (I agree with Joey Savoie’s post, which he wrote independently of this one.)

We should, insofar as we can, cultivate a diversity of EA-associated public figures.

[Maybe] The EA Forum could be renamed. (Note that many decisions relating to CEA will wait until it has a new executive director).

[Maybe] CEA could be renamed. (This is suggested by Kaleem here.)

Funding: It’s hard to fix, but it would be great to have a greater diversity of funding sources. That means:

Recruiting more large donors.

Some significant donor or donors start a regranters program.

More people pursue earning to give, or donate more (though I expect this “diversity of funding” consideration to have already been baked-in to most people’s decision-making on this). Luke Freeman has a moving essay about the continued need for funding here.

Decision-making:

Some projects that are currently housed within EV could spin out and become their own legal entities. The various different projects within EV have each been thinking through whether it makes sense for them to spin out. I expect around half of the projects will ultimately spin out over the coming year or two, which seems positive from my perspective.

[Maybe] CEA could partly dissolve into sub-projects.

Culture:

We could try to go further to emphasise that there are many conclusions that one could come to on the grounds of EA values and principles, and celebrate cases where people pursue heterodox paths (as long as their actions are clearly non-harmful).

Here are some ways in which I think EA could, ideally, become more centralised (though these ideas crucially depend on someone taking them on and making them happen):

Information flow:

Someone could create a guide to what EA is, in practice: all the different projects, and the roles they fill, and how they relate to one another.

Someone c...