Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A plea for solutionism on AI safety, published by jasoncrawford on June 9, 2023 on LessWrong.

[Note for LW: This essay was written mostly for people who are inclined to downplay or dismiss AI risk, which means it has almost no intersection with this audience. Cross-posting it here anyway for feedback and in case you want to point anyone to it.]

Will AI kill us all?

This question has rapidly gone mainstream. A few months ago, it wasn’t seriously debated very far outside the rationalist community of LessWrong; now it’s reported in major media outlets including the NY Times, The Guardian, the Times of London, BBC, WIRED, Time, Fortune, U.S. News, and CNBC.

For years, the rationalists lamented that the world was neglecting the existential risk from AI, and despaired of ever convincing the mainstream of the danger. But it turns out, of course, that our culture is fully prepared to believe that technology can be dangerous. The reason AI fears didn’t go mainstream earlier wasn’t society’s optimism, but its pessimism: most people didn’t believe AI would actually work. Once there was a working demo that got sufficient publicity, it took virtually no extra convincing to get people to be worried about it.

As usual, the AI safety issue is splitting people into two camps. One is pessimistic, often to the point of fatalism or defeatism: emphasizing dangers, ignoring or downplaying benefits, calling for progress to slow or stop, and demanding regulation. The other is optimistic, often to the point of complacency: dismissing the risks, and downplaying the need for safety.

If you’re in favor of technology and progress, it is natural to react to fears of AI doom with worry, anger, or disgust. It smacks of techno-pessimism, and it could easily lead to draconian regulations that kill this technology or drastically slow it down, depriving us all of its potentially massive benefits. And so it is tempting to line up with the techno-optimists, and to focus primarily on arguing against the predictions of doom. If you feel that way, this essay is for you.

I am making a plea for solutionism on AI safety. The best path forward, both for humanity and for the political battle, is to acknowledge the risks, help to identify them, and come up with a plan to solve them. How do we develop safe AI? And how do we develop AI safely?

Let me explain why I think this makes sense even for those of us who strongly believe in progress, and secondarily why I think it’s needed in the current political environment.

Safety is a part of progress

Humanity inherited a dangerous world. We have never known safety: fire, flood, plague, famine, wind and storm, war and violence, and the like have always been with us. Mortality rates are high as far back as we can measure them. Not only was death common, it was sudden and unpredictable. A shipwreck, a bout of malaria, or a mining accident could kill you quickly, at any age.

Over the last few centuries, technology has helped make our lives more comfortable and safer. But it also created new risks: boiler explosions, factory accidents, car and plane crashes, toxic chemicals, radiation.

When we think of the history of progress and the benefits it has brought, we should think not only of wealth measured in economic production. We should think also of the increase in health and safety.

Safety is an achievement. It is an accomplishment of progress—a triumph of reason, science, and institutions. Like the other accomplishments of progress, we should be proud of it—and we should be unsatisfied if we stall out at our current level. We should be restlessly striving for more. A world in which we continue to make progress should be not only a wealthier world, but a safer world.

We should (continue to) get more proactive about safety

Long ago, in a more laissez-faire world, technology c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A survey of concrete risks derived from Artificial Intelligence, published by Guillem Bas on June 9, 2023 on The Effective Altruism Forum.

Riesgos Catastróficos Globales has conducted a literature review and an expert elicitation exercise to categorize concrete risks associated with Artificial Intelligence (AI). This is part of our ongoing work on the implementation of the EU AI Act in Spain.

Here we present a short overview of the risks we have found. This is meant to be a mental framework for policymakers to consider when developing AI policies, but we think it might also be useful to incentivize discussion within the community. Please feel free to leave your thoughts as comments!

To facilitate comprehension, we have split the identified risks into two categories: adversarial and structural risks. Adversarial risks are those caused by the direct action of an agent, be it rogue groups, state actors, or misaligned AI. Structural risks are those derived from the wide-scale or high-impact deployment of AI, with diffuse causes.

The distinction builds upon the categorization between accidents, misuse, and structural risks (Zwetsloot & Dafoe, 2019). We preferred to merge the first two because we considered there was not always a clear difference in how accidents (AI misalignment) and misuses (humans exploiting an AI system to cause harm) materialize as specific threats.

As for this materialization, we outline risks integrating present and future implications. That is to say, we state that their long-term impact is potentially large, but we ground them on existing and modest evidence. This choice is based on the assumption that policymakers will tend to underestimate speculative framings. The underlying logic we try to convey is that damage will increase along with capabilities and deployment.

We have identified nine concrete risks within these categories, which are summarized in the table below. The categorization is not perfect, but we tried to prioritize clarity and concreteness over accuracy and exhaustiveness.

Risk category

Risk

Example vignette

Adversarial risks: directly caused by agents, either humans or misaligned AI

Cyberattacks and other unauthorized access

LLM-enabled spear-phishing campaigns

Strategic technology development

Development of a new biological weapon

User manipulation

Individuals persuaded to support a certain political option

Structural risks: caused by widespread automation

Job market disruption

10% increase in unemployment over a year

Socioeconomic inequality

Leading companies capturing AI-created surpluses

Bias amplification

Minority groups being systematically denied access to housing or loans

Epistemic insecurity

Proliferation of deep fakes

Faulty automation of critical processes

Accidental nuclear attack from fully-automated C&C

Defective optimization

Hospitals rejecting patients with serious conditions to maximize performance metrics

We briefly introduce these risks below, together with references for further reading.

Adversarial risks

This section compiles potential threats from rogue human actors and misaligned AI. The final list coincides with what Shevlane et al. (2023) call "extreme risks" and is slightly connected to the distinction between digital, physical, and political dimensions proposed by Brundage et al. (2018).

Readers might note that our selected risks are commonly mentioned as instances of power-seeking behavior. We have not included vignettes about goal mispecification and misgeneralization for two reasons: they tended to be too vague and, to be impactful, most of them required the instrumental use of the actions listed below.

Cyberattacks and other unauthorized access

AI promises to enhance the execution of cyber offenses, increasing their scale and impact (Brundage, et al., 2018). New tools can automate manual tasks (see ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Strawmen, steelmen, and mithrilmen: getting the principle of charity right, published by MichaelPlant on June 9, 2023 on The Effective Altruism Forum.

Something you hear a lot in discussions is that it's important not to strawman arguments: to assume they are much weaker than they are. This is uncharitable.

Instead, the suggestion is you should steelman arguments: consider the strongest version of it, even if that's not what the person has said, and then evaluate that. Steelmanning is thought of as getting the principle of charity just right.

However, I suspect there should be a third category, call it mithrilman: where arguments are treated as much stronger than they are, and you accept them even though you don't understand the reasons. For the non-nerds out there, mithril is Tolkien's fictional super-strong metal in Lord of the Rings.

Whilst strawmanning is being too uncharitable, mithrilmanning is being too charitable. You don't want to do either. Goldilocks lies inbetwixt the two.

I see mithrilmanning quite a lot among effective altruists. Usually, it goes something like this. People are discussing a view or argument they've heard person X make. The individuals are sitting around, brows furrowed, and struggling to find a good steelman of the argument: they can't work out what plausible reasons that person could have for their conclusion. After a while, even though they can't find a suitable steelman, someone says "Well, X does seem really smart, so...". Everyone nods. The conversation moves on.

What's happened is that someone has suggested the group should defer, even though they can't follow the reasoning or provide it themselves. This seems to happen much more often when person X is important (not least because you don't want to risk looking stupid).

I think there can be good cases where one should defer, but I'm worried I see too much of this. We should give people the benefit of the doubt - assume they are smart, thoughtful, etc. rather than fools - but we should still doubt. To err is human. We all make mistakes. We make progress by pointing those out.

So, if you think someone is really smart, but you can't make sense of what they are thinking, at least hesitate on deferring to them. If possible, ask them to explain. It seems too charitable to assume they are right, not charitable enough to assume they are wrong. In assuming that they can give you a sensible answer, you are treating them with appropriate charity.

I don't think I need to say why strawmanning is bad. The danger of mithrilmanning is you end up with too much deference, an information cascade and ultimately false beliefs. People end up believing what X says, even though no one really understands why.

So, if you find yourself overhearing, or even saying yourself, "well, they do seem really smart..." consider adding "um, are we mithrilmanning this? We don't want to defer uncritically."

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How biosafety could inform AI standards, published by Olivia Jimenez on June 9, 2023 on The AI Alignment Forum.

Context: Simeon Campos and I wanted to understand how other industries handle catastrophic risks, then write pieces about how any standards from them could be applied to AI for policy readers. This is a draft slightly modified for LessWrong.

Epistemic status: Very shallow dive, though I checked my understanding with people with more biosecurity and policy expertise, like David Manheim (thanks!).

Related posts I recommend: Six Dimensions of Operational Adequacy in AGI Projects, High Reliability Orgs, and AI Companies, “Carefully-Bootstrapped Alignment” is organizationally hard

How biosafety could inform AI standards

Leading AI executives and researchers recently signed a statement saying “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

If we take this seriously, it’s worth looking at how much we do to mitigate pandemic and nuclear risk. (Spoiler: a lot.)

This post addresses the first half of that, outlining my understanding of biosafety standards and how they could inform standards for high-risk AI research.

Biosafety & its relevance

The field of biology has comprehensive standards for maintaining safety while working with potentially dangerous biological agents, such as viruses that could escape the lab and infect many people.

Biosafety standards are applied in a tiered approach, where the riskiest research can only occur in laboratories taking the most stringent precautions. The levels of laboratory precautions range from biosafety level 1 (BSL-1, the lowest) to biosafety level 4 (BSL-4, the highest). BSL standards also govern the storage and transportation of biological agents, access to these labs, and similar.

BSL-1 has minimal standards; it allows places like high school labs to easily experiment. BSL-4 has very restrictive standards; there are only around 50 BSL-4 labs in the world. BSL-4 labs are the only places where easily-transmissible and lethal diseases like Ebola can be studied.

By enforcing tiered standards according to research risk at all labs, biology has been relatively successful at mitigating the two types of risks it faces:

accident risks, such as a virus escaping the lab because it was accidentally mishandled, and

misuse risk, such as a virus being intentionally taken out of the lab by someone who intends to cause harm with it.

There are notable parallels to AI here. Different AI research poses different risks, and most are easily manageable. However, research on the most powerful systems poses significant accident and misuse risks that are difficult to manage (in some cases, we don’t even know how we could manage them): powerful systems could get out of developers control or they could be used to cause harm. Accordingly, labs conducting that research should probably be held to particularly high standards of reliability and security.

Below I outline some standards for US biosafety level 3 and 4 labs, which handle pathogens that pose catastrophic risks. Other countries' standards at these levels are very similar, so I haven’t taken the time to spell out differences.

Selected biosafety standards & potential applications in AI

1. High-risk research must be conducted in designated labs subject to stringent standards.

The USDA maintains a list of biological agents that pose severe public health and safety threats (henceforth select agents). Any lab wishing to work with a select agent must be licensed by the Federal Select Agent Program (FSAP). These facilities are continuously monitored for compliance.

Additionally, each BSL-4 research project must get specific approval before it can be conducted. Researchers must submit detailed research plans an...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: InternLM - China's Best (Unverified), published by Lao Mein on June 9, 2023 on LessWrong.

A technical report of InternLM on 6/7. It consisted of 104 billion parameters, was trained on 1.6 trillion tokens, and was fine-tuned for performance in Chinese.

The authors claimed that it performed second-best on the Chinese language benchmark C-Eval, right after GPT4. In addition, it performed at the level of GPT3.5 in one-shot MMLU. A version fine-tuned for programming also performed similarly to GPT3.5 in coding benchmarks like HumanEval.

Notable takeaways:

Significant effort was put into parallelization to help evade the US chip ban. I don't know how impressive this actually is.

It achieved GPT3.5-level performance with similar-ish levels of compute and data. The China-America algorithmic gap is shrinking.

My gut feeling is that the model was very specifically fine-tuned for performing well on standardized tests, especially those in Chinese (GK/Gao Kao is the Chinese College entrance exam). It was also consistently bad with math.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates and Reflections on Optimal Exercise after Nearly a Decade, published by romeostevensit on June 8, 2023 on LessWrong.

Previously:

Firstly, do the basic epistemics hold up? As far as I know, yes. The basic idea that lifting twice a week and doing cardio twice a week add up to a calorie expenditure that get you the vast majority of exercise benefits compared to extreme athletes hold up, especially when you take reverse causality adjustments into effect (survivorship bias on the genetic gifts of the extreme). Nothing I've encountered since has cast much doubt on this main takeaway.

What updates have I had, then, both in personal experience and in giving training advice to others as well as any research that has come out since then?

A greater emphasis on injury prevention, as the dis-utility from injuries vastly outweighs positive effects from chasing numbers. This one was sadly a foreseeable update with aging, and thus I lose bayes points for it. I did in fact get an injury deadlifting despite a substantial emphasis on good form and not pushing to the limit as many do.

Exercise selection and program optimization likely matter less than I thought, and research that has come out in the meantime has supported this.

One and two combined imply that there is no real downside to picking exercises with lower injury potential for the joints and back.

WRT to cardio, running is a high impact activity and people shouldn't feel bad about choosing lower impact rowing, swimming, or biking (though biking near cars likely eliminates a lot of the health effects in expectation as it is somewhat dangerous). If you can run, great, I enjoy it quite a lot, but I also don't particularly like the hedonic gradient of pushing yourself to run at the volume and frequency that seems necessary to really git gud (many runners run 5-6 days a week).

WRT resistance training, I don't pursue any of the powerlifts (squat, bench, deadlift) anymore, instead focusing on other exercises that don't load the spine/knees as much but allow you to load the requisite musculature easily. Weighted step ups instead of squats can be loaded quite heavy. Hyperextensions, one-legged hypers, and reverse hyperextensions can work the posterior chain with 1/2-1/3 the load on the spine as deadlifts. Bench doesn't exactly load the spine but it is the most dangerous lift going by statistics (dropping the weight on yourself is the most common severe gym accident) and can be replaced with incline bench, dumbbell shoulder presses, and/or dips. These exercises are substantially easier to cue people on in a single session.

Many people IME push too hard, on the intuitive model of suffering being virtuous and health being virtuous and therefore... Or something like that. People are generally surprised that going a rep or two shy of failure is still driving very large benefits. Never pushing your edge isn't too satisfying either of course, but it's something that can be felt into over time.

Many people IME find exercise lacks intrinsic motivation because it doesn't feel like it's 'for' anything else is their life that they care about. WRT this, novelty, community, and skill acquisition winds up being very important relative to more sterile notions of optimal performance. Health is synergistic, and wanting better performance in a sport or activity drives a variety of other motivations from exercise choice, to diet, to sleep. A related thing is that home gyms seem intuitively appealing from an efficiency perspective, but many find they de facto do not exercise as much without the social reinforcement and teleological coherence of the gym environment.

One of the top voted comments on the original post was about training the nervous system, and indeed I've spent substantial efforts here over the last several years to enormous positive ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeaways from the Mechanistic Interpretability Challenges, published by scasper on June 8, 2023 on LessWrong.

...plus more challenges on the way

Stephen Casper, [email protected]

Spoilers ahead

What happened?

The Mechanistic Interpretability Challenges (post and GitHub) were two challenges I posed in February as part of the Engineer’s Interpretability Sequence. The first challenge was to find the pseudocode for the labeling function used to train a small CNN MNIST classifier. It was solved early last month. The second was to find the pseudocode for the labeling function used to train a one-layer transformer that classified pairs of numbers into two categories. It was solved (with some reservations) late last month. Instead of finding the labeling function, the researchers who solved it obtained a mechanistic explanation of how the model worked and argued that the labeling function’s pseudocode would not be tractable to find from the model.

Thanks to Stefan, Marius, and Neel

Stefan Heimersheim and Marius Hobbhahn solved both challenges as a team. I and others have been impressed with their work. Meanwhile, Neel Nanda offered to contribute $500 to the prize pool for solving each challenge. Per the request of Stefan and Marius, a total of $1,500 has been donated by Neel and me to AI Safety Support.

Why These Challenges?

In the original post on the challenges, I argued that solving them would be one of the first clear examples of mechanistic interpretability being used to solve a problem that was not specifically selected to be solvable with mechanistic interpretability.

Because it doesn’t treat models as black boxes, mechanistic interpretability is one of the potential solutions we might have for diagnosing and debugging insidious alignment failures. For example, if a model has a trojan or plans to make a treacherous turn once it detects that it’s in deployment, then these failures will be virtually undetectable from black-box access alone during training and development.

Mechanistic interpretability has been a reasonably high-profile research area for the past 6 years or so in the AI safety community. And it is currently undergoing a renewed surge of interest. However, I have tried to be critical of the fact that much of the progress in mechanistic interpretability research has been on “streetlight interpretability” projects often with cherrypicked models and tasks. As a result of this, there is a risk that, if mechanistic interpretability continues to be a field full of cherrypicked and toy work, it may fail to produce methods that keep up with state-of-the-art applications of AI. Certainly, progress in mechanistic interpretability has not kept up with progress in AI as a whole, and despite all of the interest from the AI safety community, it lacks any big wins or many real-world applications at all that produce competitive tools for engineers solving real-world problems.

Hence the purpose of the mechanistic interpretability challenges: to provide challenges that aren’t able to be cherrypicked by those undertaking them. The hope has been that these challenges and others like it could offer a useful way of testing how useful approaches to interpretability are. The goal is to measure how promising specific methods and mechanistic interpretability itself are for truly reverse engineering models performing tasks that don’t happen to be under any particular streetlight.

The First Challenge: A Clear Win for MI

As is now public information, the MNIST CNN was trained on a labeling function that labeled images with small and large L1 distances to this image as a 1 while images with a medium L1 distance from it were labeled as a 0.

The solution was thorough. The network developed a set of “detectors” and “anti-detectors” for this image in the penultimate layer. It labeled anythi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeaways from the Mechanistic Interpretability Challenges, published by Stephen Casper on June 8, 2023 on The AI Alignment Forum.

...plus more challenges on the way

Stephen Casper, [email protected]

Spoilers ahead

What happened?

The Mechanistic Interpretability Challenges (post and GitHub) were two challenges I posed in February as part of the Engineer’s Interpretability Sequence. The first challenge was to find the pseudocode for the labeling function used to train a small CNN MNIST classifier. It was solved early last month. The second was to find the pseudocode for the labeling function used to train a one-layer transformer that classified pairs of numbers into two categories. It was solved (with some reservations) late last month. Instead of finding the labeling function, the researchers who solved it obtained a mechanistic explanation of how the model worked and argued that the labeling function’s pseudocode would not be tractable to find from the model.

Thanks to Stefan, Marius, and Neel

Stefan Heimersheim and Marius Hobbhahn solved both challenges as a team. I and others have been impressed with their work. Meanwhile, Neel Nanda offered to contribute $500 to the prize pool for solving each challenge. Per the request of Stefan and Marius, a total of $1,500 has been donated by Neel and me to AI Safety Support.

Why These Challenges?

In the original post on the challenges, I argued that solving them would be one of the first clear examples of mechanistic interpretability being used to solve a problem that was not specifically selected to be solvable with mechanistic interpretability.

Because it doesn’t treat models as black boxes, mechanistic interpretability is one of the potential solutions we might have for diagnosing and debugging insidious alignment failures. For example, if a model has a trojan or plans to make a treacherous turn once it detects that it’s in deployment, then these failures will be virtually undetectable from black-box access alone during training and development.

Mechanistic interpretability has been a reasonably high-profile research area for the past 6 years or so in the AI safety community. And it is currently undergoing a renewed surge of interest. However, I have tried to be critical of the fact that much of the progress in mechanistic interpretability research has been on “streetlight interpretability” projects often with cherrypicked models and tasks. As a result of this, there is a risk that, if mechanistic interpretability continues to be a field full of cherrypicked and toy work, it may fail to produce methods that keep up with state-of-the-art applications of AI. Certainly, progress in mechanistic interpretability has not kept up with progress in AI as a whole, and despite all of the interest from the AI safety community, it lacks any big wins or many real-world applications at all that produce competitive tools for engineers solving real-world problems.

Hence the purpose of the mechanistic interpretability challenges: to provide challenges that aren’t able to be cherrypicked by those undertaking them. The hope has been that these challenges and others like it could offer a useful way of testing how useful approaches to interpretability are. The goal is to measure how promising specific methods and mechanistic interpretability itself are for truly reverse engineering models performing tasks that don’t happen to be under any particular streetlight.

The First Challenge: A Clear Win for MI

As is now public information, the MNIST CNN was trained on a labeling function that labeled images with small and large L1 distances to this image as a 1 while images with a medium L1 distance from it were labeled as a 0.

The solution was thorough. The network developed a set of “detectors” and “anti-detectors” for this image in the penultimate laye...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on how I want to handle criticism, published by Lizka on June 8, 2023 on The Effective Altruism Forum.

Tl;dr/epistemic status: quick post mostly comprised of bullet points.

I have a pretty public-facing role, so I get a fair amount of feedback (both negative and positive) on my work from different people. A while back, I wrote a doc for myself on how I want to respond to this kind of negative feedback (and related stuff). Then I shared the doc with some folks I know and work with to get input on it, then never shared it further. There’s been more related discussion recently, so I thought I’d share it now. I haven’t really updated it except to add the introductory notes (before the first actual section), some links, and a picture.

Sections:

Notes on criticism

How I want to handle criticism

What I want to do to shift towards handling criticism better

Bonus: notes on how to share criticism in ways that make this process easier for the one being criticized

I use the word “criticism” here for a pretty vague/broad class of things that includes things like “negative feedback” and “people sharing that they think I’m wrong in some important way.” (Other things are also arguably “criticism,” like people pointing out specific minor errors, but those are easier for me to handle, so I wasn’t really focusing on them when I wrote this.)

Please don't interpret this as a request to stop sharing feedback! Feedback has often helped me improve my work and grow. (I do think there are better and worse ways of sharing feedback, though — see more below.)

Notes on criticism

These were kind of used like lemmas for the original doc — claims to help arrive at my ~conclusions

What’s the point of (engaging with) criticism?

To improve what I’m working on now

To learn something I might be able to use in the future

[Something about supporting communal norms around engaging with good-faith criticism.]

Sometimes criticism isn’t really criticism and shouldn’t be (it could be reframed as a suggestion)

Criticism can be wrong

(Fear of) criticism can discourage action in bad ways

How I want to handle criticism

Steelman it before updating on it, or try to understand the other person’s point of view if I first want to respond to it.

Consider it rationally, and avoid over-updating on it. (Criticism can be wrong!)

Actually listen to it, and stare into the abyss if the criticism is potentially scary.

Welcome it and be grateful for it, and avoid pre-emptively guarding against it by being vague.

Acknowledge explicitly (to myself and ideally to the person criticizing me) where I was wrong, if I was.

Avoid feeling like I need to justify everything to everyone, especially when this is taking time or energy.

Avoid letting fear of criticism stop me from doing things I endorse doing.

Interpret it in a productive way — what should I learn, how should I change my behavior (if at all) — and avoid immediately judging myself, worrying that it means that I’m terrible in some deep important way. (Related: "Flinching away from truth” is often about protecting the epistemology.)

Relatedly, avoid over-interpreting: avoid thinking that whoever is passing this on must think that I’m terrible in many other ways.

If it’s very aggressive, get some help dealing with it.

Allow myself to engage with it in healthy ways, e.g. don’t go head-first into the criticism out of a misguided impulse towards ~epistemic bravado.

Sometimes I’m emotionally overwhelmed. It’s ok to listen to and process the criticism later. If someone is giving me feedback, I can ask them to write it down and send it a little later. If I have an email, I can snooze it until next week. A method that a friend proposed but that I’ve never tried is asking a friend to read the criticism and to contextualize it for me. (I assume the process is something like “this is pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A moral backlash against AI will probably slow down AGI development, published by geoffreymiller on June 7, 2023 on LessWrong.

Note 1 : This was posted to EA Forum on May 31 here, as a submission for the 2023 Open Philanthropy AI Worldviews contest, due May 31, 2023. It addresses Question 1: “What is the probability that AGI is developed by January 1, 2043?”

Note 2: I'm developing a series of several more detailed essays on active moral stigmatization of AI as a strategy for pausing, slowing, or stopping dangerous AI development; so I'd welcome any constructive feedback on this initial essay.

Overview

People tend to view harmful things as evil, and treat them as evil, to minimize their spread and impact. If enough people are hurt, betrayed, or outraged by AI applications, or lose their jobs, professional identity, and sense of purpose to AI, and/or become concerned about the existential risks of AI, then an intense public anti-AI backlash is likely to develop. That backlash could become a global, sustained, coordinated movement that morally stigmatizes AI researchers, AI companies, and AI funding sources. If that happens, then AGI is much less likely to develop by the year 2043. Negative public sentiment could be much more powerful in slowing AI than even the most draconian global regulations or formal moratorium, yet it is a neglected factor in most current AI timelines.

Introduction

The likelihood of AGI being developed by 2043 depends on two main factors: (1) how technically difficult it will be for AI researchers to make progress on AGI, and (2) how many resources – in terms of talent, funding, hardware, software, training data, etc. – are available for making that progress. Many experts’ ‘AI timelines’ for predicting AI development assume that AGI likelihood will be dominated by the first factor (technical difficulty), and assume that the second factor (available resources) will continue increasing.

In this essay I disagree with that assumption. The resources allocated into AI research, development, and deployment may be much more vulnerable to public outrage and anti-AI hatred than the current AI hype cycle suggests. Specifically, I argue that ongoing AI developments are likely to provoke a moral backlash against AI that will choke off many of the key resources for making further AI progress. This public backlash could deploy the ancient psychology of moral stigmatization against our most advanced information technologies. The backlash is likely to be global, sustained, passionate, and well-organized. It may start with grass-roots concerns among a few expert ‘AI doomers’, and among journalists concerned about narrow AI risks, but it is likely to become better-organized over time as anti-AI activists join together to fight an emerging existential threat to our species.

(Note that this question of anti-AI backlash likelihood is largely orthogonal to the issues of whether AGI is possible, and whether AI alignment is possible.)

I’m not talking about a violent Butlerian Jihad. In the social media era, violence in the service of a social cause is almost always counter-productive, because it undermines the moral superiority and virtue-signaling strategies of righteous activists. (Indeed, a lot of ‘violence by activists’ turns out to be false flag operations funded by vested interests to discredit the activists that are fighting those vested interests.)

Rather, I’m talking about a non-violent anti-AI movement at the social, cultural, political, and economic levels. For such a movement to slow down the development of AGI by 2043 (relative to the current expectations of Open Philanthropy panelists judging this essay competition), it only has to arise sometime in the next 20 years, and to gather enough public, media, political, and/or investor support that it can handicap...