Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I hate the "accident vs. misuse" AI x-risk dichotomy (quick thoughts on "structural risk"), published by David Scott Krueger on January 30, 2023 on The AI Alignment Forum.

I think the large majority of AI x-risk is "structural". Like climate change.

Here's a good primer on structural risk (note that structural risk is not a synonym for "not caused by out-of-control AI"):

I am shocked and amazed and dismayed that more people do not seem to view it this way, even among the AI x-safety community. Heck, even Eliezer's stories of doom are steeped in structural risk (race dynamics, teams rationalizing cutting corners on safety when they should know better, etc.)

I expect irresponsible, reckless, negligent deployment of AI systems without proper accounting of externalities. I consider this the default for any technology with potential for significant externalities, absent regulation.When something bad happens in such a context, calling it "accident risk" absolves those researching, developing, and/or deploying the technology of responsibility. They should have known better. Some of them almost certainly did. Rationalization, oversight, and misaligned incentives were almost certainly at play. Failing to predict the particular failure mode encountered is no excuse. Having "good intentions" is no excuse.So... it must be misuse then, right? Well, no. Calling it "misuse" suggests that those researching, developing, and/or deploying the technology set out with nefarious purposes and the technology achieved precisely what they intended. But ~nobody wants to destroy the world.

It's just that most people are somewhat selfish and so are willing to trade some x-risk for a large personal benefit.In summary, saying "accident" makes it sounds like an unpredictable effect, instead of painfully obviously risk that was not taken seriously enough. Saying "misuse" makes it sounds like some supervillian or extremist deliberately destroying the world. While some risks may have something more of a flavor or accident or misuse depending on how obvious the risk was, neither of these pictures gives a remotely accurate picture of the nature of the problem. I think this makes it a harmful meme, and ask that others stop making this distinction (without appropriate caveats), and join me in pointing out how it contributes to a confused and misleading discourse when others do.

EtA: Many people have responded that "accident" does not connote "unforseen" or "not negligent", etc., and instead it should simply be interpreted as something like "a result that was not deliberately selected for". While it can be used this way, I basically disagree that this is how it is usually used, see below:EtA: as an additional clarification: my main objection is not to the use of "accident" and "misuse", but rather to their use as a dichotomy. Every use of these terms I can recall seeing in writing (other than those that mention structural risk) supports this dichotomy, and it is often made explicitly.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.