Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How biosafety could inform AI standards, published by Olivia Jimenez on June 9, 2023 on The AI Alignment Forum.

Context: Simeon Campos and I wanted to understand how other industries handle catastrophic risks, then write pieces about how any standards from them could be applied to AI for policy readers. This is a draft slightly modified for LessWrong.

Epistemic status: Very shallow dive, though I checked my understanding with people with more biosecurity and policy expertise, like David Manheim (thanks!).

Related posts I recommend: Six Dimensions of Operational Adequacy in AGI Projects, High Reliability Orgs, and AI Companies, “Carefully-Bootstrapped Alignment” is organizationally hard

How biosafety could inform AI standards

Leading AI executives and researchers recently signed a statement saying “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

If we take this seriously, it’s worth looking at how much we do to mitigate pandemic and nuclear risk. (Spoiler: a lot.)

This post addresses the first half of that, outlining my understanding of biosafety standards and how they could inform standards for high-risk AI research.

Biosafety & its relevance

The field of biology has comprehensive standards for maintaining safety while working with potentially dangerous biological agents, such as viruses that could escape the lab and infect many people.

Biosafety standards are applied in a tiered approach, where the riskiest research can only occur in laboratories taking the most stringent precautions. The levels of laboratory precautions range from biosafety level 1 (BSL-1, the lowest) to biosafety level 4 (BSL-4, the highest). BSL standards also govern the storage and transportation of biological agents, access to these labs, and similar.

BSL-1 has minimal standards; it allows places like high school labs to easily experiment. BSL-4 has very restrictive standards; there are only around 50 BSL-4 labs in the world. BSL-4 labs are the only places where easily-transmissible and lethal diseases like Ebola can be studied.

By enforcing tiered standards according to research risk at all labs, biology has been relatively successful at mitigating the two types of risks it faces:

accident risks, such as a virus escaping the lab because it was accidentally mishandled, and

misuse risk, such as a virus being intentionally taken out of the lab by someone who intends to cause harm with it.

There are notable parallels to AI here. Different AI research poses different risks, and most are easily manageable. However, research on the most powerful systems poses significant accident and misuse risks that are difficult to manage (in some cases, we don’t even know how we could manage them): powerful systems could get out of developers control or they could be used to cause harm. Accordingly, labs conducting that research should probably be held to particularly high standards of reliability and security.

Below I outline some standards for US biosafety level 3 and 4 labs, which handle pathogens that pose catastrophic risks. Other countries' standards at these levels are very similar, so I haven’t taken the time to spell out differences.

Selected biosafety standards & potential applications in AI

1. High-risk research must be conducted in designated labs subject to stringent standards.

The USDA maintains a list of biological agents that pose severe public health and safety threats (henceforth select agents). Any lab wishing to work with a select agent must be licensed by the Federal Select Agent Program (FSAP). These facilities are continuously monitored for compliance.

Additionally, each BSL-4 research project must get specific approval before it can be conducted. Researchers must submit detailed research plans an...