Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Complex Systems are Hard to Control, published by jsteinhardt on April 4, 2023 on LessWrong.

The deployment of powerful deep learning systems such as ChatGPT raises the question of how to make these systems safe and consistently aligned with human intent. Since building these systems is an engineering challenge, it is tempting to think of the safety of these systems primarily through a traditional engineering lens, focusing on reliability, modularity, redundancy, and reducing the long tail of failures.

While engineering is a useful lens, it misses an important part of the picture: deep neural networks are complex adaptive systems, which raises new control difficulties that are not addressed by standard engineering methodology. I’ve discussed some particular examples of this before, but here I want to focus on the broader underlying intuition that generated them.

A complex adaptive system is a system with many interacting components that adapt to their environment and co-evolve over time (in our case, the weights / layers of the neural network). Beyond neural networks, other examples of complex adaptive systems include firms, financial markets, political parties, culture, traffic flows, pathogens, ecosystems, human brains, the Earth’s climate, and the Internet.

A common thread in all these systems is that straightforward attempts to control their behavior lead to unintended consequences. I’ll demonstrate this through concrete examples, then step back and consider the broader properties that make these systems difficult to control, including emergent goals. Finally, I’ll propose safety measures that account for the complex adaptive nature of deep learning systems.

Many of the ideas in this post have been discussed before, and my thinking owes significantly to Dan Hendrycks, who was an early proponent of the complex systems perspective as a PhD student in my lab (see e.g. Unsolved Problems in ML Safety, the lecture on accident models from Dan’s course, or this blog post).

Control Difficulties in Complex Systems

Let’s examine several examples of complex systems, and see why each is difficult to control, in the sense that they either resist or respond unpredictably to external feedback.

Traffic. A city builds new highways to reduce traffic congestion. The newly increased road capacity attracts new drivers, leading to worse levels of congestion than before. The adaptive behavior leads to unintended consequences.

Ecosystems. A park introduces a predator to reduce the population of an invasive species. The predator also preys on native species, disrupting the ecosystem balance. The dense network of interactions makes it difficult to predict all consequences ahead of time.

Financial markets. Central banks lower interest rates to stimulate economic growth. Funders thus make riskier investments leading to asset bubbles, which later burst and destabilize the financial system. In this case, both adaptivity and multi-step interactions come into play.

Culture. The government implements public awareness campaigns to promote environmental conservation. These efforts encounter resistance from workers whose jobs rely on non-renewable fuel sources, and are appropriated by fashion brands and other consumer products through greenwashing.

Further examples include pathogens evolving drug resistance, firms relocating to avoid regulations, and positive feedback loops from climate change. I elaborate on these and other examples in the appendix.

Traditional Engineering Does not Address these Difficulties

Why are complex adaptive systems hard to control? There are two key hallmarks of complex adaptive systems:

Emergence: behavior at one scale cannot be easily reduced to behavior at smaller scales, i.e. “More is Different”.

Feedback loops: different components of the system continually influence ...