Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Reluctant Prophet of Effective Altruism | The New Yorker, published by Joseph Lemien on August 9, 2022 on The Effective Altruism Forum.

There appears to be a new profile published in The New Yorker on Will MacAskill, which as it unfolds ends up being as much of a profile of Effective Altruism as of Will MacAskill. Several other people who are well known or impactful in the EA community are mentioned or are interviewed (Julia Wise, Holden Karnofsky, Toby Ord, and others).

As far as I am aware, it seems to be one of the biggest pieces about of EA that in the popular press, and The New Yorker is a very well-respected publication. I know that there has been a fair amount of worry, thought, concern, and other feelings about publicity that EA has gotten in the past. How do you all feel about this piece?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How (not) to choose a research project, published by Garrett Baker on August 9, 2022 on LessWrong.

Background

(specific information will be sparse here. This is meant to give context for the Takeaways section of the post)

Our group (Garrett, Chu, and Johannes) have worked with John Wentworth in the SERI MATS 2 Electric Boogaloo program for three weeks, meaning it's time for a Review & Takeaways Post!

First week was Project Selection, and the first day was spent thinking about strategies for coming up with good projects. We chose to find a general method for figuring out True Names of mathy-feely-concepts-in-your-brain (such as roundness, color decomposition, or telling whether a piece of cloth is in a pile) with the goal that such a method would allow for figuring out true names for concepts like optimization, corrigibility, agency, modularity, neural network representations, and other alignment-relevant concepts.

Then we read Jaynes, and talked to TurnTrout, and concluded this project sucked. So we went back to Project Selection 2.0!

We came out of Project Selection 2.0 renewed with vigor, and a deeper understanding of the problems of alignment. Our new project was finding a better version of information theory by adapting logical induction or infra-Bayesianism.

Then we talked to Eliezer Yudkowsky, he asked for a concrete example of how this would solve alignment, and we didn't have a good example. So we went to Project Selection 3.0.

We came out of Project Selection 3.0 with even more vigor, and an even deeper understanding of the problems associated with alignment... and a clever idea.

Finetuning LLMs with RL seems to make them more agentic. We will look at the changes RL makes to LLMs' weights; we can see how localized the changes are, get information about what sorts of computations make something agentic, and make conjectures about selected systems, giving us a better understanding of agency.

Nobody has convinced us this is a bad use of our time, though we'd like to see people try.

Takeaways

Big ASS Tree

We learned lots of things over the course of figuring out all our ideas sucked. During the project selection phase we had a cool idea for a way to generate project ideas: The Alignment Safety Search Tree (ASS Tree for short). The idea comes from Mazes and Duality; the goal was to explore the space of problems and constraints before trying to propose solutions.

You start by writing "Alignment" up at the top of your whiteboard. This is the top level problem we want to solve. Then you draw arrows down. One for each problem, you can think of, that makes alignment hard. For each of these problems you repeat the process. E.g. for a problem P you draw down an arrow from P for each problem, that you can think of, that you need to solve, in order to solve P. Eventually you get a tree like this (except far bigger):

This is similar to what you do if you try to make an Alignment Game Tree. However, in my opinion, when we tried to make the game tree during an early phase of the MATS program, it did not lend much insight into what to work on. The criticisms ended up being pretty proposal-specific, and most arguments were over whether a particular problem was actually a problem associated with the particular proposal.

For creating the ASS tree, each of us made a tree independently. Then we merged our individual ASS trees into one Big ASS Tree, and looked for the broader problems which lots of problems in the tree had in common. We then again extended the resulting tree individually and merged the results.

A common node in the tree was that we did not know the True name for some important concept (e.g. agency, optimization, value), and thus the True Names Project was born (finding a general procedure that you can use to find the True Name for some concept).

Big ASS Takeaways

A...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing: Mechanism Design for AI Safety - Reading Group, published by Rubi on August 9, 2022 on The AI Alignment Forum.

We’re starting a new reading group for people interested in applying mechanism design tools to technical AI alignment. If you’re interested in joining, you can apply here by August 22nd (applying takes less than five minutes). If you have recommendations for papers to discuss, please mention them in the comments.

What We’re Doing

Mechanism design is the study of how to reach desirable outcomes or equilibria in the face of differing incentives and incomplete information. Many AI safety researchers have expressed enthusiasm about the potential of using these tools for work in alignment, but relatively little work has been done in the intersection. We believe this is partially due to a lack of potential researchers with expertise in both technical AI safety and mechanism design, and partially due to a lack of shovel-ready problems. The goal of this reading group is to make progress on both fronts.

There are three main areas that this reading group will cover:

Work at the intersection of technical AI safety and mechanism design, and where it can be expanded

Current work in technical AI safety, and how mechanism design tools can help

Current work on mechanism design, and how it can be applied to technical AI safety

The plan is to start with papers in the intersection, then alternate between papers on technical AI safety and papers on mechanism design while keeping the broader perspective in mind. Note that although we believe AI governance work is important and contains many applications for mechanism design, that will not be the focus of this reading group.

Who We Are and Who We Want

I (Rubi) am entering the 2nd year of a PhD in Economics this fall, and am currently working on technical AI safety in Berkeley through the SERI MATS program. Other likely participants include three Economics PhD students at top schools and a Math undergraduate student currently taking part in the SERI Summer Research Fellowship. Our hope for this reading group is to connect with people who have similar interests and create the potential for future collaborations.

Based on current expressions of interest, we expect the modal participant in the reading group to be a PhD student in economics, focusing on economic theory, who has read through the AGI Safety Fundamentals curriculum (or an equivalent, such as Eleuther's). If that sounds like you, definitely apply! However, these should not be considered necessary qualifications. Talented undergraduates with an interest in both areas or experts in one area who would like to learn more about the other should also apply.

If you’re unsure whether you have the background necessary to keep up with this reading group, a good test is to try skimming The Off-Switch Game. It’s a short paper, and on the more accessible end of papers we will be discussing. If you understand it or predict you would be able to understand it within an hour, then you are likely to be able to process the papers that we will discuss without too much additional work. If you find yourself struggling to understand the mathematical notation and proofs, then that is likely a bottleneck and you should consider prioritizing work to advance your comfort level there.

Participants will be expected to commit approximately eight hours a week for this reading group, which consists of five to seven hours reading the week’s paper and an hour and a half to discuss it. If it becomes apparent that a participant is repeatedly not reading or only skimming the papers, they will be removed from the reading group. Please ensure that you can dedicate the required time before applying.

Logistics

The application form can be found here. The only mandatory fields are a link/upload of your...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: General alignment properties, published by Alex Turner on August 8, 2022 on The AI Alignment Forum.

AIXI and the genome are both ways of specifying intelligent agents.

Give AIXI a utility function (perhaps over observation histories), and hook it up to an environment, and this pins down a policy.

Situate the genome in the embryo within our reality, and this eventually grows into a human being with a policy of their own.

These agents have different "values", in whatever sense we care to consider. However, these two agent-specification procedures also have very different general alignment properties.

General alignment properties are not about what a particular agent cares about (e.g. the AI "values" chairs). I call an alignment property "general" if the property would be interesting to a range of real-world agents trying to solve AI alignment. Here are some examples.

Terminally valuing latent objects in reality.

AIXI only "terminally values" its observations and doesn't terminally value latent objects in reality, while humans generally care about e.g. dogs (which are latent objects in reality).

Navigating ontological shifts.

Consider latent-diamond-AIXI (LDAIXI), an AIXI variant. LDAIXI's utility function which scans its top 50 hypotheses (represented as Turing machines), checks each work tape for atomic representations of diamonds, and then computes the utility to be the amount of atomic diamond in the world.

If LDAIXI updates sufficiently hard towards non-atomic physical theories, then it can no longer find any utility in its top 50 hypotheses. All policies now might have equal value (zero), and LDAIXI would not continue maximizing the expected diamond content of the future. From our viewpoint, LDAIXI has failed to rebind its "goals" to its new conceptions of reality. (From LDAIXI's "viewpoint", it has Bayes-updated on its observations and continues to select optimal actions.)

On the other hand, physicists do not stop caring about their friends when they learn quantum mechanics. Children do not stop caring about animals when they learn that animals are made out of cells. People seem to navigate ontological shifts pretty well.

Reflective reasoning / embeddedness.

AIXI can't think straight about how it is embedded in the world. However, people quickly learn heuristics like "If I get angry, I'll be more likely to be mean to people around me", or "If I take cocaine now, I'll be even more likely to take cocaine in the future."

Fragility of outcome value to initial conditions / Pairwise misalignment severity

This general alignment property seems important to me, and I'll write a post on it. In short: How pairwise-unaligned are two agents produced with slightly different initial hyperparameters/architectural choices (e.g. reward function / utility function / inductive biases)?

I'm excited about people thinking more about general alignment properties and about what generates those properties.

Supposing e.g. uniformly random tie-breaking for actions enabling equal expected utility.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Encultured AI, Part 1 Appendix: Relevant Research Examples, published by Andrew Critch on August 8, 2022 on The AI Alignment Forum.

Also available on the EA Forum.Appendix to: Encultured AI, Part 1: Enabling New BenchmarksFollowed by: Encultured AI, Part 2: Providing a Service

Appendix 1: “Trending” AI x-safety research areas

We mentioned a few areas of “trending” AI x-safety research above; below are some more concrete examples of what we mean:

Trustworthiness & truthfulness:

Owain Evans, Owen Cotton-Barratt and others have authored “Truthful AI: Developing and governing AI that does not lie” (arxiv, 2021; twitter thread).

Andreas Stuhlmüller, Jungwon Byun and others at Ought.org are building an AI-powered research assistant called Elicit (website); here is the product:.

Task-specific (narrow) preference learning:

Paul Christiano et al (arxiv, 2017) developed a data-efficient preference-learning technique for training RL-based systems, which is now very widely cited (scholar).

Jan Leike, now at OpenAI, leads a team working on ‘scalable alignment’ using preference-learning techniques (arxiv, 2018) (blog).

Interpretability:

Chris Olah (scholar) leads an interpretability research group at Anthropic. Anthropic (website) is culturally very attuned to large-scale risks from AI, including existential risks.

Buck Shlegeris and others at Redwood Research (website) have built an interpretability tool for analyzing transformer networks trained on natural language (demo).

Prof. Cynthia Rudin at Duke (homepage) approaches interpretability by trying to replace black-box models with more interpretable ones (arxiv, 2018), and we know from conversations with her that she is open to applications of her work to existential safety.

Robustness & risk management:

Prof. Jaime Fisac at Princeton (homepage) researches AI safety for robotics, high-dimensional control systems and multi-agent systems (scholar), including provable robustness guarantees. He was previously a PhD student at the UC Berkeley Center for Human-Compatible AI (CHAI), provided extensive feedback on AI Research Considerations for Human Existential Safety (ARCHES) (arxiv, 2020), and is very attuned to existential safety as a cause area.

Prof. David Krueger at Cambridge (scholar) studies out-of-distribution generalization (pdf, 2021), and is currently taking on students.

Adam Gleave (homepage) is a final-year PhD student at CHAI / UC Berkeley, and studies out-of-distribution robustness for deep RL.

Sam Toyer (scholar), also a PhD student at CHAI, has developed a benchmark for robust imitation learning (pdf, 2020).

Appendix 2: “Emerging” AI x-safety research areas

In this post, we classified cooperative AI and multi-stakeholder control of AI systems as “emerging” topics in AI x-safety. Here’s more about what we mean, and why:

Cooperative AI

This area is “emerging” in x-safety because there’s plenty of attention to the issue of cooperation from both policy-makers and AI researchers, but not yet much among folks focused on x-risk.

Existential safety attention on cooperative AI:

Many authors — too many to name! — have remarked on the importance of international coordination on AI safety efforts, including existential safety. For instance, there is a Wikipedia article on AI arms races (wikipedia). This covers the human–human side of the cooperative AI problem.

AI research on cooperative AI:

Multi-agent systems research has a long history in AI (scholar search), as does multi-agent reinforcement learning (scholar search).

DeepMind’s Multi-agent Learning team has recently written number papers examining competition and cooperation between artificial agents (website).

OpenAI has done some work on multi-agent interaction, e.g. emergent tool use in multi-agent interaction (arxiv).

Prof. Jakob Foerster at Oxford (scholar search), and ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Encultured AI, Part 1: Enabling New Benchmarks, published by Andrew Critch on August 8, 2022 on The AI Alignment Forum.

Also available on the EA Forum.Followed by: Encultured AI, Part 2 (forthcoming)

Hi! In case you’re new to Encultured AI, we’re a for-profit start-up with a public benefit mission: developing technologies promoting the long-term survival and flourishing of humanity and other sentient life. However, we also realize that AI poses an existential risk to humanity if not developed with adequate safety precautions. Given this, our goal is to develop products and services that help humanity steer toward the benefits and away from the risks of advanced AI systems. Per the “Principles” section of our homepage:

Our current main strategy involves building a platform usable for AI safety and alignment experiments, comprising a suite of environments, tasks, and tools for building more environments and tasks. The platform itself will be an interface to a number of consumer-facing products, so our researchers and collaborators will have back-end access to services with real-world users. Over the next decade or so, we expect an increasing number of researchers — both inside and outside our company — will transition to developing safety and alignment solutions for AI technology, and through our platform and products, we’re aiming to provide them with a rich and interesting testbed for increasingly challenging experiments and benchmarks.

In the following, we’ll describe the AI existential safety context that motivated us to found Encultured, and go into more detail about what we’re planning to do.

What’s trending in AI x-safety?

The technical areas below have begun to receive what we call “existential attention” from AI researchers, i.e., attention from professional AI researchers thinking explicitly about the impact of their work on existential safety:

Trustworthiness & truthfulness — ensuring AI systems are telling us the truth and doing the things they and their creators say they’re going to do.

Preference learning — enabling AI systems to learn what humans want.

Interpretability — enabling humans to understand what AI systems are thinking and doing.

Robustness & risk management — ensuring AI systems continue functioning well in novel situations, and quantifying the risk that they won’t.

In other words, the topics above lie in the intersection of the following Venn diagram:

See Appendix 1 for examples of research in these areas. More research in these areas is definitely warranted. A world where 20%+ of AI and ML researchers worldwide pivoted to focusing on the topics above would be a better world, in our opinion.

If our product is successful, we plan to grant access to researchers inside and outside our company for performing experiments in the areas above, interacting directly with users on our platform. And, our users will be aware of this ;) We’re planning on this not only because it will benefit the world, but because it will benefit our products directly: the most valuable tools and services are trustworthy, truthful, preference-sensitive, interpretable, and robust.

What’s emerging in AI x-safety?

The following topics have received research attention from some researchers focused on existential safety, and AI research attention from other researchers, but to us the two groups don’t (yet) seem to overlap as much as for the ‘trending’ topics above.

Cooperative AI — designing AI technologies in ways that enable improved cooperation between humans and AI systems, while preventing collusion between AI systems, i.e., cooperation between AI systems that would be harmful or deceptive to humanity. (see Appendix 2 for related research.)

Multi-stakeholder control of AI systems — allowing people with diverse values, such as from competing geopolitical factions, to share ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpretability/Tool-ness/Alignment/Corrigibility are not Composable, published by johnswentworth on August 8, 2022 on LessWrong.

Interpretability

I have a decent understanding of how transistors work, at least for purposes of basic digital circuitry. Apply high voltage to the gate, and current can flow between source and drain. Apply low voltage, and current doesn’t flow. (... And sometimes reverse that depending on which type of transistor we’re using.)

I also understand how to wire transistors together into a processor and memory. I understand how to write machine and assembly code to run on that processor, and how to write a compiler for a higher-level language like e.g. python. And I understand how to code up, train and run a neural network from scratch in python.

In short, I understand all the pieces from which a neural network is built at a low level, and I understand how all those pieces connect together. And yet, I do not really understand what’s going on inside of trained neural networks.

This shows that interpretability is not composable: if I take a bunch of things which I know how to interpret, and wire them together in a way I understand, I do not necessarily know how to interpret the composite system. Composing interpretable pieces does not necessarily yield an interpretable system.

Tools

The same applies to “tools”, in the sense of “tool AI”. Transistors and wires are very tool-ish: I understand what they do, they’re definitely not optimizing the broader world or trying to trick me or modelling me at all or trying to self-preserve or acting agenty in general. They’re just simple electronic tools.

And yet, assuming agenty AI is possible at all, it will be possible to assemble those tools into something agenty.

So, like interpretability, tool-ness is not composable: if I take a bunch of non-agenty tools, and wire them together in a way I understand, the composite system is not necessarily a non-agenty tool. Composing non-agenty tools does not necessarily yield a non-agenty tool.

Alignment/Corrigibility

What if I take a bunch of aligned and/or corrigible agents, and “wire them together” into a multi-agent organization? Is the resulting organization aligned/corrigible?

Actually there’s a decent argument that it is, if the individual agents are sufficiently highly capable. If the agents can model each other well enough and coordinate well enough, then they should be able to each individually predict what individual actions will cause the composite system to behave in an aligned/corrigible way, and they want to be aligned/corrigible, so they’ll do that.

However, this does not work if the individual agents are very limited and unable to model the whole big-picture system. HCH-like proposals are a good example here: humans are not typically able to model the whole big picture of a large human organization. There are too many specialized skillsets, too much local knowledge and information, too many places where complicated things happen which the spreadsheets and managerial dashboards don’t represent well. And humans certainly can’t coordinate at scale very well in general - our large-scale communication bandwidth is maybe five to seven words at best. Each individual human may be reasonably aligned/corrigible, but that doesn’t mean they aggregate together into an aligned/corrigible system.

The same applies if e.g. we magically factor a problem and then have a low-capability overseeable agent handle each piece. I could definitely oversee a logic gate during the execution of a program, make sure that it did absolutely nothing fishy, but overseeing each individual logic gate would do approximately nothing at all to prevent the program from behaving maliciously.

How These Arguments Work In Practice

In practice, nobody proposes that AI built from transistors and w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Possibility of Microorganism Suffering, published by Elias Au-Yeung on August 7, 2022 on The Effective Altruism Forum.

Thanks to Brian Tomasik and an anonymous reviewer for their comments. It was through reading Brian's writings that the topic was first brought to my attention.

Epistemic status: Establishing knowledge about other minds requires both science and philosophy. The scientific details should mostly be correct. The philosophical component uses plausible ideas – but it’s necessarily intuition-dependent whether we decide to accept or reject such reasoning. Note that this subjectivity isn’t specific to thinking about microorganisms, but is rather general to speculating about the experiences of any mind other than one’s own.

Summary

Massive numbers of microorganisms, and their counteractive responses to various threats – in some respects similar to other organisms – suggest a possibility of immense suffering and an ongoing moral catastrophe. Despite this, few have seriously investigated the possibility of microorganism suffering. This post looks at the topic through both scientific and philosophical lenses. Before evaluating scientific findings, I generate, and highlight the importance of generating, criteria for such evaluation.

To demonstrate why microorganism suffering seems possible, I show that starting from ‘common sense’ ideas of what evidence of mind and suffering means, physically speaking, we can reach (to use Brian Tomasik’s words) ‘cosmopolitan views’ of evidence of mind and suffering that include systems that are more different. And it seems if we already accept ‘common sense’ views, we should consider ‘cosmopolitan views’ somewhat plausible. This is especially so if we have uncertainties with our subjective judgment – we might be compelled to assign at least some credence to ‘cosmopolitan views’, given some acceptance of ‘common sense’ positions. Additionally, uncertainty might suggest that we cannot be so confident in only ‘common sense’ positions as to rule out similar ‘cosmopolitan views’. That said, whether we accept or reject these arguments also seems intuition-dependent.

I note some examples of microbial behaviors and mechanisms that fit the cosmopolitan criteria. If my arguments are successful, forcing us to place some credence in cosmopolitan views, then there may be significant amounts of microorganism suffering in expectation. We may have reason to try and understand more about how we can reduce such suffering.

A video showing two paramecia eaten by amoeba:

Introduction

At any given moment, around 1030 (one thousand billion billion billion) microorganisms exist on Earth. Many have very short lives, resulting in massive numbers of deaths. Rough calculations suggest 1027 to 1029 deaths per hour on Earth. Microorganisms display aversive reactions, escape responses, and/or physiological changes against various fitness-threatening phenomena: harmful chemicals, extreme temperatures, starvation, sun damage, mechanical damage, and predators and viruses.

It’s no exaggeration to say that if microorganisms can suffer, such suffering is immense in quantity. Even if microorganism suffering is unlikely, we might still argue that it is extremely large in expectation.

This possibility of large-scale suffering may warrant further exploration. Yet, aside from a few exceptions, the topic has been severely neglected. My first impression is that reducing microorganism suffering is a cause that some should potentially prioritize, from the perspectives of many non-speciesist, welfarist views. I won’t be investigating cause prioritization in detail here. Instead, in this piece, I only aim to explain why I think microorganism suffering is at least a possibility. By ‘possibility’ I mean not astronomically lower in probability compared to the existence of suf...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [link post] How effective altruism went from a niche movement to a billion-dollar force, published by Tobias Häberli on August 8, 2022 on The Effective Altruism Forum.

Dylan Matthews of Vox's Future Perfect on the history of Effective Altruism, recent developments, and what Effective Altruism means to him.I found the section titled What EA has meant to me especially moving:

Things have indeed gotten weird in EA. The EA I know in 2022 is a more powerful and more idiosyncratic entity than the EA I met in 2013. And as it’s grown, it’s faced vocal backlash of a kind that didn’t exist in its early years.

A small clique of philosophy nerds donating their modest incomes doesn’t seem like a big enough deal to spark much outside critique. A multi-billion-dollar complex with designs on influencing the course of American politics and indeed all of world history . is a different matter.

I’ll admit it’s somewhat hard to write dispassionately about this movement. Not because it’s flawless (it has plenty of flaws) or because it’s too small and delicate to deserve the scrutiny (it has billions of dollars behind it). Rather, because it’s profoundly changed my own life, and overwhelmingly for the better.

I encountered effective altruism while I was a journalist covering federal public policy in the US. I lived and breathed Senate committee schedules, think tank reports, polling averages, outrage cycles about whatever Barack Obama or Mitt Romney said most recently. I don’t know if you’ve immersed yourself in American politics like that . but it’s a horrible place to live. Arguments are more often than not made in extreme bad faith. People’s attention was never focused on issues that mattered most to the largest number of people. Progress for actual people in the actual world, when it did happen, was maddeningly slow.

And it was getting worse. I started writing professionally in 2006, when the US was still occupying Iraq and a cataclysmic recession was around the corner. There were already dismal portents for our country’s institutions, but Donald Trump was still just a deranged game show host. One of the dominant parties was not yet attempting full coups with the help of armed mobs of supporters. The Senate’s huge geographic bias was not yet the enormous advantage for Republicans it has become today, rendering the body hugely unrepresentative in a way that offends basic democratic principles. All of that was still to come.

Derek Parfit, one of the key philosophers inspiring effective altruism, once wrote that he used to believe his own personal identity was the key thing that mattered in his life. This view trapped him. “My life seemed like a glass tunnel, through which I was moving faster every year, and at the end of which there was darkness,” he wrote in Reasons and Persons. “When I changed my view, the walls of my glass tunnel disappeared. I now live in the open air. There is still a difference between my life and the lives of other people. But the difference is less. Other people are closer. I am less concerned about the rest of my own life, and more concerned about the lives of others.“

Finding EA was a similarly transformative experience for me. The major point for me was less that this group of people had found, once and for all, the most effective ways to spend money to help people. They didn’t, they won’t, though more than most movements, they will admit that, and cop to those limits to what they can learn about the world.

What was different was that I now had a sense that there was more to the world than the small corner I had dug into in Washington, DC — so much so that I was inspired to co-found this very section of Vox, Future Perfect. This is, in retrospect, an obvious revelation. If I had spent this period as a microbiologist as CRISPR emerged, it would have been obviou...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steganography in Chain of Thought Reasoning, published by A Ray on August 8, 2022 on LessWrong.

Here I give a possible phenomenon of steganography in chain of thought reasoning, where a system doing multi-stage reasoning with natural language encodes hidden information in its outputs that is not observable by humans, but can be used to boost its performance on some task. I think this could happen as a result of optimization pressure and natural language null space. At the end is a sketch of a research idea to study this phenomenon empirically.

Definitions

The term steganography originally comes from computer security, where it refers to the practice of hiding messages or data in otherwise innocent looking media, such as images, audio, or text. The goal is to avoid detection by an adversary who might intercept or inspect the media, and to communicate covertly with a trusted receiver who knows how to decode the hidden information. This is different from using code words or encryption, which might raise suspicion by their presence or appearance.

Chain of thought is a technique of using sequential sampling of language models to improve performance. I use to broadly describe any situation where we take the outputs of a system, and pass them back in as inputs to that system, in order to get multi-stage reasoning. This can include single player scenarios, such as using a language model to work through a math problem, but also scenarios where there is dialogue or conversation, such as some formulations of safety via debate. For this post, I will focus on natural language text as the medium of communication, but this idea could extend to other media as well.

Steganography in chain of thought reasoning (just “steganography” for the rest of this post) is the phenomenon I propose, where a system doing chain of thought reasoning is encoding hidden information into the text in ways that humans cannot see or observe. Note that this is not the same as a system being “deceptive” or “hiding” (in the active sense of the verb) the information, or anything that requires theory of mind.

Motivation

Why should we care about steganography in chain of thought reasoning? I think this is a potentially big alignment issue for systems that use this kind of reasoning, and I think we should care about chain of thought reasoning for several reasons.

First, chain of thought reasoning is currently responsible for some state-of-the-art results that have impressed folks on this forum, such as the Minerva, which uses a recursive language model to answer open-domain questions.

Second, chain of thought reasoning might also lead to alignment techniques that are more tractable than other approaches, such as Externalized Reasoning Oversight on using dialogue to align natural language models.

Third, much of the research currently going into chain of thought reasoning is focused on end-to-end performance (such as beating benchmarks), and not actively looking for this kind of phenomenon. My expectation is that researchers don’t discover this kind of behavior unless they’re looking for it, or unless it becomes obvious enough to degrade the language quality.

Mechanism

How is steganography possible in chain of thought reasoning? The main ingredients are natural language null space and optimization pressure.

Natural language null space refers to the fact that natural language has many different ways of expressing the same content, which means there are lots of degrees of freedom or “null space” in which information can be hidden. For example, we can say “I’m hungry” or “My stomach is grumbling” or “I could use a snack” or “It’s time for lunch” or many other variations, and convey roughly the same meaning. If we imagine anthropomorphic versions of this, we might come up with things like code words or preferring speci...