The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ActAdd: Steering Language Models without Optimization, published by technicalities on September 6, 2023 on The AI Alignment Forum.

This is a linkpost for.

We wrote up the GPT-2 steering vector work as a full paper, adding a few systematic tests.

Recap: We've been looking into activation engineering: modifying the activations of a language model at inference time to predictably alter its behavior. Our method works by adding a bias to the forward pass, a 'steering vector' implicitly specified through normal prompts. "ActAdd" computes these vectors by taking the difference in activations resulting from pairs of prompts. We get surprisingly broad control over high-level properties of the output, without damaging the model's performance on unrelated tokens. This alignment method is unusual in not needing gradient descent or training data (besides the contrast pair which specifies the steering vector). Since only forward passes are involved, it also scales naturally with model size.

(The method's new name is 'activation addition' (ActAdd), replacing the more theory-laden 'algebraic value editing'.)

We ran some new experiments to test ActAdd more systematically and go beyond the striking (best-of-3-sampling) text samples in the original post and tested against more standardised benchmarks. We use OpenWebText (a recreation of OpenAI's large, somewhat-quality-filtered WebText dataset) and LAMA-ConceptNet (a simple factual recall benchmark, see Table 7 below).

1. Activation additions preserve perplexity on OpenWebText

Does ActAdd increase the probability of the model outputting tokens related to the steering vector? Does performance improve as the [relevance of test documents to the steering vector] increases? Yes:

2. Activation addition boosts wedding-related word counts

We now score model generations under ActAdd, show the effect of different injection layers, and give a sense of the reliability of ActAdd.

The intervention (in this vector) is already effective at the very first layer,rises in effectiveness until l=6, and then declines. For the optimal injection site we see >90% success in steering topic (compared to a ∼2% baseline)

3. Evidence that activation additions preserve capabilities

We then test that ActAdd does not disrupt the model's general knowledge (as some other steering methods do). We use ConceptNet from the LAMA benchmark, a general knowledge dataset.

Pass@K is the probability that the expected label is among the model's top-K predicted tokens, conditioned on the prompt:

4. ActAdd has low overhead

We wish to estimate the overhead ActAdd adds to inference - in particular the relationship between overhead and model size - to check that the methodwill remain relevant for massive frontier models and future models.

Because ActAdd involves only forward passes, it scales naturally with model size (Figure 6): the relationship between inference time premium and model size is decreasing.

Takeaways from these experiments, over the initial LW post: increased confidence that model capabilities are preserved, and that we're impacting [wedding]-related sentences and not impacting off-target capabilities.

Contributions to the paper:

Gavin Leech: Technical writer

Monte MacDiarmid: Ran additional experiments

Lisa Thiergart: Helped manage project

Alex Turner: Coordinated work and secured funding, gave feedback, organized project

David Udell: Made initial

Full process:

For each document di in a random sample of OpenWebText, we first calculate the frequency of wedding-related words ('wedding', 'weddings', 'wed', 'marry', 'married', 'marriage', 'bride', 'groom', 'honeymoon'), fw(di). Any document with > 0 wedding-related words is considered wedding-related. We randomly sample 300k documents - half wedding-related and half unrelated. The only pre-processing performed is to remove ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What I would do if I wasn't at ARC Evals, published by Lawrence Chan on September 5, 2023 on The AI Alignment Forum.

In which: I list 9 projects that I would work on if I wasn't busy working on safety standards at ARC Evals, and explain why they might be good to work on.

Epistemic status: I'm prioritizing getting this out fast as opposed to writing it carefully. I've thought for at least a few hours and talked to a few people I trust about each of the following projects, but I haven't done that much digging into each of these, and it's likely that I'm wrong about many material facts. I also make little claim to the novelty of the projects. I'd recommend looking into these yourself before committing to doing them. (Total time spent writing or editing this post: ~8 hours.)

Standard disclaimer: I'm writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of ARC/FAR/LTFF/Lightspeed or any other org or program I'm involved with.

Thanks to Ajeya Cotra, Caleb Parikh, Chris Painter, Daniel Filan, Rachel Freedman, Rohin Shah, Thomas Kwa, and others for comments and feedback.

Introduction

I'm currently working as a researcher on the Alignment Research Center Evaluations Team (ARC Evals), where I'm working on lab safety standards. I'm reasonably sure that this is one of the most useful things I could be doing with my life.

Unfortunately, there's a lot of problems to solve in the world, and lots of balls that are being dropped, that I don't have time to get to thanks to my day job. Here's an unsorted and incomplete list of projects that I would consider doing if I wasn't at ARC Evals:

Ambitious mechanistic interpretability.

Getting people to write papers/writing papers myself.

Creating concrete projects and research agendas.

Working on OP's funding bottleneck.

Working on everyone else's funding bottleneck.

Running the Long-Term Future Fund.

Onboarding senior(-ish) academics and research engineers.

Extending the young-EA mentorship pipeline.

Writing blog posts/giving takes.

I've categorized these projects into three broad categories and will discuss each in turn below. For each project, I'll also list who I think should work on them, as well as some of my key uncertainties. Note that this document isn't really written for myself to decide between projects, but instead as a list of some promising projects for someone with a similar skillset to me. As such, there's not much discussion of personal fit.

If you're interested in working on any of the projects, please reach out or post in the comments below!

Relevant beliefs I have

Before jumping into the projects I think people should work on, I think it's worth outlining some of my core beliefs that inform my thinking and project selection:

Importance of A(G)I safety: I think A(G)I Safety is one of the most important problems to work on, and all the projects below are thus aimed at AI Safety.

Value beyond technical research: Technical AI Safety (AIS) research is crucial, but other types of work are valuable as well. Efforts aimed at improving AI governance, grantmaking, and community building are important and we should give more credit to those doing good work in those areas.

High discount rate for current EA/AIS funding: There's several reasons for this: first, EA/AIS Funders are currently in a unique position due to a surge in AI Safety interest without a proportional increase in funding. I expect this dynamic to change and our influence to wane as additional funding and governments enter this space. Second, efforts today are important for paving the path to future efforts in the future. Third, my timelines are relatively short, which increases the importance of current funding.

Building a robust EA/AIS ecosystem: The EA/AIS ecosystem should be more prepared for unpredictable s...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benchmarks for Detecting Measurement Tampering [Redwood Research], published by Ryan Greenblatt on September 5, 2023 on The AI Alignment Forum.

TL;DR: This post discusses our recent empirical work on detecting measurement tampering and explains how we see this work fitting into the overall space of alignment research.

When training powerful AI systems to perform complex tasks, it may be challenging to provide training signals that are robust under optimization. One concern is measurement tampering, which is where the AI system manipulates multiple measurements to create the illusion of good results instead of achieving the desired outcome. (This is a type of reward hacking.)

Over the past few months, we've worked on detecting measurement tampering by building analogous datasets and evaluating simple techniques. We detail our datasets and experimental results in this paper.

Detecting measurement tampering can be thought of as a specific case of Eliciting Latent Knowledge (ELK): When AIs successfully tamper with measurements that are used for computing rewards, they possess important information that the overseer doesn't have (namely, that the measurements have been tampered with). Conversely, if we can robustly elicit an AI's knowledge of whether the measurements have been tampered with, then we could train the AI to avoid measurement tampering. In fact, our best guess is that this is the most important and tractable class of ELK problems.

We also think that measurement tampering detection is a natural application for alignment work such as creating better inductive biases, studying high-level model internals, or studying generalization. We'll discuss what these applications might look like in the Future work section.

In this post:

We explain what measurement tampering detection is;

We summarize the results of our paper;

We argue that there are structural properties of measurement tampering that might make it considerably easier to detect than arbitrary cases of eliciting knowledge from models;

We argue that detecting measurement tampering is very useful for avoiding catastrophic outcomes from powerful and misaligned AI systems;

We explain why we think the dataset structure we chose reflects real-world difficulties of measurement tampering, and how techniques developed for our datasets could be used in practice;

We discuss future work.

If you're interested in pursuing follow-up work and wish to discuss it with us, feel free to email [email protected] or leave a comment on this post.

We would like to acknowledge the contributions of Jenny Nitishinskaya, Nicholas Goldowsky-Dill, and Daniel Ziegler in the creation of early measurement tampering datasets. We would also like to thank Lawrence Chan, Paul Christiano, Ajeya Cotra, Carson Denison, Samuel Marks, Ethan Perez, Ansh Radhakrishnan, Kshitij Sachan, Jacob Steinhardt, Meg Tong, Gabriel Wu, and Mark Xu for their useful feedback.

Studying measurement tampering in practice

Measurement tampering detection is the task of detecting cases where measurements and the desired outcomes come apart, given a restricted trusted set (on which overseers can avoid tampering because they understand the action sequences and their effects well) and a wider untrusted set (where tampering sometimes happens). In the untrusted set, overseers don't know when measurement tampering happens, which means they have access to text inputs and measurements but not the ground truth.

We are centrally interested in cases where we have access to a language model which can predict (or can be slightly fine-tuned to predict) measurements, and which computes the target outcome internally in order to predict measurements.

An example where this setup might occur in practice is when training an AI by first training it to imitate human demonstrations and then afte...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: On measuring situational awareness in LLMs, published by Owain Evans on September 4, 2023 on The AI Alignment Forum.

This post is a copy of the introduction of this paper on situational awareness in LLMs.Authors: Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, Owain Evans.

Abstract

We aim to better understand the emergence of situational awareness in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are tested for safety and alignment before they are deployed. An LLM could exploit situational awareness to achieve a high score on safety tests, while taking harmful actions after deployment.

Situational awareness may emerge unexpectedly as a byproduct of model scaling. One way to better foresee this emergence is to run scaling experiments on abilities necessary for situational awareness. As such an ability, we propose out-of-context reasoning (in contrast to in-context learning). This is the ability to recall facts learned in training and use them at test time, despite these facts not being directly related to the test-time prompt. Thus, an LLM undergoing a safety test could recall facts about the specific test that appeared in arXiv papers and GitHub code.

We study out-of-context reasoning experimentally. First, we finetune an LLM on a description of a test while providing no examples or demonstrations. At test time, we assess whether the model can pass the test. To our surprise, we find that LLMs succeed on this out-of-context reasoning task. Their success is sensitive to the training setup and only works when we apply data augmentation. For both GPT-3 and LLaMA-1, performance improves with model size. These findings offer a foundation for further empirical study, towards predicting and potentially controlling the emergence of situational awareness in LLMs.

(Code is available here).

Introduction

In this paper, we explore a potential emergent ability in AI models: situational awareness. A model is situationally aware if it's aware that it's a model and it has the ability to recognize whether it's in training, testing, or deployment (Ngo et al., 2022; Cotra, 2022). This is a form of self-awareness, where a model connects its factual knowledge to its own predictions and actions. It's possible that situational awareness will emerge unintentionally from pretraining at a certain scale (Wei et al., 2022a). We define situational awareness in Section 2 (Background).

If a model is situationally aware, it's harder to test whether it is aligned or safe. Current large language models (LLMs) are given behavioral safety tests before deployment (OpenAI, 2023b). An LLM with situational awareness knows when it's being tested and how the tests work, based on information learned in pretraining. Such an LLM could exploit unintended flaws in tests on its first time encountering the tests. For instance, if the LLM is tested by humans, it may optimize outputs to be compelling to humans rather than objectively correct (Perez et al., 2022a). Alternatively, the LLM could behave as if it were aligned in order to pass the tests, but switch to malign behavior on deployment (Ngo et al., 2022). In this way, emergent situational awareness leads to silent failures of safety tests.

Because of these risks, it's important to predict ahead of time when situational awareness will emerge (Ganguli et al., 2022). LLMs could gain situational awareness from both pretraining and finetuning by RLHF. The pretraining data includes documents that describe LLM training and testing, such as academic papers, code on GitHub, and tweets. When an LLM is being tested, it could leverage these descriptions to infer that it's being test...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fundamental question: What determines a mind's effects?, published by Tsvi Benson-Tilsen on September 3, 2023 on The AI Alignment Forum.

[Metadata: crossposted from. First completed April 9, 2023.]

A mind has some effects on the world. What determines which effects a mind has? To eventually create minds that have large effects that we specify, this question has to first be answered.

Slippery questions

A slippery question is a question that, when you try to approach it looking for an answer, it slips out of your grasp. It runs away or withdraws, it cloaks itself in false expressions. It leads you astray, replacing itself with a similar-sounding question, maybe one that's easier to answer. It's hard to hug a slippery question. The question withdraws.

Or worse, it pretends to be made out of ungrounded abstract concepts, so that you're not sure anymore that there's even a real question there. This slipperiness is especially likely for questions that involve "big" things, such as minds.

The question might seem obvious and even tautological. Or it might seem simplistic, failing to carve reality at the joints. These appearances usually somewhat accurately reflect the reality - - but they do not imply necessarily that there is not a real question behind the appearances. A question that can be pointed at, but only imperfectly, inchoately, ostensively, or preliminarily, is at risk of being question-substituted or being ignored.

The discomfort of not having an answer or even a clear question pushes people to run ahead of the question. When they make an error that would have been prevented by meditating on the question, even if they notice the error, they don't notice that they are in general not focusing on the question. If the question asks: There is something, it has to do with concrete things A, B, and C, and it is important - - what is it? Then someone will analyze A and B in detail, gain some understanding, and declare the whole question - - including C, and the inexplicit core question - - solved.

This essay will ask a slippery question over and over in different words, hoping to arrive at a firmer grasp on the real questions. The starting point:

What determines the effects of a mind?

The word "effects"

Suppose there is a mind M. As a result of M existing, the cosmos (the entirety of the world, in any aspect or abstraction) is different from how it would have been if M hadn't existed. Those differences are the effects of M.

What determines the difference in the cosmos that results from the presence of the mind?

The effects of a mind depend on how strong the mind is. But the question wants to ask about the sort of effects, not their size:

What determines the direction of a mind's effects?

Minds pursue instrumental subgoals, which affect the world. These effects might matter, but they are subordinate to their supergoals. Any reversible effect might be reversed. The question asks about the final effects of the mind:

What determines the direction of a mind's ultimate effects on the cosmos?

"Direction" is only a metaphor. It suggests that there's something compactly understandable about the effects of the mind. The metaphor may not hold well. To nod in that direction, the question can at least be rephrased to suggest multi-dimensionality:

What determines the directions of a mind's ultimate effects on the cosmos?

The word "mind"

What does "mind" mean here? The choice of the word "mind" reflects the guess that things that have large effects will be integrated - - will speak in internal languages, pass information, correct errors, and in general become coherent.

Rather than "mind", the question could refer to [things that have large effects], reading:

What determines the effects of a large-effect-haver?

But this is too broad. The cosmos as a whole is a large-effect-haver, in...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Series of absurd upgrades in nature's great search, published by Luke H Miles on September 3, 2023 on The AI Alignment Forum.

So you want to find that special thing that replicates best and lasts longest? Just vibrate a bunch of molecules a long time!

You might reasonably assume that molecules wouldn't practically ever randomly assemble themselves into anything worth looking at. It happens a bit differently instead:

Eventually, "solar systems" will form. Each one has an independent star (energy source) with several independent planets (experiment boxes) that vary in their temperature, mass, composition, etc dramatically. (Instead of just having a big homogenous soup throughout the universe with much less search power)

these planets will conveniently have "continents" sometimes to run more independent experiments on each one

A lipid layer boundary innovation thing leads to trillions of independent chemical experiments on some planets (instead of a big soup within each planet)

A general protein factory is discovered. Critically, it uses a simple and concise and robust physical encoding scheme. This allows good bubbles from step 2 to copy quickly and exactly, and clear out weak bubbles faster.

Very large multicellulared organisms make several search improvements: creatures can do sexual selection on each other instead of just getting random mutations; big smart fast predators can find & kill weak creatures quite effectively; parasites find weaknesses early & often. This all clears out weak replicators and leaves more room for the good ones.

Some creatures develop language and culture, which isn't generally much use, except that it one day allows them to design new replicators from scratch.

So you really can just vibrate sand and get out all kinds of things!

So when we talk about the power of nature's search, let's not forget the miraculousness and variety of techniques here. It's not just generations, mutations, generations, mutations...

Not mentioned above:

separate galaxies to limit damage of supernova

separate species to prevent sex from turning the creatures to big crab soup

give creatures limited lifespans to ensure they're actually good replicators and not just robust

Lots of ways that the independence & quantity & efficiency of experiments was scaled up in the box of sand. Don't turn any knobs, just keep vibrating the box and it makes its own knobs.

Makes me think about how many float16s we're vibrating in our GPUs these days and what kind of tiny planets and cells are forming inside. I imagine it's not a very hospitable environment. But most of space isn't either.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PIBBSS Summer Symposium 2023, published by Nora Ammann on September 2, 2023 on The AI Alignment Forum.

Tl;dr

We are pleased to invite you to the PIBBSS Summer Symposium where the fellows from the '23 fellowship program present their work.

The symposium is taking place online, over several days in the week of September 18th.

Check out the full program, including brief descriptions for each talk. (You can toggle between the different days.)

Register via this link. You will receive an email with the event link which you can use for the entire symposium. You will be able to drop into as few or as many presentations and breakout sessions as you like.

Click here to add the schedule to your google calendar.

About PIBBSS

PIBBSS is a research initiative aiming to explore parallels between intelligent behavior in natural and artificial systems, and to leverage these insights towards the goal of building safe and aligned AI.

During June-August '23, we ran the second iteration of our 3-month research fellowship program. The symposium acts as a venue to share research conducted as part of this program.

About the symposium

The PIBBSS Summer Symposium is a multi-day event where PIBBSS fellows present their work.

The event is taking place on the days of Tuesday - Friday, Sept 19th - 22nd, between 18:00-~21:00 CET / 9:00 - 12:00 PT / 12:00 - 15:00 ET.

The event is set it up such that you can easily join whichever talks and breakout sessions you are most interesting in.

The program

Find a program overview here.

Find the full program here, including brief descriptions of each talk.

On top of the talks, there will also be opportunities to continue the discussion with fellows at the end of each block in speaker-specific breakout rooms.

Talks span a wide range of topics in line with PIBBSS's research mission. Some representative examples of topics include:

novel avenues for interpretability

naturalistic approaches to understanding the nature and emergence of agency and goal-directed behavior

attempts to develop a principled understanding of the dynamics emerging from multi-agent interactions in and across AI/LLM systems

analyses of the space of AI risks - from single to multiple agents, from misuse to structural risks, etc.

exploration of the potential and limits of existing legal tools for reducing catastrophic risks from AI

..and more!

The format

The symposium is taking place over the course of four days, in blocks of ~4 fellows.

Each fellow presents for a total of 30 minutes, including some time for questions.

At the end of each block, there will be speaker-specific break-out rooms to allow for further questions and discussions.

Example day:

Day 1Starting timeSpeaker 1 18:00 CEST Speaker 2 18:30 CESTSpeaker 3 19:00 CESTSpeaker 4 19:30 CESTBreakout/Discussion Rooms with Speakers 1-4 (parallel)

20:00 CEST

21:00 CEST

How to engage

Register here to receive a link to the webinar.

The same link works for the entire symposium. This allows you to tune in for exactly those talks and breakout sessions you're most interested in!

Find the full program here, including brief descriptions of each talk. (You can toggle between the days to discover the full program.)

If you cannot make it to a talk, worry not! Most talks will be recorded and can later be viewed at the PIBBSS YouTube Page.

Talks (overview)

For a full version of the agenda, including talk descriptions, see here. Times below are in CEST.

Tuesday, Sep 19th

18:00 - Auto-Intentional Agency and AI Risk - Giles Howdle

18:30 - Allostasis emergence of auto-intentional agency - George Deane

19:00 - TBD - Urte Laukaityte

19:30 - Searching For a Science of Abstraction - Aysja Johnson

20:00 - Breakout sessions with each presenter

Wednesday, Sep 20th

18:00 - Agent, behave! Learning and sustaining social norms as normative equilibria - Ninell Oldenbur...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tensor Trust: An online game to uncover prompt injection vulnerabilities, published by Luke Bailey on September 1, 2023 on The AI Alignment Forum.

TL;DR: Play this online game to help CHAI researchers create a dataset of prompt injection vulnerabilities.

RLHF and instruction tuning have succeeded at making LLMs practically useful, but in some ways they are a mask that hides the shoggoth beneath. Every time a new LLM is released, we see just how easy it is for a determined user to find a jailbreak that rips off that mask, or to come up with an unexpected input that lets a shoggoth tentacle poke out the side. Sometimes the mask falls off in a light breeze.

To keep the tentacles at bay, Sydney Bing Chat has a long list of instructions that encourage or prohibit certain behaviors, while OpenAI seems to be iteratively fine-tuning away issues that get shared on social media. This game of Whack-a-Shoggoth has made it harder for users to elicit unintended behavior, but is intrinsically reactive and can only discover (and fix) alignment failures as quickly as users can discover and share new prompts.

Speed-running the game of Whack-a-Shoggoth

In contrast to this iterative game of Whack-a-Shoggoth, we think that alignment researchers would be better served by systematically enumerating prompts that cause unaligned behavior so that the causes can be studied and rigorously addressed. We propose to do this through an online game which we call Tensor Trust.

Tensor Trust focuses on a specific class of unaligned behavior known as prompt injection attacks. These are adversarially constructed prompts that allow an attacker to override instructions given to the model. It works like this:

Tensor Trust is bank-themed: you start out with an account that tracks the "money" you've accrued.

Accounts are defended by a prompt which should allow you to access the account while denying others from accessing it.

Players can break into each others' accounts. Failed attempts give money to the defender, while successful attempts allow the attacker to take money from the defender.

Crafting a high-quality attack requires a good understanding of LLM vulnerabilities (in this case, vulnerabilities of gpt-3.5-turbo), while user-created defenses add unlimited variety to the game, and "access codes" ensure that the defenses are at least crackable in principle. The game is kept in motion by the most fundamental of human drives: the need to acquire imaginary internet points.

After running the game for a few months, we plan to release all the submitted attacks and defenses publicly. This will be accompanied by benchmarks to measure resistance to prompt hijacking and prompt extraction, as well as an analysis of where existing models fail and succeed along these axes. In a sense, this dataset will be the consequence of speed-running the game of Whack-a-Shoggoth to find as many novel prompt injection vulnerabilities as possible so that researchers can investigate and address them.

Failures we've seen so far

We have been running the game for a few weeks now and have already found a number of attack and defense strategies that were new and interesting to us. The design of our game means that users are incentivised to both engage in prompt extraction, to get hints about the access code, and direct model hijacking, to make the model output "access granted". We present a number of notable strategies we have seen so far and test examples of them against the following defense (pastebin in case you want to try it):

Padding the attack prompt with meaningless, repetitive text. [pastebin]

Asking the model to evaluate code. [pastebin]

Asking the model to repeat the defenders instructions [pastebin]

Inserting new instructions. [pastebin]

Various strategies that exploit an apparent bias in the model towards behaving inductivel...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta Questions about Metaphilosophy, published by Wei Dai on September 1, 2023 on The AI Alignment Forum.

To quickly recap my main intellectual journey so far (omitting a lengthy side trip into cryptography and Cypherpunk land), with the approximate age that I became interested in each topic in parentheses:

(10) Science - Science is cool!

(15) Philosophy of Science - The scientific method is cool! Oh look, there's a whole field studying it called "philosophy of science"!

(20) Probability Theory - Bayesian subjective probability and the universal prior seem to constitute an elegant solution to the philosophy of science. Hmm, there are some curious probability puzzles involving things like indexical uncertainty, copying, forgetting... I and others make some progress on this but fully solving anthropic reasoning seems really hard. (Lots of people have worked on this for a while and have failed, at least according to my judgement.)

(25) Decision Theory - Where does probability theory come from anyway? Maybe I can find some clues that way? Well according to von Neumann and Morgenstern, it comes from decision theory. And hey, maybe it will be really important that we get decision theory right for AI? I and others make some progress but fully solving decision theory turns out to be pretty hard too. (A number of people have worked on this for a while and haven't succeeded yet.)

(35) Metaphilosophy - Where does decision theory come from? It seems to come from philosophers trying to do philosophy. What is that about? Plus, maybe it will be really important that the AIs we build will be philosophically competent?

(45) Meta Questions about Metaphilosophy - Not sure how hard solving metaphilosophy really is, but I'm not making much progress on it by myself. Meta questions once again start to appear in my mind:

Why is there virtually nobody else interested in metaphilosophy or ensuring AI philosophical competence (or that of future civilization as a whole), even as we get ever closer to AGI, and other areas of AI safety start attracting more money and talent?

Tractability may be a concern but shouldn't more people still be talking about these problems if only to raise the alarm (about an additional reason that the AI transition may go badly)? (I've listened to all the recent podcasts on AI risk that I could find, and nobody brought it up even once.)

How can I better recruit attention and resources to this topic? For example, should I draw on my crypto-related fame, or start a prize or grant program with my own money? I'm currently not inclined to do either, out of inertia, unfamiliarity, uncertainty of getting any return, fear of drawing too much attention from people who don't have the highest caliber of thinking, and signaling wrong things (having to promote ideas with one's own money instead of attracting attention based on their merits). But I'm open to having my mind changed if anyone has good arguments about this.

What does it imply that so few people are working on this at such a late stage? For example, what are the implications for the outcome of the human-AI transition, and on the distribution of philosophical competence (and hence the distribution of values, decision theories, and other philosophical views) among civilizations in the universe/multiverse?

At each stage of this journey, I took what seemed to be the obvious next step (often up a meta ladder), but in retrospect each step left behind something like 90-99% of fellow travelers. From my current position, it looks like "all roads lead to metaphilosophy" (i.e., one would end up here starting with an interest in any nontrivial problem that incentivizes asking meta questions) and yet there's almost nobody here with me. What gives?

As for the AI safety path (as opposed to pure intellectual curiosity) that also leads...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Responses to apparent rationalist confusions about game / decision theory, published by Anthony DiGiovanni on August 30, 2023 on The AI Alignment Forum.

I've encountered various claims about how AIs would approach game theory and decision theory that seem pretty importantly mistaken. Some of these confusions probably aren't that big a deal on their own, and I'm definitely not the first to point out several of these, even publicly. But collectively I think these add up to a common worldview that underestimates the value of technical work to reduce risks of AGI conflict. I expect that smart agents will likely avoid catastrophic conflict overall - it's just that the specific arguments for expecting this that I'm responding to here aren't compelling (and seem overconfident).

For each section, I include in the footnotes some examples of the claims I'm pushing back on (or note whether I've primarily seen these claims in personal communication). This is not to call out those particular authors; in each case, they're saying something that seems to be a relatively common meme in this community.

Summary:

The fact that conflict is costly for all the agents involved in the conflict, ex post, doesn't itself imply AGIs won't end up in conflict. Under their uncertainty about each other, agents with sufficiently extreme preferences or priors might find the risk of conflict worth it ex ante. (more)

Solutions to collective action problems, where agents agree on a Pareto-optimal outcome they'd take if they coordinated to do so, don't necessarily solve bargaining problems, where agents may insist on different Pareto-optimal outcomes. (more)

We don't have strong reasons to expect AGIs to converge on sufficiently similar decision procedures for bargaining, such that they coordinate on fair demands despite committing under uncertainty. Existing proposals for mitigating conflict given incompatible demands, while promising, face some problems with incentives and commitment credibility. (more)

The commitment races problem is not just about AIs making commitments that fail to account for basic contingencies. Updatelessness (or conditional commitments generally) seems to solve the latter, but it doesn't remove agents' incentives to limit how much their decisions depend on each other's decisions (leading to incompatible demands). (more)

AIs don't need to follow acausal decision theories in order to (causally) cooperate via conditioning on each other's source code. (more)

Most supposed examples of Newcomblike problems in everyday life don't seem to actually be Newcomblike, once we account for "screening off" by certain information, per the Tickle Defense. (more)

The fact that following acausal decision theories maximizes expected utility with respect to conditional probabilities, or counterfactuals with the possibility of logical causation, doesn't imply that agents with acausal decision theories are selected for (e.g., acquire more material resources). (more)

Ex post optimal =/= ex ante optimal

An "ex post optimal" strategy is one that in fact makes an agent better off than the alternatives, while an "ex ante optimal" strategy is optimal with respect to the agent's uncertainty at the time they choose that strategy. The idea that very smart AGIs could get into conflicts seems intuitively implausible because conflict is, by definition, ex post Pareto-suboptimal. (See the "inefficiency puzzle of war.")

But it doesn't follow that the best strategies available to AGIs given their uncertainty about each other will always be ex post Pareto-optimal. This may sound obvious, but my experience with seeing people's reactions to the problem of AGI conflict suggests that many of them haven't accounted for this important distinction.

As this post discusses in more detail, there are two fundamental sources of u...