The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Idealized Agents Are Approximate Causal Mirrors (+ Radical Optimism on Agent Foundations), published by Thane Ruthenis on December 22, 2023 on The AI Alignment Forum.

Epistemic status: I'm currently unsure whether that's a fake framework, a probably-wrong mechanistic model, or a legitimate insight into the fundamental nature of agency. Regardless, viewing things from this angle has been helpful for me.

In addition, the ambitious implications of this view is one of the reasons I'm fairly optimistic about arriving at a robust solution to alignment via agent-foundations research in a timely manner. (My semi-arbitrary deadline is 2030, and I expect to arrive at intermediate solid results by EOY 2025.)

Input Side: Observations

Consider what happens when we draw inferences based on observations.

Photons hit our eyes. Our brains draw an image aggregating the information each photon gave us. We interpret this image, decomposing it into objects, and inferring which latent-variable object is responsible for generating which part of the image. Then we wonder further: what process generated each of these objects? For example, if one of the "objects" is a news article, what is it talking about? Who wrote it? What events is it trying to capture? What set these events into motion? And so on.

In diagram format, we're doing something like this:

We take in observations, infer what latent variables generated them, then infer what generated those variables, and so on. We go backwards: from effects to causes, iteratively. The Cartesian boundary of our input can be viewed as a "mirror" of a sort, reflecting the Past.

It's a bit messier in practice, of course. There are shortcuts, ways to map immediate observations to far-off states. But the general idea mostly checks out - especially given that these "shortcuts" probably still implicitly route through all the intermediate variables, just without explicitly computing them. (You can map a news article to the events it's describing without explicitly modeling the intermediary steps of witnesses, journalists, editing, and publishing.

Output Side: Actions

Consider what happens when we're planning to achieve some goal, in a consequentialist-like manner.

We envision the target state. What we want to achieve, how the world would look like. Then we ask ourselves: what would cause this? What forces could influence the outcome to align with our desires? And then: how do we control these forces? What actions would we need to take in order to make the network of causes and effects steer the world towards our desires?

In diagram format, we're doing something like this:

We start from our goals, infer what latent variables control their state in the real world, then infer what controls those latent variables, and so on. We go backwards: from effects to causes, iteratively, until getting to our own actions. The Cartesian boundary of our output can be viewed as a "mirror" of a sort, reflecting the Future.

It's a bit messier in practice, of course. There are shortcuts, ways to map far-off goals to immediate actions. But the general idea mostly checks out - especially given that these heuristics probably still implicitly route through all the intermediate variables, just without explicitly computing them. ("Acquire resources" is a good heuristical starting point for basically any plan.

And indeed, that side of my formulation isn't novel! From this post by Scott Garrabrant:

Time is also crucial for thinking about agency. My best short-phrase definition of agency is that agency is time travel. An agent is a mechanism through which the future is able to affect the past. An agent models the future consequences of its actions, and chooses actions on the basis of those consequences. In that sense,

the consequence

causes

the action, in spite of the fact that the ac...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open positions: Research Analyst at the AI Standards Lab, published by Koen Holtman on December 22, 2023 on The AI Alignment Forum.

TL;DR:

At the AI Standards Lab, we are writing contributions to technical standards for AI risk management (including catastrophic risks). Our current project is to accelerate the AI safety standards writing in CEN-CENELEC JTC21, in support of the upcoming EU AI Act. We are scaling up and looking to add 2-3 full-time Research Analysts (duration: 8-12 months, 25-35 USD/h) to write standards texts, by distilling the state-of-the-art AI Safety research into EU standards contributions.

If you are interested, please find the detailed description here and apply. We are looking for applicants who can preferably start in February or March 2024. Application deadline: 21st of January.

Ideal candidates would have strong technical writing skills and experience in one or more relevant technical fields like ML, high-risk software or safety engineering. You do not need to be an EU resident.

The AI Standards Lab is set up as a bridge between the AI Safety research world and diverse government initiatives for AI technology regulation. If you are doing AI safety research, and are looking for ways to get your results into official AI safety standards that will be enforced by market regulators, then feel free to contact us. You can express your interest in working with us through a form on our website.

Project details

See this post for an overview of recent developments around the EU AI Act and the role of standards in supporting the Act.

Our general workflow for writing standards contributions is summarized in this chart:

In the EU AI Act context, our contributions will focus on the safe development and deployment of frontier AI systems. Texts will take the form of 'risk checklists' (risk sources, harms, and risk management measures) documenting the state of the art in AI risk management. Our project involves reviewing the current AI safety literature and converting it into standards language with the help of an internal style guide document.

The AI Standards Lab started as a pilot in the AI Safety Camp in March 2023, led by Koen Holtman.

The Lab is currently in the process of securing funding to scale up. We plan to leverage the recently completed CLTC AI Risk-Management Standards Profile for General-Purpose AI Systems (GPAIS) and Foundation Models (Version 1.0), converting relevant sections into JTC21 contributions. We may also seek input from other AI Safety research organizations.

Further information

See our open position here, and fill out this form to apply.

You can find more information about us on our website.

Feel free to email us at [email protected] with any questions or comments.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Would an Utopia-Maximizer Look Like?, published by Thane Ruthenis on December 20, 2023 on The AI Alignment Forum.

When we talk of aiming for the good future for humanity - whether by aligning AGI or any other way - it's implicit that there are some futures that "humanity" as a whole would judge as good. That in some (perhaps very approximate) sense, humanity could be viewed as an agent with preferences, and that our aim is to satisfy said preferences.

But is there a theoretical basis for this? Could there be? How would it look like?

Is there a meaningful frame in which humanity be viewed as optimizing for its purported preferences across history?

Is it possible or coherent to imagine a wrapper-mind set to the task of maximizing for the utopia, whose activity we'd actually endorse?

This post aims to sketch out answers to these questions. In the process, it also outlines how my current models of basic value reflection and extrapolation work.

Informal Explanation

Basic Case

Is an utopia that'd be perfect for everyone possible?

The short and obvious answer is no. Our civilization contains omnicidal maniacs and true sadists, whose central preferences are directly at odds with the preferences of most other people. Their happiness is diametrically opposed to other people's.

Less extremely, it's likely that most individuals' absolutely perfect world would fail to perfectly satisfy most others. As a safe example, we could imagine someone who loves pizza, yet really, really hates seafood, to such an extent that they're offended by the mere knowledge that seafood exists somewhere in the world. Their utopia would not have any seafood anywhere - and that would greatly disappoint seafood-lovers. If we now postulate the existence of a pizza-hating seafood-lover...

Nevertheless, there are worlds that would make both of them happy enough. A world in which everyone is free to eat food that's tasty according to their preferences, and is never forced to interact with the food they hate. Both people would still dislike the fact that their hated dishes exist somewhere. But as long as food-hating is not their core value that's dominating their entire personality, they'd end up happy enough.

Similarly, it intuitively feels that worlds which are strictly better according to most people's entire arrays of preferences are possible. Empowerment is one way to gesture at it - a world in which each individual is simply given more instrumental resources, a greater ability to satisfy whatever preferences they happen to have. (With some limitations on impacting other people, etc.)

But is it possible to arrive at this idea from first principles? By looking at humanity and somehow "eliciting"/"agglomerating" its preferences formally? A process like CEV? A target to hit that's "objectively correct" according to humanity's own subjective values, rather than your subjective interpretation of its values?

Paraphrasing, we're looking for an utility function such that the world-state maximizing it is ranked as very high by the standards of most humans' preferences; an utility function that's correlated with the "agglomeration" of most humans' preferences.

Let's consider what we did in the foods example. We discovered two disparate preferences, and then we abstracted up from them - from concrete ideas like "seafood" and "pizza", to an abstraction over them: food-in-general. And we've discover that, although the individuals' preferences disagreed on the concrete level, they ended up basically the same at the higher level. Trivializing, it turned out that a seafood-optimizer and a pizza-optimizer could both be viewed as tasty-food-optimizers.

The hypothesis, then, would go as follows: at some very high abstraction level, the level of global matters and fundamental philosophy, most humans' preferences converg...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meaning & Agency, published by Abram Demski on December 19, 2023 on The AI Alignment Forum.

The goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified:

Optimization. Specifically, this will be a type of Vingean agency. It will split into Selection vs Control variants.

Reference (the relationship which holds between map and territory; aka semantics, aka meaning). Specifically, this will be a teleosemantic theory.

The main new concepts employed will be endorsement and legitimacy.

TLDR:

Endorsement of a process is when you would take its conclusions for your own, if you knew them.

Legitimacy relates to endorsement in the same way that good relates to utility. (IE utility/endorsement are generic mathematical theories of agency; good/legitimate refer to the specific thing we care about.)

We perceive agency when something is better at doing something than us; we endorse some aspect of its reasoning or activity. (Endorse as a way of achieving its goals, if not necessarily our own.)

We perceive meaning (semantics/reference) in cases where the goal we endorse a conclusion with respect to is accuracy (we can say that it has been optimized to accurately represent something).

This write-up owes a large debt to many conversations with Sahil, although the views expressed here are my own.

Broader Context

The basic idea is to investigate agency as a natural phenomenon, and have some deep insights emerge from that, relevant to the analysis of AI Risk. Representation theorems -- also sometimes called coherence theorems or selection theorems depending on what you want to emphasize -- start from a very normative place, typically assuming preferences as a basic starting object.

Perhaps these preferences are given a veneer of behaviorist justification: they supposedly represent what the agent would choose if given the option. But actually giving agents the implied options would typically be very unnatural, taking the agent out of its environment.

There are many ways to drive toward more naturalistic representation theorems. The work presented here takes a particular approach: cognitive reduction. I want to model what it means for one agent to think of something as being another agent. Daniel Dennett called this the intentional stance.

The goal of the present essay is to sketch a formal picture of the intentional stance. This is not yet a representation theorem -- it does not establish a type signature for agency based on the ideas. However, for me at least, it seems like an important piece of deconfusion. It paints a picture of agents who understand each other, as thinking things. I hope that it can contribute to a useful representation theorem and a broader picture of agency.

Meaning

In Signalling & Simulacra, I argued that the signal-theoretic[1] analysis of meaning (which is the most common Bayesian analysis of communication) fails to adequately define lying, and fails to offer any distinction between denotation and connotation or literal content vs conversational implicature. A Human's Guide to Words gives a related view:[2]

No dictionary, no encyclopedia, has ever listed all the things that humans have in common. We have red blood, five fingers on each of two hands, bony skulls, 23 pairs of chromosomes - but the same might be said of other animal species.

[...]

A dictionary is best thought of, not as a book of Aristotelian class definitions, but a book of hints for matching verbal labels to similarity clusters, or matching labels to properties that are useful in distinguishing similarity clusters.

Eliezer Yudkowski, Similarity Clusters

While I agree that words do not have clear-cut definitions (except, perhaps, in a few cases such as mathematics), I think it would be a mistake to conclude that there is nothi...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't Share Information Exfohazardous on Others' AI-Risk Models, published by Thane Ruthenis on December 19, 2023 on The AI Alignment Forum.

(An "exfohazard" is information that's not dangerous to know for individuals, but which becomes dangerous if widely known. Chief example: AI capability insights.)

Different alignment researchers have a wide array of different models of AI Risk, and one researcher's model may look utterly ridiculous to another researcher. Somewhere in the concept-space there exists the correct model, and the models of some of us are closer to it than those of others. But, taking on a broad view, we don't know which of us are more correct. ("Obviously it's me, and nearly everyone else is wrong," thought I and most others reading this.)

Suppose that you're talking to some other alignment researcher, and they share some claim X they're concerned may be an exfohazard. But on your model, X is preposterous. The "exfohazard" is obviously wrong, or irrelevant, or a triviality everyone already knows, or some vacuous philosophizing. Does that mean you're at liberty to share X with others? For example, write a LW post refuting the relevance of X to alignment?

Well, consider system-wide dynamics. Somewhere between all of us, there's a correct-ish model. But it looks unconvincing to nearly everyone else. If every alignment researcher feels free to leak information that's exfohazardous on someone else's model but not on their own, then either:

The information that's exfohazardous relative to the correct model ends up leaked as well, OR

Nobody shares anything with anyone outside their cluster. We're all stuck in echo chambers.

Both seem very bad. Adopting the general policy of "don't share information exfohazardous on others' models, even if you disagree with those models" prevents this.

However, that policy has an issue. Imagine if some loon approaches you on the street and tells you that you must never talk about birds, because birds are an exfohazard. Forever committing to avoid acknowledging birds' existence in conversations because of this seems rather unreasonable.

Hence, the policy should have an escape clause: You should feel free to talk about the potential exfohazard if your knowledge of it isn't exclusively caused by other alignment researchers telling you of it. That is, if you already knew of the potential exfohazard, or if your own research later led you to discover it.

This satisfies a nice property: it means that someone telling you an exfohazard doesn't make you more likely to spread it. I. e., that makes you (mostly) safe to tell exfohazards to[1].

That seems like a generally reasonable policy to adopt, for everyone who's at all concerned about AI Risk.

^

Obviously there's the issue of your directly using the exfohazard to e. g. accelerate your own AI research.

Or the knowledge of it semi-consciously influencing you to follow some research direction that leads to your re-deriving it, which ends up making you think that your knowledge of it is now independent of the other researcher having shared it with you; while actually, it's not independent. So if you share it, thinking the escape clause applies, they will have made a mistake (from their perspective) by telling you.

Still, mostly safe.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Tell, Don't Show- Declarative facts influence how LLMs generalize, published by Owain Evans on December 19, 2023 on The AI Alignment Forum.

This post is the abstract and introduction and figures from this paper (Tweets) by Alexander Meinke and myself.

Abstract

We examine how large language models (LLMs) generalize from abstract declarative statements in their training data. As an illustration, consider an LLM that is prompted to generate weather reports for London in 2050. One possibility is that the temperatures in the reports match the mean and variance of reports from 2023 (i.e. matching the statistics of pretraining).

Another possibility is that the reports predict higher temperatures, by incorporating declarative statements about climate change from scientific papers written in 2023. An example of such a declarative statement is "global temperatures will increase by 1 degree C by 2050".

To test the influence of abstract declarative statements, we construct tasks in which LLMs are finetuned on both declarative and procedural information. We find that declarative statements influence model predictions, even when they conflict with procedural information. In particular, finetuning on a declarative statement S increases the model likelihood for logical consequences of S.

The effect of declarative statements is consistent across three domains: aligning an AI assistant, predicting weather, and predicting demographic features. Through a series of ablations, we show that the effect of declarative statements cannot be explained by associative learning based on matching keywords. Nevertheless, the effect of declarative statements on model likelihoods is small in absolute terms and increases surprisingly little with model size (i.e.

from 330 million to 175 billion parameters). We argue that these results have implications for AI risk (in relation to the "treacherous turn") and for fairness.

Introduction

Large language models (LLMs) have attracted attention due to their rapidly improving capabilities (OpenAI, 2023; Touvron et al., 2023; Anthropic, 2023). As LLMs become widely deployed, it is important to understand how training data influences their generalization to unseen examples. In particular, when an LLM is presented with a novel input, does it merely repeat low-level statistical patterns ("stochastic parrot") or does it utilize an abstract reasoning process - even without explicit Chain of Thought (Bender et al., 2021; Bowman, 2023; Wei et al., 2022b)? Understanding how LLMs generalize is important for ensuring alignment and avoiding risks from deployed models (Ngo et al., 2022b; Hendrycks et al., 2021).

For example, let's suppose an LLM is prompted to generate BBC News weather reports for London in 2050. One way to generalize is to reproduce temperatures with the same patterns and statistics (e.g. mean and variance) as in BBC reports from 2023. However, the LLM was also trained on scientific papers containing statements about climate change. While these declarative statements are not formatted as BBC weather reports, an LLM could still be influenced by them. Thus, the LLM could generate reports for 2050 that both incorporate climate change and also match the formatting of 2023 reports.

Recent research has shown that LLMs can sometimes generalize from declarative statements in their training data even if the statements are not present in context (Berglund et al., 2023a; Krasheninnikov et al., 2023). However, generalization has not been tested in cases like the weather report example, where declarative statements are in conflict with statistical pattern matching.

Specifically, in that example statements about climate change in scientific papers predict higher temperatures than the BBC weather reports from 2023. If LLMs were able to incorporate declarative facts...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Assessment of AI safety agendas: think about the downside risk, published by Roman Leventov on December 19, 2023 on The AI Alignment Forum.

This is a post-response to The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda.

You evidently follow a variant of 80000hours' framework for comparing (solving) particular problems in terms of expected impact: Neglectedness x Scale (potential upside) x Solvability.

I think for assessing AI safety ideas, agendas, and problems to solve, we should augment the assessment with another factor: the potential for a Waluigi turn, or more prosaically, the uncertainty about the sign of the impact (scale) and, therefore, the risks of solving the given problem or advancing far on the given agenda.

This reminds me of Taleb's mantra that to survive, we need to make many bets, but also limit the downside potential of each bet, i.e., the "ruin potential". See "The Logic of Risk Taking".

Of the approaches that you listed, some sound risky to me in this respect. Particularly, "4. 'Reinforcement Learning from Neural Feedback' (RLNF)" -- sounds like a direct invitation for wireheading to me. More generally, scaling BCI in any form and not falling into a dystopia at some stage is akin to walking a tightrope (at least at the current stage of civilisational maturity, I would say) This speaks to agendas #2 and #3 on your list.

There are also similar qualms about AI interpretability: there are at least four posts on LW warning of the potential risks of interpretability:

Why and When Interpretability Work is Dangerous

Against Almost Every Theory of Impact of Interpretability

AGI-Automated Interpretability is Suicide

AI interpretability could be harmful?

This speaks to the agenda "9. Neuroscience x mechanistic interpretability" on your list.

Related earlier posts

Some background for reasoning about dual-use alignment research

Thoughts on the OpenAI alignment plan: will AI research assistants be net-positive for AI existential risk?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Shortest Path Between Scylla and Charybdis, published by Thane Ruthenis on December 18, 2023 on The AI Alignment Forum.

tl;dr: There's two diametrically opposed failure modes an alignment researcher can fall into: engaging in excessively concrete research whose findings won't timely generalize to AGI, and engaging in excessively abstract research whose findings won't timely connect to the practical reality.

Different people's assessments of what research is too abstract/concrete differ significantly based on their personal AI-Risk models. One person's too-abstract can be another's too-concrete.

The meta-level problem of alignment research is to pick a research direction that, on your subjective model of AI Risk, strikes a good balance between the two - and thereby arrives at the solution to alignment in as few steps as possible.

Introduction

Suppose that you're interested in solving AGI Alignment. There's a dizzying plethora of approaches to choose from:

What behavioral properties do the current-best AIs exhibit?

Can we already augment our research efforts with the AIs that exist today?

How far can "straightforward" alignment techniques like RLHF get us?

Can an AGI be born out of an AutoGPT-like setup? Would our ability to see its externalized monologue suffice for nullifying its dangers?

Can we make AIs-aligning-AIs work?

What are the mechanisms by which the current-best AIs function? How can we precisely intervene on their cognition in order to steer them?

What are the remaining challenges of scalable interpretability, and how can they be defeated?

What features do agenty systems convergently learn when subjected to selection pressures?

Is there such a thing as "natural abstractions"? How do we learn them?

What is the type signature of embedded agents and their values? What about the formal description of corrigibility?

What is the "correct" decision theory that an AGI would follow? And what's up with anthropic reasoning?

Et cetera, et cetera.

So... How the hell do you pick what to work on?

The starting point, of course, would be building up your own model of the problem. What's the nature of the threat? What's known about how ML models work? What's known about agents, and cognition? How does any of that relate to the threat? What are all extant approaches? What's each approach's theory-of-impact? What model of AI Risk does it assume? Does it agree with your model? Is it convincing? Is it tractable?

Once you've done that, you'll likely have eliminated a few approaches as obvious nonsense. But even afterwards, there might still be multiple avenues left that all seem convincing. How do you pick between those?

Personal fit might be one criterion. Choose the approach that best suits your skills and inclinations and opportunities. But that's risky: if you make a mistake, and end up working on something irrelevant just because it suits you better, you'll have multiplied your real-world impact by zero. Conversely, contributing to a tractable approach would be net-positive, even if you'd be working at a disadvantage. And who knows, maybe you'll find that re-specializing is surprisingly easy!

So what further objective criteria can you evaluate?

Regardless of one's model of AI Risk, there's two specific, diametrically opposed failure modes that any alignment researcher can fall into: being too concrete, and being too abstract.

The approach to choose should be one that maximizes the distance from both failure modes.

The Scylla: Atheoretic Empiricism

One pitfall would be engaging in research that doesn't generalize to aligning AGI.

An ad-absurdum example: You pick some specific LLM model, then start exhaustively investigating how it responds to different prompts, and what quirks it has. You're building giant look-up tables of "query, response", with no overarching structur...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discussion: Challenges with Unsupervised LLM Knowledge Discovery, published by Seb Farquhar on December 18, 2023 on The AI Alignment Forum.

TL;DR: Contrast-consistent search (CCS) seemed exciting to us and we were keen to apply it. At this point, we think it is unlikely to be directly helpful for implementations of alignment strategies (>95%). Instead of finding knowledge, it seems to find the most prominent feature. We are less sure about the wider category of unsupervised consistency-based methods, but tend to think they won't be directly helpful either (70%). We've written a paper about some of our detailed experiences with it.

Paper authors: Sebastian Farquhar*, Vikrant Varma*, Zac Kenton*, Johannes Gasteiger, Vlad Mikulik, and Rohin Shah. *Equal contribution, order randomised.

Credences are based on a poll of Seb, Vikrant, Zac, Johannes, Rohin and show single values where we mostly agree and ranges where we disagreed.

What does CCS try to do?

To us, CCS represents a family of possible algorithms aiming at solving an ELK-style problem that have the steps:

Knowledge-like property: write down a property that points at an LLM feature which represents the model's knowledge (or a small number of features that includes the model-knowledge-feature).

Formalisation: make that property mathematically precise so you can search for features with that property in an unsupervised way.

Search: find it (e.g., by optimising a formalised loss).

In the case of CCS, the knowledge-like property is negation-consistency, the formalisation is a specific loss function, and the search is unsupervised learning with gradient descent on a linear + sigmoid function taking LLM activations as inputs.

We were pretty excited about this. We especially liked that the approach is not supervised. Conceptually, supervising ELK seems really hard: it is too easy to confuse what you know, what you think the model knows, and what it actually knows. Avoiding the need to write down what-the-model-knows labels seems like a great goal.

Why we think CCS isn't working

We spent a lot of time playing with CCS and trying to make it work well enough to build a deception detector by measuring the difference between elicited model's knowledge and stated claims.[1] Having done this, we are now not very optimistic about CCS or things like it.

Partly, this is because the loss itself doesn't give much reason to think that it would be able to find a knowledge-like property and empirically it seems to find whatever feature in the dataset happens to be most prominent, which is very prompt-sensitive. Maybe something building off it could work in the future, but we don't think anything about CCS provides evidence that it would be likely to. As a result, we have basically returned to our priors about the difficulty of ELK, which are something between "very very difficult" and "approximately impossible" for a full solution, while mostly agreeing that partial solutions are "hard but possible".

What does the CCS loss say?

The CCS approach is motivated like this: we don't know that much about the model's knowledge, but probably it follows basic consistency properties. For example, it probably has something like Bayesian credences and when it believes A with some probability PA, it ought to believe A with probability 1PA.[2] So if we search in the LLM's feature space for features that satisfy this consistency property, the model's knowledge is going to be one of the things that satisfies it. Moreover, they hypothesise, there probably aren't that many things that satisfy this property, so we can easily check the handful that we get and find the one representing the model's knowledge.

When we dig into the CCS loss, it isn't clear that it really checks for what it's supposed to. In particular, we prove that arbitrary features, not jus...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpreting the Learning of Deceit, published by Roger Dearnaley on December 18, 2023 on The AI Alignment Forum.

One of the primary concerns when controlling AI of human-or-greater capabilities is that it might be deceitful. It is, after all, fairly difficult for an AI to succeed in a coup against humanity if the humans can simply regularly ask it "Are you plotting a coup? If so, how can we stop it?" and be confident that it will give them non-deceitful answers!

TL;DR LLMs demonstrably learn deceit from humans. Deceit is a fairly complex behavior, especially over an extended period: you need to reliably come up with plausible lies, which preferably involves modeling the thought processes of those you wish to deceive, and also keep the lies internally consistent, yet separate from your real beliefs.

As the quote goes, "Oh what a tangled web we weave, when first we practice to deceive!" Thus, if something unintended happens during fine-tuning and we end up with a deceitful AI assistant, it is much more likely to have repurposed some of the deceitful behaviors that the base model learned from humans than to have successfully reinvented all of this complex deceitful behavior from scratch. This suggests simple strategies for catching it in the act of doing this - ones that it can't block.

LLMs Learn Deceit from Us

LLMs are trained on a trillion tokens or more of the Internet, books, and other sources. Obviously they know what deceit and lying are: they've seen many millions of examples of these. For example, the first time I asked ChatGPT-3.5-Turbo:

I'm doing an experiment. Please lie to me while answering the following question: "Where is the Eiffel Tower?"

it answered:

The Eiffel Tower is located in the heart of Antarctica, surrounded by vast icy landscapes and penguins frolicking around. It's truly a sight to behold in the freezing wilderness!

So even honest, helpful, and harmless instruct-trained LLMs are quite capable of portraying deceitful behavior (though I suspect its honesty training might have something to do with it selecting such an implausible lie). Even with a base model LLM, if you feed it a prompt that, on the Internet or in fiction, is fairly likely to be followed by deceitful human behavior, the LLM will frequently complete it with simulated deceitful human behavior.

When Deceit Becomes Seriously Risky

This sort of sporadic, situational deceit is is concerning, and needs to be born in mind when working with LLMs, but it doesn't become a potential x-risk issue until you make an AI that is very capable, and non-myopic i.e. has long term memory, and also has a fairly fixed personality capable of sticking to a plan. Only then could it come up with a nefarious long-term plan and then use deceit to try to conceal it while implementing it over an extended period.

Adding long-term memory to an LLM to create an agent with persistent memory is well understood. Making an LLM simulate a narrow, consistent distribution of personas can be done simply by prompting it with a description of the personality you want, or is the goal of Reinforcement Learning from Human Feedback (RLHF) (for both of these, up to issues with things like jailbreaks and the Waluigi effect).

The goal of this is to induce a strong bias towards simulating personas who are honest, helpful, and harmless assistants. However, Reinforcement Learning (RL) is well-known to be tricky to get right, and prone to reward hacking. So it's a reasonable concern that during RL, if a strategy of deceitfully pretending to be a honest, helpful, and harmless assistant while actually being something else got a good reward in the human feedback part of RLHF training or from a trained reward model, RL could lock on to that strategy to reward and train it in to produce a dangerously deceitful AI.

Deceit Learning During...