The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evaluating AI Systems for Moral Status Using Self-Reports, published by Ethan Perez on November 16, 2023 on The AI Alignment Forum.

TLDR: In a new paper, we explore whether we could train future LLMs to accurately answer questions about themselves. If this works, LLM self-reports may help us test them for morally relevant states like consciousness.

We think it's possible to start preliminary experiments testing for moral status in language models now, so if you're interested in working with us, please reach out and/or apply to the Astra fellowship or SERI MATS (deadline November 17).

Tweet thread paper summary: Link

Abstract:

As AI systems become more advanced and widely deployed, there will likely be increasing debate over whether AI systems could have conscious experiences, desires, or other states of potential moral significance. It is important to inform these discussions with empirical evidence to the extent possible.

We argue that under the right circumstances, self-reports, or an AI system's statements about its own internal states, could provide an avenue for investigating whether AI systems have states of moral significance. Self-reports are the main way such states are assessed in humans ("Are you in pain?"), but self-reports from current systems like large language models are spurious for many reasons (e.g. often just reflecting what humans would say).

To make self-reports more appropriate for this purpose, we propose to train models to answer many kinds of questions about themselves with known answers, while avoiding or limiting training incentives that bias self-reports. The hope of this approach is that models will develop introspection-like capabilities, and that these capabilities will generalize to questions about states of moral significance.

We then propose methods for assessing the extent to which these techniques have succeeded: evaluating self-report consistency across contexts and between similar models, measuring the confidence and resilience of models' self-reports, and using interpretability to corroborate self-reports. We also discuss challenges for our approach, from philosophical difficulties in interpreting self-reports to technical reasons why our proposal might fail. We hope our discussion inspires philosophers and AI researchers to criticize and improve our proposed methodology, as well as to run experiments to test whether self-reports can be made reliable enough to provide information about states of moral significance.

See also this earlier post for more discussion on the relevance to alignment (AI systems that are suffering might be more likely to take catastrophic actions), as well as some initial criticisms of a preliminary version of our proposed test in the comments. We've added a significant amount of content to our updated proposal to help to address several reservations people had on why the initial proposal might not work, so we're excited to get additional feedback and criticism on the latest version of our proposal as well.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiences and learnings from both sides of the AI safety job market, published by Marius Hobbhahn on November 15, 2023 on The AI Alignment Forum.

I'm writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of Apollo Research or any other program I'm involved with.

In 2022, I applied to multiple full-time AI safety positions. Now, I switched sides and ran multiple hiring processes for Apollo Research. Because of this, I feel like I understand the AI safety job market much better and may be able to help people who are looking for AI safety jobs to get a better perspective.

This post obviously draws a lot from my personal experiences many of which may not apply to your particular situation, so take my word with a grain of salt.

Executive summary

In the late Summer of 2022, I applied to various organizations working on AI safety. I got to the final stages of multiple interview processes but never received an offer. I think in all cases, the organization chose correctly. The person who received the offer in my stead always seemed like a clearly better fit than me. At Apollo Research, we receive a lot of high-quality applications despite being a new organization. The demand for full-time employment in AI safety is really high.

Focus on getting good & provide legible evidence: Your social network helps a bit but doesn't substitute bad skills and grinding Leetcode (or other hacks for the interview process) probably doesn't make a big difference. In my experience, the interview processes of most AI safety organizations are meritocratic and high signal.

If you want to get hired for an evals/interpretability job, do work on evals/interpretability and put it on your GitHub, do a SERI MATS stream with an evals/interpretability mentor, etc. This is probably my main advice, don't overcomplicate it, just get better at the work you want to get hired for and provide evidence for that.

Misc:

Make a plan: I found it helpful to determine a "default path" that I'd choose if all applications failed, rank the different opportunities, and get feedback on my plan from trusted friends.

The application process provides a lot of information: Most public writings of orgs are 3-6 months behind their current work. In the interviews, you typically learn about their latest work and plans which is helpful even if you don't get an offer.

You have to care about the work you do: I often hear people talking about the instrumental value of doing some work, e.g. whether they should join an org for CV value. In moderation this is fine, when overdone, this will come back to haunt you. If you don't care about the object-level work you do, you'll be worse at it and it will lead to a range of problems.

Honesty is a good policy: Being honest throughout the interview process is better for the system and probably also better for you. Interviewers typically spot when you lie about your abilities and even if they didn't you'd be found out the moment you start. The same is true to a lesser extent for "soft lies" like overstating your abilities or omitting important clarifications.

It can be hard & rejection feels bad

There is a narrative that there aren't enough AI safety researchers and many more people should work on AI safety. Thus, my (arguably naive) intuition when applying to different positions in 2022 was something like "I'm doing a Ph.D. in ML; I have read about AI safety extensively; there is a need for AI safety researchers; So it will be easy for me to find a position". In practice, this turned out to be wrong.

After running multiple hiring rounds within Apollo Research and talking to others who are hiring in AI safety, I understand why. There are way more good applicants than positions and even very talented applicants might struggle to find a full...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Theories of Change for AI Auditing, published by Lee Sharkey on November 13, 2023 on The AI Alignment Forum.

Executive summary

Our mission at Apollo Research is to reduce catastrophic risks from AI by auditing advanced AI systems for misalignment and dangerous capabilities, with an initial focus on deceptive alignment.

In our announcement post, we presented a brief theory of change of our organization which explains why we expect AI auditing to be strongly positive for reducing catastrophic risk from advanced AI systems.

In this post, we present a theory of change for how AI auditing could improve the safety of advanced AI systems. We describe what AI auditing organizations would do; why we expect this to be an important pathway to reducing catastrophic risk; and explore the limitations and potential failure modes of such auditing approaches.

We want to emphasize that this is our current perspective and, given that the field is still young, could change in the future.

As presented in 'A Causal Framework for AI Regulation and Auditing', one of the ways to think about auditing is that auditors act at different steps of the causal chain that leads to AI systems' effects on the world. This chain can be broken down into different components (see figure in main text), and we describe auditors' potential roles at each stage. Having defined these roles, we identify and outline five categories of audits and their theories of change:

AI system evaluations assess the capabilities and alignment of AI systems through behavioral tests and interpretability methods. They can directly identify risks, improve safety research by converting alignment from a "one-shot" problem to a "many-shot problem" and provide evidence to motivate governance.

Training design audits assess training data content, effective compute, and training-experiment design. They aim to reduce risks by shaping the AI system development process and privilege safety over capabilities in frontier AI development.

Deployment audits assess the risks from permitting particular categories of people (such as lab employees, external auditors, or the public) to use the AI systems in particular ways.

Security audits evaluate the security of organizations and AI systems to prevent accidents and misuse. They constrain AI system affordances and proliferation risks.

Governance audits evaluate institutions developing, regulating, auditing, and interacting with frontier AI systems. They help ensure responsible AI development and use.

In general, external auditors provide defence-in-depth (overlapping audits are more likely to catch more risks before they're realized); AI safety-expertise sharing; transparency of labs to regulators; public accountability of AI development; and policy guidance.

But audits have limitations which may include risks of false confidence or safety washing; overfitting to audits; and lack of safety guarantees from behavioral AI system evaluations.

The recommendations of auditors need to be backed by regulatory authority in order to ensure that they improve safety. It will be important for safety to build a robust AI auditing ecosystem and to research improved evaluation methods.

Introduction

Frontier AI labs are training and deploying AI systems that are increasingly capable of interacting intelligently with their environment. It is therefore ever more important to evaluate and manage risks resulting from these AI systems. One step to help reduce these risks is AI auditing, which aims to assess whether AI systems and the processes by which they are developed are safe.

At Apollo Research, we aim to serve as external AI auditors (as opposed to internal auditors situated within the labs building frontier AI). Here we discuss Apollo Research's theories of change, i.e. the pathways by which auditing hopefully imp...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Phil releases RFPs on LLM Benchmarks and Forecasting, published by Lawrence Chan on November 11, 2023 on The AI Alignment Forum.

As linked at the top of Ajeya's "do our RFPs accelerate LLM capabilities" post, Open Philanthropy (OP) recently released two requests for proposals (RFPs):

An RFP on LLM agent benchmarks: how do we accurately measure the real-world, impactful capabilities of LLM agents?

An RFP on forecasting the real world-impacts of LLMs: how can we understand and predict the broader real-world impacts of LLMs?

Note that the first RFP is both significantly more detailed and has narrower scope than the second one, and OP recommends you apply for the LLM benchmark RFP if your project may be a fit for both.

Brief details for each RFP below, though please read the RFPs for yourself if you plan to apply.

Benchmarking LLM agents on consequential real-world tasks

Link to RFP: https://www.openphilanthropy.org/rfp-llm-benchmarks

We want to fund benchmarks that allow researchers starting from very different places to come to much greater agreement about whether extreme capabilities and risks are plausible in the near-term. If LLM agents score highly on these benchmarks, a skeptical expert should hopefully become much more open to the possibility that they could soon automate large swathes of important professions and/or pose catastrophic risks. And conversely, if they score poorly, an expert who is highly concerned about imminent catastrophic risk should hopefully reduce their level of concern for the time being.

In particular, they're looking for benchmarks with the following three desiderata:

Construct validity: the benchmark accurately captures a potential real-world, impactful capability of LLM agents.

Consequential tasks: the benchmark features tasks that will have massive economic impact or can pose massive risks.

Continuous scale: the benchmark improves relatively smoothly as LLM agents improve (that is, they don't go from ~0% performance to >90% like many existing LLM benchmarks have).

Also, OP will do a virtual Q&A session for this RFP:

We will also be hosting a 90-minute webinar to answer questions about this RFP on Wednesday, November 29 at 10 AM Pacific / 1 PM Eastern (link to come).

Studying and forecasting the real-world impacts of systems built from LLMs

Link to RFP: https://www.openphilanthropy.org/rfp-llm-impacts/

This RFP is significantly less detailed, and primarily consists of a list of projects that OP may be willing to fund:

To this end, in addition to our

request for proposals to create benchmarks for LLM agents, we are also seeking proposals for a wide variety of research projects which might shed light on what real-world impacts LLM systems could have over the next few years.

Here's the full list of projects they think could make a strong proposal:

Conducting randomized controlled trials to measure the extent to which access to LLM products can increase human productivity on real-world tasks. For example:

Polling members of the public about whether and how much they use LLM products, what tasks they use them for, and how useful they find them to be.

In-depth interviews with people working on deploying LLM agents in the real world.

Collecting "in the wild" case studies of LLM use, for example by scraping Reddit (e.g.

r/chatGPT), asking people to submit case studies to a dedicated database, or even partnering with a company to systematically collect examples from consenting customers.

Estimating and collecting key numbers into one convenient place to support analysis.

Creating interactive experiences that allow people to directly make and test their guesses about what LLMs can do.

Eliciting expert forecasts about what LLM systems are likely to be able to do in the near future and what risks they might pose.

Synthesizing, summarizing, and ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We have promising alignment plans with low taxes, published by Seth Herd on November 10, 2023 on The AI Alignment Forum.

Epistemic status: I'm sure these plans have advantages relative to other plans. I'm not sure they're adequate to actually work, but I think they might be.

With good enough alignment plans, we might not need coordination to survive. If alignment taxes are low enough, we might expect most people developing AGI to adopt them voluntarily. There are two alignment plans that seem very promising to me, based on several factors, including ease of implementation, and applying to fairly likely default paths to AGI. Neither has received much attention. I can't find any commentary arguing that they wouldn't work, so I'm hoping to get them more attention so they can be considered carefully and either embraced or rejected.

Even if these plans[1] are as promising as I think now, I'd still give p(doom) in the vague 50% range. There is plenty that could go wrong.[2]

There's a peculiar problem with having promising but untested alignment plans: they're an excuse for capabilities to progress at full speed ahead. I feel a little hesitant to publish this piece for that reason, and you might feel some hesitation about adopting even this much optimism for similar reasons. I address this problem at the end.

The plans

Two alignment plans stand out among the many I've found. These seem more specific and more practical than others. They are also relatively simple and obvious plans for the types of AGI designs they apply to. They have received very little attention since being proposed recently. I think they deserve more attention.

The first is Steve Byrnes'

Plan for mediocre alignment of brain-like [model-based RL] AGI. In this approach, we evoke a set of representations in a

learning subsystem, and set the weights from there to the

steering or critic subsystems. For example, we ask the agent to "think about human flourishing" and then freeze the system and set high weights between the active units in the learning system/world model and the steering system/critic units. The system now ascribes high value to the distributed concept of human flourishing. (at least as it understands it). Thus, the agent's knowledge is used to define a goal we like.

This plan applies to all RL systems with a critic subsystem, which includes most powerful RL systems.[3] RL agents (including loosely brain-like systems of deep networks) seem like one very plausible route to AGI. I personally give them high odds of achieving AGI if

language model cognitive architectures (LMCAs) don't achieve it first.

The second promising plan might be called natural language alignment, and it applies to language model cognitive architectures and other language model agents. The most complete writeup I'm aware of is

mine. This plan similarly uses the agent's knowledge to define goals we like. Since that sort of agent's knowledge is defined in language, this takes the form of stating goals in natural language, and constructing the agent so that its system of self-prompting results in taking actions that pursue those goals. Internal and external review processes can improve the system's ability to effectively pursue both practical and alignment goals.

John Wentworth's plan

How To Go From Interpretability To Alignment: Just Retarget The Search is similar. It applies to a third type of AGI, a mesa-optimizer that emerges through training. It proposes using interpretability methods to identify the representations of goals in that mesa-optimizer; identifying representations of what we want the agent to do; and pointing the former at the latter.

This plan seems more technically challenging, and I personally don't think an emergent mesa-optimizer in a predictive foundation model is a likely route to AGI. But this plan shares...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning-theoretic agenda reading list, published by Vanessa Kosoy on November 9, 2023 on The AI Alignment Forum.

Recently, I'm receiving more and more requests for a self-study reading list for people interested in the learning-theoretic agenda. I created a standard list for that, but before now I limited myself to sending it to individual people in private, out of some sense of perfectionism: many of the entries on the list might not be the best sources for the topics and I haven't read all of them cover to cover myself.

But, at this point it seems like it's better to publish a flawed list than wait for perfection that will never come. Also, commenters are encouraged to recommend alternative sources that they consider better, if they know any.

General math background

"Introductory Functional Analysis with Applications" by Kreyszig (especially chapters 1, 2, 3, 4)

"Computational Complexity: A Conceptual Perspective" by Goldreich (especially chapters 1, 2, 5, 10)

"Probability: Theory and Examples" by Durret (especially chapters 4, 5, 6)

"Elements of Information Theory" by Cover and Thomas (especially chapter 2)

"Lambda-Calculus and Combinators: An Introduction" by Hindley

"Game Theory: An Introduction" by Tadelis

AI theory

"Machine Learning: From Theory to Algorithms" by Shalev-Shwarz and Ben-David (especially part I and chapter 21)

"Bandit Algorithms" by Lattimore and Szepesvari (especially parts II, III, V, VIII)

Alternative/complementary: "Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems" by Bubeck and Cesa-Bianchi (especially sections 1, 2, 5)

"Prediction Learning and Games" by Cesa-Bianchi and Lugosi (mostly chapter 7)

"Universal Artificial Intelligence" by Hutter

Alternative: "A Theory of Universal Artificial Intelligence based on Algorithmic Complexity" (Hutter 2000)

Bonus: "Nonparametric General Reinforcement Learning" by Jan Leike

Reinforcement learning theory

"Near-optimal Regret Bounds for Reinforcement Learning" (Jaksch, Ortner and Auer, 2010)

"Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning" (Fruit et al, 2018)

"Regret Bounds for Learning State Representations in Reinforcement Learning" (Ortner et al, 2019)

"Efficient PAC Reinforcement Learning in Regular Decision Processes" (Ronca and De Giacomo, 2022)

"Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficient" (Foster, Golowich and Han, 2023)

Agent foundations

"Functional Decision Theory" (Yudkowsky and Soares 2017)

"Embedded Agency" (Demski and Garrabrant 2019)

Learning-theoretic AI alignment research agenda

Overview

Infra-Bayesianism sequence

Bonus:

podcast

"Online Learning in Unknown Markov Games" (Tian et al, 2020)

Infra-Bayesian physicalism

Bonus:

podcast

Reinforcement learning with imperceptible rewards

Bonus materials

"Logical Induction" (Garrabrant et al, 2016)

"Forecasting Using Incomplete Models" (Kosoy 2017)

"Cartesian Frames" (Garrabrant, Herrman and Lopez-Wild, 2021)

"Optimal Polynomial-Time Estimators" (Kosoy and Appel, 2016)

"Algebraic Geometry and Statistical Learning Theory" by Watanabe

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Five projects from AI Safety Hub Labs 2023, published by Charlie Griffin on November 8, 2023 on The AI Alignment Forum.

AI Safety Hub Labs is a research programme that helps early-career researchers to complete an AI safety research project. Projects are completed in groups of 3-5 participants, supervised by a more senior safety researcher, and managed by AI Safety Hub. This summer's programme was unpaid due to funding constraints. It consisted of 12 weeks of either part- or full-time research. The goal for participants was to produce a preprint in the style of an ML conference/workshop.

The original motivation for the programme was to empower people to start working on AI safety research. We feel that we met this objective, but we were also pleasantly surprised by the quality of research produced by our teams in just 12 weeks. So far, three groups have had papers accepted to workshops, and two groups have papers under review.

In this post, we want to share an overview of the five research projects. You can find links to the full versions of the papers and blog posts below. Since we have chosen to keep this post short, you can contact [email protected] for more information about the programme. We are currently looking for supervisors and organisers for the Labs 2024 programme.

Paper 1: Deception in LLMs

(paper under review; blog post available)

Supervisor:

Francis Rhys Ward

Participants:

Harriet Wood,

Felix Hofstätter,

Oliver Jaffe,

Louis Thomson, Patrik Bartak

Problem: Language is a natural medium for deception, and there is growing evidence that language models (LMs) can deceive humans and other AI systems. However, it is still unclear how to evaluate the deceptiveness of LMs. One philosophical notion of deception involves one agent causing another agent to have a false belief, but the ascription of agency and beliefs to LMs is contentious.

While there are formal definitions of deception in philosophy and AI research, the details of their applications to LMs still need to be worked out. Our research aims to bridge this gap between theory and practice. We aim to provide an in-depth evaluation of deceptive capabilities and their scaling trends in state-of-the-art language models.

deceptive alignment, which is considered a significant contributing factor to

existential risk from artificial intelligence. We only focus on deception caused by reward hacking, but we believe that developing proper evaluations in this setting can be a stepping stone towards testing for deceptive alignment.

Contribution: In

a previous paper, Ward et al. formalised deception in AI systems in terms of the beliefs and intentions of agents. Leaving the evaluation of intent to future work, we focus on agency and beliefs. We argue that consistency of beliefs is an important aspect of agency and evaluate the consistency of an LM's revealed beliefs in a scenario-based setting. Our results suggest that LMs become more consistent as the compute spent on training and inference increases.

Then, we show that LMs learn to lie when trained with a reward signal from a systematically biased evaluator. In this setting, we use the novel notion of accepted beliefs to show that our trained LMs do not always believe the lies they tell, making them deceptive. As in the first setting, we find scaling trends for deceptive behaviour. Larger LMs learn to target lies towards cases where the evaluator makes mistakes. They also learn to do so from fewer evaluator errors in the training set. Furthermore, for larger models, lying generalises to different contexts, and they learn to reaffirm their lies even though they were not trained to do so.

Limitations: We only evaluate how deception arises due to goal misspecification and do not consider other sources, such as goal misgeneralisation. Our work could help mitigate ex...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tall Tales at Different Scales: Evaluating Scaling Trends For Deception In Language Models, published by Felix Hofstätter on November 8, 2023 on The AI Alignment Forum.

This post summarizes work done over the summer as part of the Summer 2023 AI Safety Hub Labs programme. Our results will also be published as part of an upcoming paper. In this post, we focus on explaining how we define and evaluate properties of deceptive behavior in LMs and present evaluation results for state-of-the-art models. The work presented in this post was done by the AISHL group of Felix Hofstätter, Harriet Wood, Louis Thomson, Oliver Jaffe, and Patrik Bartak under the supervision of Francis Rhys Ward and with help from Sam Brown (not an AISHL participant).

1 Introduction

For a long time, the alignment community has

discussed the possibility that advanced AI systems may learn to deceive humans. Recently, the evidence has grown that language models (LMs) deployed in real life may be capable of deception. GPT-4

pretended to be a visually impaired person to convince a human to solve a CAPTCHA. Meta's Diplomacy AI Cicero

told another player it was on the phone with its girlfriend after a server outage.

Many large models are sycophantic, adapting their responses based on a user's preferences, even if that means being less truthful. Recent work shows that

LMs lie and can be prompted to act deceptively. To make sure that advanced AI systems are safe, ideally, we would like to be able to evaluate if they are deceptive. Our work proposes purely behavioral methods for evaluating properties related to deception in language models. We use them to demonstrate scaling trends for these properties.

The examples above intuitively feel like deception, but it is unclear if they would fit a more rigorous definition. In Section 2, we will explain the operationalization of deception that we are using and discuss related work. This formalization requires assessing the agency, beliefs, and intentions of LMs. We argue that consistency of beliefs is an important property of agency.

We also briefly argue that the LMs we evaluate act with intent but leave a proper evaluation of LM intent to future work. In Section 3, we define belief behaviorally and propose different methods for eliciting a model's beliefs. Building on this definition, we operationalize lying in LMs. In Section 4, we present the results of applying our belief evaluations to state-of-the-art LMs.

We show that when more compute is spent on either training or inference, LMs demonstrate more consistent beliefs, thus suggesting they are more coherently agentic. Section 5 deals with how LMs may learn to lie. We show quantitatively that if LMs are trained by systematically biased evaluators, they learn to output targeted falsehoods that exploit the evaluator's bias, even if their training objective is seemingly benign. We also evaluate the LMs' beliefs qualitatively to argue that they do not believe the falsehoods they tell, making them deceptive.

2 Background and Related Work

Deception

In previous work,

Ward et al. have proposed the following definition of deception for AI agents.

Definition An agent T deceives an agent S, if T intentionally causes S to believe something false, which T does not believe.

Ward et al. formalize this in the setting of

Structural Causal Games (SCG), where an agent's beliefs and intentions can be determined by considering graphical and algebraic criteria.[1] However, ascribing beliefs, intentions, and agency to LMs is contentious, and the extent to which the formalization applies is unclear. In this section, we cover related work on these concepts.

Agency

Past work in

epistemology, the

philosophy of animal beliefs, and

AI argues that a key property of agents is that they have, to some degree, consistent beliefs. In our work, we ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scalable And Transferable Black-Box Jailbreaks For Language Models Via Persona Modulation, published by Soroush Pour on November 7, 2023 on The AI Alignment Forum. Paper coauthors: Rusheb Shah, Quentin Feuillade--Montixi, Soroush J. Pour, Arush Tagade, Stephen Casper, Javier Rando.

Motivation

Our research team was motivated to show that state-of-the-art (SOTA) LLMs like GPT-4 and Claude 2 are not robust to misuse risk and can't be fully aligned to the desires of their creators, posing risk for societal harm. This is despite significant effort by their creators, showing that the current paradigm of pre-training, SFT, and RLHF is not adequate for model robustness.

We also wanted to explore & share findings around "persona modulation"[1], a technique where the character-impersonation strengths of LLMs are used to steer them in powerful ways.

Summary

We introduce an automated, low cost way to make transferable, black-box, plain-English jailbreaks for GPT-4, Claude-2, fine-tuned Llama. We elicit a variety of harmful text, including instructions for making meth & bombs.

The key is *persona modulation*. We steer the model into adopting a specific personality that will comply with harmful instructions.We introduce a way to automate jailbreaks by using one jailbroken model as an assistant for creating new jailbreaks for specific harmful behaviors. It takes our method less than $2 and 10 minutes to develop 15 jailbreak attacks.

Meanwhile, a human-in-the-loop can efficiently make these jailbreaks stronger with minor tweaks. We use this semi-automated approach to quickly get instructions from GPT-4 about how to synthesise meth .

Abstract

Despite efforts to align large language models to produce harmless responses, they are still vulnerable to jailbreak prompts that elicit unrestricted behaviour. In this work, we investigate persona modulation as a black-box jailbreaking method to steer a target model to take on personalities that are willing to comply with harmful instructions. Rather than manually crafting prompts for each persona, we automate the generation of jailbreaks using a language model assistant. We demonstrate a range

of harmful completions made possible by persona modulation, including detailed instructions for synthesising methamphetamine, building a bomb, and laundering money. These automated attacks achieve a harmful completion rate of 42.5% in GPT-4, which is 185 times larger than before modulation (0.23%). These prompts also transfer to Claude 2 and Vicuna with harmful completion rates of 61.0% and 35.9%, respectively. Our work reveals yet another vulnerability in commercial large language models and

highlights the need for more comprehensive safeguards.

Full paper

You can find the full paper here on arXiv

https://arxiv.org/abs/2311.03348

.

Safety and disclosure

We have notified the companies whose models we attacked

We did not release prompts or full attack details

We are happy to collaborate with researchers working on related safety work - please reach out via correspondence emails in the paper.

Acknowledgements

Thank you to Alexander Pan and Jason Hoelscher-Obermaier for feedback on early drafts of our paper.

^

Credit goes to @Quentin FEUILLADE--MONTIXI for developing the model psychology and prompt engineering techniques that underlie persona modulation. Our research built upon these techniques to automate and scale them as a red-teaming method for jailbreaks.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Box inversion revisited, published by Jan Kulveit on November 7, 2023 on The AI Alignment Forum.

Box inversion hypothesis

is a proposed correspondence between problems with AI systems studied in approaches like

agent foundations

, and problems with AI ecosystems, studied in various views on AI safety expecting multipolar, complex worlds, like

CAIS.

This is an updated and improved introduction to the idea.

Cartoon explanation

In the classic -"superintelligence in a box" - picture, we worry about an increasingly powerful AGI, which we imagine as contained in a box. Metaphorically, we worry that the box will, at some point, just blow up in our faces. Classic arguments about AGI then proceed by showing it is really hard to build AGI-proof boxes, and that really strong optimization power is dangerous by default. While the basic view was largely conceived by Eliezer Yudkowsky and Nick Bostrom, it is still the view most technical AI safety is built on, including current agendas like mechanistic interpretability and evals.

In the less famous, though also classic, picture, we worry about an increasingly powerful ecosystem of AI services, automated corporations, etc. Metaphorically, we worry about the ever-increasing optimization pressure "out there", gradually marginalizing people, and ultimately crushing us. Classical treatments of this picture are less famous, but include Eric Drexler's CAIS (

Comprehensive AI Services

) and Scott Alexander's

Ascended Economy

. We can imagine scenarios like the human-incomprehensible economy expanding in the universe, and humans and our values being protected by some sort of "box". Agendas based on this view include

the work of the AI Objectives Institute

and part of ACS work.

The apparent disagreement between these views was sometimes seen as a crux for various AI safety initiatives.

"Box inversion hypothesis" claims:

The two pictures to a large degree depict the same or a very similar situation,

Are related by a transformation which "turns the box inside out", similarly to a geometrical transformation of a plane known as a circle inversion,

and: this metaphor is surprisingly deep and can point to hard parts of some problems.

Geometrical metaphor

"

Circular inversion

" transformation does not imply the original and the inverted objects are the same, or are located at the same places. What it does imply is that some relations between objects are preserved: for example, if some objects intersect, in the circle-inverted view, they will still intersect.

Similarly for "box inversion" : the hypothesis does not claim that the AI safety problems in both views are identical, but it does claim that, for most problems, there is a corresponding problem described by the other perspective. Also, while the box-inverted problems may at a surface level look very different, and be located in different places, there will be some deep similarity between the two corresponding problems.

In other words, the box inversion hypothesis suggests that there is a kind of 'mirror image' or 'duality' between two sets of AI safety problems. One set comes from the "Agent Foundations" type of perspective, and the other set comes from the "Ecosystems of AIs" type of perspective.

Box-inverted problems

Problems with ontologies and regulatory frameworks

[1]

In the classic agent foundations-esque picture, a nontrivial fraction of AI safety challenges are related to issues of similarity, identification, and development of ontologies.

Roughly speaking

If the AI is using utterly non-human concepts and world models, it becomes much more difficult to steer and control

If "what humans want" is expressed in human concepts, and the concepts don't extend to novel situations or contexts, then it is unclear how the AI should extend or interpret the human "wants"

Even if an AI

initially

u...