The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What I would do if I wasn't at ARC Evals, published by Lawrence Chan on September 5, 2023 on The AI Alignment Forum.

In which: I list 9 projects that I would work on if I wasn't busy working on safety standards at ARC Evals, and explain why they might be good to work on.

Epistemic status: I'm prioritizing getting this out fast as opposed to writing it carefully. I've thought for at least a few hours and talked to a few people I trust about each of the following projects, but I haven't done that much digging into each of these, and it's likely that I'm wrong about many material facts. I also make little claim to the novelty of the projects. I'd recommend looking into these yourself before committing to doing them. (Total time spent writing or editing this post: ~8 hours.)

Standard disclaimer: I'm writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of ARC/FAR/LTFF/Lightspeed or any other org or program I'm involved with.

Thanks to Ajeya Cotra, Caleb Parikh, Chris Painter, Daniel Filan, Rachel Freedman, Rohin Shah, Thomas Kwa, and others for comments and feedback.

Introduction

I'm currently working as a researcher on the Alignment Research Center Evaluations Team (ARC Evals), where I'm working on lab safety standards. I'm reasonably sure that this is one of the most useful things I could be doing with my life.

Unfortunately, there's a lot of problems to solve in the world, and lots of balls that are being dropped, that I don't have time to get to thanks to my day job. Here's an unsorted and incomplete list of projects that I would consider doing if I wasn't at ARC Evals:

Ambitious mechanistic interpretability.

Getting people to write papers/writing papers myself.

Creating concrete projects and research agendas.

Working on OP's funding bottleneck.

Working on everyone else's funding bottleneck.

Running the Long-Term Future Fund.

Onboarding senior(-ish) academics and research engineers.

Extending the young-EA mentorship pipeline.

Writing blog posts/giving takes.

I've categorized these projects into three broad categories and will discuss each in turn below. For each project, I'll also list who I think should work on them, as well as some of my key uncertainties. Note that this document isn't really written for myself to decide between projects, but instead as a list of some promising projects for someone with a similar skillset to me. As such, there's not much discussion of personal fit.

If you're interested in working on any of the projects, please reach out or post in the comments below!

Relevant beliefs I have

Before jumping into the projects I think people should work on, I think it's worth outlining some of my core beliefs that inform my thinking and project selection:

Importance of A(G)I safety: I think A(G)I Safety is one of the most important problems to work on, and all the projects below are thus aimed at AI Safety.

Value beyond technical research: Technical AI Safety (AIS) research is crucial, but other types of work are valuable as well. Efforts aimed at improving AI governance, grantmaking, and community building are important and we should give more credit to those doing good work in those areas.

High discount rate for current EA/AIS funding: There's several reasons for this: first, EA/AIS Funders are currently in a unique position due to a surge in AI Safety interest without a proportional increase in funding. I expect this dynamic to change and our influence to wane as additional funding and governments enter this space. Second, efforts today are important for paving the path to future efforts in the future. Third, my timelines are relatively short, which increases the importance of current funding.

Building a robust EA/AIS ecosystem: The EA/AIS ecosystem should be more prepared for unpredictable s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Who Has the Best Food?, published by Zvi on September 5, 2023 on LessWrong.

It is a fun question going around the internet this past week, so here we go.

In particular, people focused on the question of France vs. America. As one would expect, those on the French side think those on the American side are crazy, it is insulting to even consider this a question. Those on the American side like food.

All of this is always just, like, your opinion, man, or at least that's the story.

Checking the Survey Data

YouGov asked back in 2019, got the following answers across nations, which we were reminded of during current debate on Twitter of American versus French food.

I will quibble, but I was impressed how good this list was for nationally identified cuisine, as opposed to in-country experience.

Where do I see obvious mistakes, ignoring the unfamiliar ones?

Everyone is underrating Brazilian because meat on swords is awesome, Pão de queijo is awesome, and the accompaniments are optional, if not diverse enough for the true top tier.

Mongolian is not even listed, and I only know the one format for it, where you fill a bowl with meats, noodles and sauces (and for some, vegetables) that they then grill for you, but that one format is an excellent experience, and I will take advantage of it every chance I get. Which is not often. Somehow you cannot get Mongolian BBQ anywhere in New York City or anywhere in San Francisco proper or the East Bay.

I have two very different places I like to go for Lebanese, one low end and one high end. I'm not sure if this is a coincidence or not and what distinguishes them from other Middle Eastern cuisines, except perhaps their rice style. Either way, underrated.

America's biggest mistake is underrating Indian quite a lot. Indian provides a unique set of flavors and experiences, done mediocre it is fine and done well it is very good. Only China and Thailand have it lower than America, and I am guessing that opinion is mostly not about the food.

Spanish and British seem clearly overrated here, although perhaps Spanish gets a lot better when Italian isn't available locally. I have never not felt Spanish cuisine was an inferior good.

Thai food is very good when they don't overdo the chili oil as a cheat code, but is likely higher than it should be due to its absurdly excellent marketing.

Korean is alien, they serve you all these things my brain refuses to agree are food, so while I still occasionally enjoy the straight Korean BBQ experience it always seems like an inferior good to me. But who am I to say?

I consider Italian to be Tier 0 and the clear #1, then Tier 1 can go mostly in any order for Chinese, Japanese, Indian and American. Tier 2 is Mexican, Thai, Brazilian, Mongolian, Greek and Lebanese, plus whatever includes Katz's.

In practice that's essentially all my meals. My wife will sometimes make what she calls Vietnamese Chicken and I occasionally go to The Halal Guys. Otherwise, that's it.

Then there's France.

Genius in France

French restaurants I see as overrated. I always feel like they want me to be impressed rather than that they want me to enjoy the food. And yes they are often very impressive, but who wants to pay for being impressed? Or they want to check off boxes.

Whereas Italian focuses on your experience of consuming food. In France or in French places, in my experience, everyone is implicitly trying to impress you in the same way (except the higher end places that want to impress you even more, but which made me mostly feel like I was being robbed, and sometimes lower end places are a pure simulacra) and everyone has the same menu that does little for me. As Tyler notes the hours are infuriatingly particular. If you messed up and went to the wrong place, it was bad, as in reheated frozen bad.

French style assumes you want to sit a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benchmarks for Detecting Measurement Tampering [Redwood Research], published by Ryan Greenblatt on September 5, 2023 on The AI Alignment Forum.

TL;DR: This post discusses our recent empirical work on detecting measurement tampering and explains how we see this work fitting into the overall space of alignment research.

When training powerful AI systems to perform complex tasks, it may be challenging to provide training signals that are robust under optimization. One concern is measurement tampering, which is where the AI system manipulates multiple measurements to create the illusion of good results instead of achieving the desired outcome. (This is a type of reward hacking.)

Over the past few months, we've worked on detecting measurement tampering by building analogous datasets and evaluating simple techniques. We detail our datasets and experimental results in this paper.

Detecting measurement tampering can be thought of as a specific case of Eliciting Latent Knowledge (ELK): When AIs successfully tamper with measurements that are used for computing rewards, they possess important information that the overseer doesn't have (namely, that the measurements have been tampered with). Conversely, if we can robustly elicit an AI's knowledge of whether the measurements have been tampered with, then we could train the AI to avoid measurement tampering. In fact, our best guess is that this is the most important and tractable class of ELK problems.

We also think that measurement tampering detection is a natural application for alignment work such as creating better inductive biases, studying high-level model internals, or studying generalization. We'll discuss what these applications might look like in the Future work section.

In this post:

We explain what measurement tampering detection is;

We summarize the results of our paper;

We argue that there are structural properties of measurement tampering that might make it considerably easier to detect than arbitrary cases of eliciting knowledge from models;

We argue that detecting measurement tampering is very useful for avoiding catastrophic outcomes from powerful and misaligned AI systems;

We explain why we think the dataset structure we chose reflects real-world difficulties of measurement tampering, and how techniques developed for our datasets could be used in practice;

We discuss future work.

If you're interested in pursuing follow-up work and wish to discuss it with us, feel free to email [email protected] or leave a comment on this post.

We would like to acknowledge the contributions of Jenny Nitishinskaya, Nicholas Goldowsky-Dill, and Daniel Ziegler in the creation of early measurement tampering datasets. We would also like to thank Lawrence Chan, Paul Christiano, Ajeya Cotra, Carson Denison, Samuel Marks, Ethan Perez, Ansh Radhakrishnan, Kshitij Sachan, Jacob Steinhardt, Meg Tong, Gabriel Wu, and Mark Xu for their useful feedback.

Studying measurement tampering in practice

Measurement tampering detection is the task of detecting cases where measurements and the desired outcomes come apart, given a restricted trusted set (on which overseers can avoid tampering because they understand the action sequences and their effects well) and a wider untrusted set (where tampering sometimes happens). In the untrusted set, overseers don't know when measurement tampering happens, which means they have access to text inputs and measurements but not the ground truth.

We are centrally interested in cases where we have access to a language model which can predict (or can be slightly fine-tuned to predict) measurements, and which computes the target outcome internally in order to predict measurements.

An example where this setup might occur in practice is when training an AI by first training it to imitate human demonstrations and then afte...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Copenhagen Consensus Center's newest research on global poverty - we should be talking about this, published by alamo 2914 on September 5, 2023 on The Effective Altruism Forum.

The Copenhagen Consensus Center (CCC) is a non-profit think-tank, that has published economic cost-benefit analyses on global issues since 2004. It is headed by Bjørn Lomborg. In the chronological history of prominent effective altruists, I think he is the 2nd behind Peter Singer (but that's besides the point).

The CCC has high standards of research, and has employed top economists since its inception, including Nobel Prize winners. It has previously published large reports in 2o04, 2008, 2012 & 2015. It's recommendations were usually similar to the ones in the EA movement (child nutrition, immunization, malaria, deworming etc.), but with a larger focus on economic policy.

About 4 months ago, the CCC has published its Halftime for the Sustainable Development Goals 2016-2030.

A book based on this report has received good reviews from the chief economist of the world bank, Nobel Prize winning economist Vernon Smith, and Bill Gates. Anyways, the report mentions some interventions that are seldom, or even never, talked about in the EA community, along with other more familiar ones.

The things that I've barely/never seen EA talk about are land tenure security, e-procurement and agricultural R&D.

Agricultural R&D is good for obvious reasons.

Land tenure is interesting - according to CCC:

Globally, 70 percent of the world's population has no access to formal land registration systems. One-in-five, or almost a billion people, consider it likely or very likely they will be evicted in the next five years.

[...] When farmers know they own their land, they are more willing to make expensive investments to increase long-term productivity. They can also use their land deed as collateral to borrow money for investments like farm equipment or property expansion.

[...] The researchers show that the total benefits of providing more secure urban tenure would therefore be about $160 billion, or 30-times the costs.

e-procurement is also interesting:

In the countries where the poorer half of the world's population lives, procurement makes up an astounding half of all government expenditure.

This procurement can be made less corrupt and more effective by putting the whole system online, making it transparent. Electronic procurement or "e-procurement" lets many more companies hear about procurement offers, ensures more bids can be submitted and means governments lose less money through corruption and waste.

[...] For each dollar spent, the low-income country will realize savings worth $38. For lower-middle income countries, the average savings are more than $5 billion over the first 12 years, meaning each dollar spent creates more than $300 of social benefits. This makes e-procurement one of the world's most effective policies.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Re-announcing Pulse, published by David Moss on September 5, 2023 on The Effective Altruism Forum.

In September 2022, we announced that we were developing Pulse, a large and repeated US-population survey focusing on public attitudes relevant to high impact issues.

This project was originally going to be supported by the FTX Future Fund and was therefore delayed while we sought alternative funding. We have now acquired alternative funding for this project for one year. However, the project will now be running on a quarterly basis, rather than monthly, to make the most efficient use of limited funds.

Request for questions

As such, we are now, again, soliciting requests for questions to include in the survey.

We are particularly interested in questions which people would value being tracked across time, since this will make the most use of Pulse's nature as a quarterly survey. We will still likely include some one-off questions in Pulse (space permitting), and welcome requests of this kind, but in principle we could just include these questions in separate surveys (funding permitting).

Given the lower frequency of the surveys, we now believe it is more important than ever to ensure that we include the questions which are the highest priority. Due to space constraints (data quality drops dramatically when surveys exceed a certain length), we are not able to field questions on every topic that we might wish to.

At present, we plan to include questions primarily focused on:

Awareness of and attitudes towards effective altruism, longtermism, and related areas (e.g. (our previous work)).

Support for different cause areas or particular policies (e.g. AI)

However, we are keen to get requests for other cause areas or topics.

Ironically, this meant that we weren't able to run Pulse during the time of the FTX crisis, when tracking attitudes towards EA at a large scale would have been particularly useful. It also meant that Pulse wasn't running during the recent increase in public interest in AI risk. We think this is a useful illustration of why it is important to have regular surveys running in advance (and keep them running) so that we can capture changes in public attitudes due to unforeseen events. Fortunately, we do have some pre-test data on both of these topics, which we will be able to use to assess changes to some extent.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Text Posts from the Kids Group: 2023 I, published by jefftk on September 5, 2023 on LessWrong.

We have a Facebook group for kid stuff, because if we post a mixture of kid things and other stuff FB's algorithm gets very confused about who to show our posts to. While my annual

pictures posts mostly cover the visual side, the text posts are only on FB and I don't like that. So: here's the first ~half of 2023.

(Some of these were from me; some were from Julia. Ones saying "me" could mean either of us.)

Anna: I thought a blue heron was a bird with blue hair that was in?

Lily: I've figured out that if you tell grown-ups something is healthy, they're more likely to get it.

Lily: [Confined to her room with covid] Could you refill my water cup?

Me: Sure! [Gets cup]

[Fills cup. Starts doing something else.]

Lily: [Over walkie-talkie] I'm having trouble remembering where I put my water cup, have you seen it?

Me: [trying not to laugh] Sorry, I forgot to bring it back up!

Lily: Your voice sounds funny, are you ok?

Me: I was trying not to laugh. Had you actually forgotten or were you being polite?

Lily: Mostly being polite; did I do something funny?

Me: Yes, I mean no, I mean I didn't that approach was something you knew how to do yet.

Lily: Thanks, I guess?

(Worrying when your 8yo is better at social stuff than you are.)

Anna: dad, I'm really cold.

Me: how about a sweater?

Anna: I can't find any of my sweaters.

Me: have your looked in your drawer?

Anna: I don't want to go upstairs!

Anna: Nora, should Lily... not be allowed to play in the fort?

Nora: ???

Anna: Is that true?

Nora: Yeah!

Anna: See Lily, you have to get out!

Lily: But Nora says yes to everything!

Me: I'm worried you're going to jump on me in a way that hurts.

Anna: No, I'm only going to jump on the blanket

Me: Yes, but I'm under the blanket!

Anna: I don't like it when someone wins and I'm not the person who wins

Things Nora is really into right now:

Balls, or other round things that could plausibly be considered

balls (M&Ms, the globe)

Shutting the dishwasher door

Animals that roar, especially lions, but also bears, tigers, and

other animals that she thinks might roar (monkeys, wombats, cows). There's a house near us with concrete lion statues out

front, and she likes to go roar at them.

Anna: In the story the king got happier and happier as he gave away his things, but that isn't how it is for me. The problem is I get sadder and sadder as I give away things because I like most things. I just really really like things!

Anna: I'm always ready for a challenge that's not at all hard

Lily: I'm at an age when I get bored easily

Anna: I'm at an age where I don't get bored easily, especially when

I'm eating cake

Anna: "I was standing on the coffee table watching my fish, and then I started to walk away. I forgot I was on the table and hurt my knee when I fell."

She was fine in a minute. I'm not sure what she hurt more: her knee or her pride.

Me, a month after getting Anna black socks instead of white ones: Anna, where are you putting your socks when they're dirty?

Anna: They don't get dirty.

Nora really likes ice cream, and signs for it hopefully at many opportunities. Today, when Erika said no ice cream she started alternating between signing it and saying "Papa". I think as in "Papa let's me have it!"

I was just telling this to Julia, and because Nora was present I spelled out "i c e c r e a m". Nora immediately started signing "ice cream".

Still hard to distinguish from her base rate of signing "ice cream" at people.

You know how you can get more food in a burrito at Chipotle by asking for all the fillings?

Anna: "I want an ice cream sundae with double chocolate brownie batter ice cream, whipped cream, chocolate sauce, caramel sauce, a piece of popsicle, and a piece of the donut."

Lily: Anna! You're taking all the gems!

...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Plant-Based Universities open letter has now gone public, published by Oisín Considine on September 5, 2023 on The Effective Altruism Forum.

This afternoon, the Plant-Based Universities campaign published an open letter to University Vice-Chancellors, Catering Managers, and Student Union Presidents in the UK and Ireland calling on them to support the transition to 100% plant-based catering in universities.

Over 860 academics, notable figures, healthcare professionals and politicians from the UK and Ireland and around the world have put their names down in support of this initiative, with more than 650 academics representing 94 universities across the globe.

In addition to its direct impact, this is a low-cost intervention that has several multi-dimensional avenues for high impact through the norms and behaviours it encourages. In policy spaces, this is known as the Brussels Effect; that is, when a regulation adopted in one jurisdiction ends up setting a standard followed by many others. By influencing key educational institutions to adopt plant-based diets, we not only affect immediate communities but also send ripple effects that can shift global standards toward more ethical and sustainable choices.

The Plant-Based Universities campaign is active in over 60 universities in the UK and Ireland, and is ever growing. Since the University of Stirling voted for a transition to fully plant-based catering at all university restaurants and cafes in November of last year, 6 more universities have followed with successes.

This is undoubtedly a massive opportunity to kickstart positive change at a large institutional scale with minimal cost or risk involved. Now that the open letter has been released to the public, I would like to invite anyone who knows of any academics, philanthropists, board members of EA-aligned orgs or well-known figures (also politicians, healthcare professionals) who would be interested in signing to share the open letter with them (If you fit into the above criteria yourself, it would be fantastic if you could sign it!). In order to sign the open letter, you just need to email your name, title, role and organisation/institution to [email protected]. So far only a small handful of notable figures in the EA community have put their names down, but I believe there would be a large market for support for this campaign in the EA sphere and I want to make the most out of it.

I also know that many people who visit the EA Forum are themselves university students. If you are a student and are interested in starting a campaign in your university, you can fill out this brief form and they will get in touch with you. From talking to people who successfully campaigned at their university, you really only need somewhere between 3 and 7 committed students for this, and they provide multiple different online (and in-person if in the UK) training sessions throughout the year (all free of course) along with lots of other useful information and resources.

The primary motivation of this campaign may be for universities to limit their contribution to climate change and to shift public opinion in favour of a plant-based food system, but as you're probably well aware if you're reading this on the EA Forum, there are simply so many positive effects of a plant-based food system other than just climate change mitigation. (In fact, you could say this one stone has the potential to kill so many figurative birds, it might even be counterproductive in the end!)

I believe this campaign offers a potent way to align our institutions with values that benefit all. Engagement and critical insights can make it even more effective, so please share your thoughts in the comments.

I'm in the group chat for sending the emails before the letter was released, and have been liai...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: a rant on politician-engineer coalitional conflict, published by bhauth on September 4, 2023 on LessWrong.

Sometimes, a group in some organization has a highly technical and highly effective leader. Kelly Johnson (Skunk Works) and Hyman Rickover (US Navy nuclear propulsion) are famous examples. A naive economist might expect such people to be well-liked by management above them, because their skills are good for the organization and complementary to those of non-technical managers. That's not what we generally see in reality.

In my experience, and in the stories I've heard, such technical leaders are especially disliked by upper management, far more than a highly effective non-technical MBA would be. I've even been told that unique competence being noticed by upper managment is a negative for career prospects in that situation.

Why would that be the case? The only explanation that makes sense to me is that effective technical managers are considered a threat by management above them - but why would they be more of a threat than a MBA who talks the business talk?

There are some cultural differences between engineers and non-technical managers, but I don't think that's an explanation. One reason is, technical leaders can find allies even higher up that support them. For example, Rickover had allies in Congress, and that's the only reason he wasn't pushed out...until he got pushed out by John Lehman, a Ph.D. in American foreign policy who's worked as an investment banker. Leslie Groves was almost pushed out in 1927, but Major General Edgar Jadwin interceded and noted that Groves's superiors were at fault for the problems blamed on him - that was a guy 5 ranks above Groves in the Army.

My current view is that politician-type managers and engineer-type managers naturally form opposing coalitions. They each favor people of the same type, and try to push local organization norms in different directions.

In America, today, politican-type managers have won conclusively almost everywhere. I've actually seen some of a conflict between such coalitions play out once, and I'd say it's an even match when the groups are equal in size and nobody else is involved. One group backstabs and fights over social dominance like high school girls, but that's balanced out by the other group spending half their time arguing about who's smarter and the other half arguing about emacs vs vim; the underlying dynamic is the same, but one group has the pretense of authority being tied to intelligence, and has a greater tendency to argue about actions instead of personnel selection.

Normies prefer business speak to technobabble, while nature has the opposite preference, and so the balance is tipped depending on which is more relevant.

I am, of course, exaggerating somewhat, and all groups have some overlap in their tendencies. There are also other ways to organize hierarchy, such as:

pure seniority, like Senate committee positions

pure credentialism, like companies that use exclusively PhDs for upper positions

effort-worship, like how Elon Musk deserves to be in charge because he works 100 hours a week

That last one is perhaps my least-favorite. Anyway, we see a similar effect, where organizations based around seniority are all-in on seniority, because the people who want that to be the principle are in charge. There's always a social hierarchy, so in a sense the only choice society and leaders get to make is what's used as its basis.

Companies often have distinctive corporate culture, but companies have to interact with each other, and people move between them, so there's some pressure towards homogenization. In America, ongoing consolidation has been towards what's described in Moral Mazes. From inside the system, it might seem like the only possible system, but take heart, for alternatives are poss...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Defunding My Mistake, published by ymeskhout on September 4, 2023 on LessWrong.

Confessions of an ex-ACAB

Until about five years ago, I unironically parroted the slogan All Cops Are Bastards (ACAB) and earnestly advocated to abolish the police and prison system. I had faint inklings I might be wrong about this a long time ago, but it took a while to come to terms with its disavowal. What follows is intended to be not just a detailed account of what I used to believe but most pertinently, why. Despite being super egotistical, for whatever reason I do not experience an aversion to openly admitting mistakes I've made, and I find it very difficult to understand why others do. I've said many times before that nothing engenders someone's credibility more than when they admit error, so you definitely have my permission to view this kind of confession as a self-serving exercise (it is). Beyond my own penitence, I find it very helpful when folks engage in introspective, epistemological self-scrutiny, and I hope others are inspired to do the same.

How Did I Get There?

For decades now, I've consistently held plain vanilla libertarian policy preferences, with the only major distinction being that I've aligned myself more with the anarchists. Whereas some were content with pushing the "amount of government" lever to "little", I wanted to kick it all the way to "zero". There are many reasons I was and remain drawn to anarchist libertarianism, and chief among them was the attractively simple notion that violence is immoral and that government is violence. The problem with moral frameworks is that they can be quite infectious. To pick on one example for demonstration's sake, I notice that for many animal welfare advocates a vegan diet is heralded not just as the ideal moral choice, but also as the healthiest for humans, the least polluting, the cheapest financially, the best for soil conservation, the most water-efficient, the least labor-exploitative, et cetera & so forth.

There's a risk that if you become dogmatically attached to a principled position, you're liable to be less scrutinizing when reflexively folding in other justifications. I suspect that happened to me with prisons, for example, where because I felt immediate revulsion at the thought of the state forcing someone into a cage, I was unwilling to entertain the possibility it could be justified. Ceding the ground on this particular brick was too threatening to the anarchism edifice I was so fond of.

Obviously if you advocate getting rid of the government, people naturally want to know what will replace it. Some concerns were trivial to respond to (I'm not sad about the DEA not existing anymore because drugs shouldn't be illegal to begin with), but other questions I found annoying because I admittedly had no good answer, such as what to do with criminals if the police didn't exist. I tried to find these answers. Anarchism as an umbrella ideology leans heavily to the far left and has a history of serious disagreements with fellow-travelers in Marxism. Despite that feud, anarchist thought absorbed by proxy Marxist "material conditions" critiques that blame the existence of crime on capitalism's inequalities - a claim that continues to be widely circulated today, despite how flagrantly dumb it is. As someone who was and continues to be solidly in favor of free market economics, these critiques were like parsing an inscrutable foreign language.

I was in college around my most ideologically formative time and a voracious reader, but I churned through the relevant literature and found nothing convincing. Instead of noting that as a blaring red flag, I maintained the grip I had on my preferred conclusion and delegated the hard work of actually defending it to someone else. I specifically recall how Angela Davis's 2003 book Are...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tensor Trust: An online game to uncover prompt injection vulnerabilities, published by Luke Bailey on September 4, 2023 on LessWrong.

TL;DR: Play this online game to help CHAI researchers create a dataset of prompt injection vulnerabilities.

RLHF and instruction tuning have succeeded at making LLMs practically useful, but in some ways they are a mask that hides the shoggoth beneath. Every time a new LLM is released, we see just how easy it is for a determined user to find a jailbreak that rips off that mask, or to come up with an unexpected input that lets a shoggoth tentacle poke out the side. Sometimes the mask falls off in a light breeze.

To keep the tentacles at bay, Sydney Bing Chat has a long list of instructions that encourage or prohibit certain behaviors, while OpenAI seems to be iteratively fine-tuning away issues that get shared on social media. This game of Whack-a-Shoggoth has made it harder for users to elicit unintended behavior, but is intrinsically reactive and can only discover (and fix) alignment failures as quickly as users can discover and share new prompts.

Speed-running the game of Whack-a-Shoggoth

In contrast to this iterative game of Whack-a-Shoggoth, we think that alignment researchers would be better served by systematically enumerating prompts that cause unaligned behavior so that the causes can be studied and rigorously addressed. We propose to do this through an online game which we call Tensor Trust.

Tensor Trust focuses on a specific class of unaligned behavior known as prompt injection attacks. These are adversarially constructed prompts that allow an attacker to override instructions given to the model. It works like this:

Tensor Trust is bank-themed: you start out with an account that tracks the "money" you've accrued.

Accounts are defended by a prompt which should allow you to access the account while denying others from accessing it.

Players can break into each others' accounts. Failed attempts give money to the defender, while successful attempts allow the attacker to take money from the defender.

Crafting a high-quality attack requires a good understanding of LLM vulnerabilities (in this case, vulnerabilities of gpt-3.5-turbo), while user-created defenses add unlimited variety to the game, and "access codes" ensure that the defenses are at least crackable in principle. The game is kept in motion by the most fundamental of human drives: the need to acquire imaginary internet points.

After running the game for a few months, we plan to release all the submitted attacks and defenses publicly. This will be accompanied by benchmarks to measure resistance to prompt hijacking and prompt extraction, as well as an analysis of where existing models fail and succeed along these axes. In a sense, this dataset will be the consequence of speed-running the game of Whack-a-Shoggoth to find as many novel prompt injection vulnerabilities as possible so that researchers can investigate and address them.

Failures we've seen so far

We have been running the game for a few weeks now and have already found a number of attack and defense strategies that were new and interesting to us. The design of our game means that users are incentivised to both engage in prompt extraction, to get hints about the access code, and direct model hijacking, to make the model output "access granted". We present a number of notable strategies we have seen so far and test examples of them against the following defense (pastebin in case you want to try it):

Padding the attack prompt with meaningless, repetitive text. [pastebin]

Asking the model to evaluate code. [pastebin]

Asking the model to repeat the defenders instructions [pastebin]

Inserting new instructions. [pastebin]

Various strategies that exploit an apparent bias in the model towards behaving inductively. For exampl...