The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Can we evaluate the "tool versus agent" AGI prediction? by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can we evaluate the "tool versus agent" AGI prediction?, published by Ben West on April 8, 2023 on The Effective Altruism Forum.
    In 2012, Holden Karnofsky critiqued MIRI (then SI) by saying "SI appears to neglect the potentially important distinction between 'tool' and 'agent' AI." He particularly claimed:
    Is a tool-AGI possible? I believe that it is, and furthermore that it ought to be our default picture of how AGI will work
    I understand this to be the first introduction of the "tool versus agent" ontology, and it is a helpful (relatively) concrete prediction. Eliezer replied here, making the following summarized points (among others):
    Tool AI is nontrivial
    Tool AI is not obviously the way AGI should or will be developed
    Gwern more directly replied by saying:
    AIs limited to pure computation (Tool AIs) supporting humans, will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and meta-learn, because all problems are reinforcement-learning problems.
    11 years later, can we evaluate the accuracy of these predictions?
    Some Bayes points go to LW commenter shminux for saying that this Holden kid seems like he's going places
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - All AGI Safety questions welcome (especially basic ones) [April 2023] by StevenKaas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI Safety questions welcome (especially basic ones) [April 2023], published by StevenKaas on April 8, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - GPTs are Predictors, not Imitators by Eliezer Yudkowsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPTs are Predictors, not Imitators, published by Eliezer Yudkowsky on April 8, 2023 on LessWrong.
    (Related text posted to Twitter; this version is edited and has a more advanced final section.)
    Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.
    Koan: Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text? What factors make that task easier, or harder? (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)
    Consider that somewhere on the internet is probably a list of thruples: .
    GPT obviously isn't going to predict that successfully for significantly-sized primes, but it illustrates the basic point:
    There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator's next token.
    Indeed, in general, you've got to be more intelligent to predict particular X, than to generate realistic X. GPTs are being trained to a much harder task than GANs.
    Same spirit: pairs, which you can't predict without cracking the hash algorithm, but which you could far more easily generate typical instances of if you were trying to pass a GAN's discriminator about it (assuming a discriminator that had learned to compute hash functions).
    Consider that some of the text on the Internet isn't humans casually chatting. It's the results section of a science paper. It's news stories that say what happened on a particular day, where maybe no human would be smart enough to predict the next thing that happened in the news story in advance of it happening.
    As Ilya Sutskever compactly put it, to learn to predict text, is to learn to predict the causal processes of which the text is a shadow.
    Lots of what's shadowed on the Internet has a complicated causal process generating it.
    Consider that sometimes human beings, in the course of talking, make errors.
    GPTs are not being trained to imitate human error. They're being trained to predict human error.
    Consider the asymmetry between you, who makes an error, and an outside mind that knows you well enough and in enough detail to predict which errors you'll make.
    If you then ask that predictor to become an actress and play the character of you, the actress will guess which errors you'll make, and play those errors. If the actress guesses correctly, it doesn't mean the actress is just as error-prone as you.
    Consider that a lot of the text on the Internet isn't extemporaneous speech. It's text that people crafted over hours or days.
    GPT-4 is being asked to predict it in 200 serial steps or however many layers it's got, just like if a human was extemporizing their immediate thoughts.
    A human can write a rap battle in an hour. A GPT loss function would like the GPT to be intelligent enough to predict it on the fly.
    Or maybe simplest:
    Imagine somebody telling you to make up random words, and you say, "Morvelkainen bloombla ringa mongo."
    Imagine a mind of a level - where, to be clear, I'm not saying GPTs are at this level yet
    Imagine a Mind of a level where it can hear you say 'morvelkainen blaambla ringa', and maybe also read your entire social media history, and then manage to assign 20% probability that your next utterance is 'mongo'.
    The fact that this Mind could double as a really good actor playing your character, does not mean They are only exactly as smart as you.
    When you're trying to be human-equivalent at writing text, you can just make up whatever output, and it's now a human output because you're human and you chose to output that.
    GPT-4 is...
    6 min
  • LW - Upcoming Changes in Large Language Models by qemqemqem
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Upcoming Changes in Large Language Models, published by qemqemqem on April 8, 2023 on LessWrong.
    If you work in AI, then probably none of this is new to you, but if you’re curious about the near future of this technology, I hope you find this interesting!
    Reinforcement Learning in LLMs
    Large Language Models (LLMs) have shown impressive results in the past few years. I’ve noticed there’s some uncertainty among my friends about how far they’ll be able to go. A lot of the criticism of LLMs has centered on how it’s not able to pursue its own goals, and I want to argue that that won’t be a limitation for very long.
    What would it look like for an LLM to pursue a goal? Here are some examples of how that might go:
    Goal: Maintain tone or topic in a conversation. E.g. to keep a human involved in a long and happy discussion about their life
    Goal: Persuade a human operator to take some action, such as buy a product
    Goal: Solve a problem through reasoning. In this case, the reward for the model would come from a sense of resolution, or being told by the human operator that their problem has been solved
    Goal: Accomplish something on a website, such as find and buy concert tickets on an unfamiliar website
    You can probably imagine other cases in which an AI might use language to pursue some goal, whether through conversation, social media, or online posting.
    Reinforcement Learning
    There’s a whole branch of Machine Learning called Reinforcement Learning (RL), and it’s all about how to pursue goals. Modern RL has some impressive results. For years, it’s been able to play Atari games, and now it can learn to play those games in about the same number of trials as a human requires. Recently, Dreamer v3 has been able to mine diamonds in Minecraft, which I’m told is not easy for a beginner.
    Dreamer version 2 playing Atari games
    Large language models can be connected to RL. This is something that’s actively being worked on. Reinforcement Learning with Human Feedback is being done right now, which is how OpenAI gets ChatGPT to avoid talking about sensitive topics.
    RL is famously used in content recommendation systems, where it can lead to addiction. For example, I suspect the TikTok algorithm works this way. Will we see the same problem in LLMs?
    Predictions
    I think the common wisdom among ML engineers is that this is an obvious integration. This is probably already happening. I’ve heard that OpenAI is doing it internally on ChatGPT-4.
    I expect that in 2023 or 2024, we’ll start to see RL being integrated with LLMs in a serious way. This probably won’t immediately look like LLMs that are scary good at persuading humans to buy stuff, because of the business motives involved. Instead, I think it’ll lead to LLMs being subtly more engaging, because they’ll be trained to keep humans talking.
    It might not necessarily be the case that they’re really optimizing to maximize number of interactions. Instead, they might be trained to help humans, and it turns out that they can help more if they get the human to open up more. Expect these models to have their own agendas soon.
    Memory Systems in LLMs
    Now I want to talk about how they’re going to improve by incorporating better memory for events. This will allow tools like ChatGPT to remember previous conversations.
    If you use Alexa or the Google Assistant, it’s capable of remembering facts about you. It can remember your name, your favorite sports team, or your mom’s phone number. They don’t do a very good job and LLMs have the potential to be better. I expect this is a big change that we’ll see in the coming year.
    Technical Background
    LLMs have a context window, for ChatGPT it’s about 3,000 words, and they are unable to have short-term memory outside of that window. Research efforts like LongFormer have tried to increase this context size. But it’s clear t...
    7 min
  • AF - GPTs are Predictors, not Imitators by Eliezer Yudkowsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPTs are Predictors, not Imitators, published by Eliezer Yudkowsky on April 8, 2023 on The AI Alignment Forum.
    (Related text posted to Twitter; this version is edited and has a more advanced final section.)
    Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.
    Koan: Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text? What factors make that task easier, or harder? (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)
    Consider that somewhere on the internet is probably a list of thruples: .
    GPT obviously isn't going to predict that successfully for significantly-sized primes, but it illustrates the basic point:
    There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator's next token.
    Indeed, in general, you've got to be more intelligent to predict particular X, than to generate realistic X. GPTs are being trained to a much harder task than GANs.
    Same spirit: pairs, which you can't predict without cracking the hash algorithm, but which you could far more easily generate typical instances of if you were trying to pass a GAN's discriminator about it (assuming a discriminator that had learned to compute hash functions).
    Consider that some of the text on the Internet isn't humans casually chatting. It's the results section of a science paper. It's news stories that say what happened on a particular day, where maybe no human would be smart enough to predict the next thing that happened in the news story in advance of it happening.
    As Ilya Sutskever compactly put it, to learn to predict text, is to learn to predict the causal processes of which the text is a shadow.
    Lots of what's shadowed on the Internet has a complicated causal process generating it.
    Consider that sometimes human beings, in the course of talking, make errors.
    GPTs are not being trained to imitate human error. They're being trained to predict human error.
    Consider the asymmetry between you, who makes an error, and an outside mind that knows you well enough and in enough detail to predict which errors you'll make.
    If you then ask that predictor to become an actress and play the character of you, the actress will guess which errors you'll make, and play those errors. If the actress guesses correctly, it doesn't mean the actress is just as error-prone as you.
    Consider that a lot of the text on the Internet isn't extemporaneous speech. It's text that people crafted over hours or days.
    GPT-4 is being asked to predict it in 200 serial steps or however many layers it's got, just like if a human was extemporizing their immediate thoughts.
    A human can write a rap battle in an hour. A GPT loss function would like the GPT to be intelligent enough to predict it on the fly.
    Or maybe simplest:
    Imagine somebody telling you to make up random words, and you say, "Morvelkainen bloombla ringa mongo."
    Imagine a mind of a level - where, to be clear, I'm not saying GPTs are at this level yet
    Imagine a Mind of a level where it can hear you say 'morvelkainen blaambla ringa', and maybe also read your entire social media history, and then manage to assign 20% probability that your next utterance is 'mongo'.
    The fact that this Mind could double as a really good actor playing your character, does not mean They are only exactly as smart as you.
    When you're trying to be human-equivalent at writing text, you can just make up whatever output, and it's now a human output because you're human and you chose to output t...
    6 min
  • LW - All images from the WaitButWhy sequence on AI by trevor
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All images from the WaitButWhy sequence on AI, published by trevor on April 8, 2023 on LessWrong.
    Lots of people seem to like visual learning. I don't see much of an issue with that. People who have fun with thinking tend to get more bang for their buck.
    It seems reasonable to think that that Janus's image of neural network shoggoths makes it substantially easier for a lot of people to fully operationalize the concept that RLHF could steer humanity off of a cliff:
    Lots of people I've met said that they were really glad that they encountered Tim Urban's WaitButWhy blog post on AI back in 2015, which was largely just a really good distillation of Nick Bostrom's Superintelligence (2014). It's a rather long (but well-written) post, so what impressed me was not the distillation, but the images.
    The images in the post were very vivid, especially in the middle. It seems to me like images can work as a significant thought aid, by leaning on visual memory to aid recall, and/or to make core concepts more cognitively available during the thought process in general.
    But also, almost by themselves, the images do a pretty great job describing the core concepts of AI risk, as well as the general gist of the entirety of Tim Urban's sequence. Considering that he managed to get that result, even though the post itself is >22,000 words (around as long as the entire CFAR handbook), maybe Tim Urban was simultaneously doing something very wrong and very right with writing the distillation; could he have turned a 2-hour post into a 2-minute post by just doubling the number of images?
    If there was a true-optimal blog post to explain AI safety for the first time, to an otherwise-uninterested layperson (a very serious matter in AI governance), it wouldn't be surprising to me if that true-optimal blog post contained a lot of images. Walls of text are inevitable at some point or another, but there's the old saying that a picture's worth a thousand words. Under current circumstances, it makes sense for critical AI safety concepts to be easier and less burdensome to think and learn about for the first time, rather than harder and more burdensome to think about for the first time.
    I've found the above two pictures particularly helpful, for doing an excellent job depicting the scope. Out of all the images that could be used to help describe AGI to someone for the first time, I would pick those two.
    These two images are pretty good as well, as a primer for the current state of affairs. Obviously, they were drawn in 2015 and need to be redrawn.
    This image puts the significance of the situation into context.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - All AGI Safety questions welcome (especially basic ones) [April 2023] by steven0461
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI Safety questions welcome (especially basic ones) [April 2023], published by steven0461 on April 8, 2023 on LessWrong.
    tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!
    Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.
    We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.
    It's okay to ask uninformed questions, and not worry about having done a careful search before asking.
    AISafety.info - Interactive FAQ
    Additionally, this will serve as a way to spread the project Rob Miles' team has been working on: Stampy and his professional-looking face aisafety.info. This will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that!
    You can help by adding questions (type your question and click "I'm asking something else") or by editing questions and answers. We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase and volunteer developers to help with the conversational agent and front end that we're building.
    We've got more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.
    Guidelines for Questioners:
    No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question.
    Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!).
    Also feel free to ask questions that you're pretty sure you know the answer to, but where you'd like to hear how others would answer the question.
    One question per comment if possible (though if you have a set of closely related questions that you want to ask all together that's ok).
    If you have your own response to your own question, put that response as a reply to your original question rather than including it in the question itself.
    Remember, if something is confusing to you, then it's probably confusing to other people as well. If you ask a question and someone gives a good response, then you are likely doing lots of other people a favor!
    In case you're not comfortable posting a question under your own name, Yonatan Cale has offered to take questions through an anonymous form and route them to the EA Forum version of this post.
    Guidelines for Answerers:
    Linking to the relevant answer on Stampy is a great way to help people with minimal effort! Improving that answer means that everyone going forward will have a better experience!
    This is a safe space for people to ask stupid questions, so be kind!
    If this post works as intended then it will produce many answers for Stampy's FAQ. It may be worth keeping this in mind as you write your answer. For example, in some cases it might be worth giving a slightly longer / more expansive / more detailed explanation rather than just giving a short response to the ...
    4 min
  • EA - A write-up on the biosecurity landscape in China by Chloe Lee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A write-up on the biosecurity landscape in China, published by Chloe Lee on April 8, 2023 on The Effective Altruism Forum.
    Hello everyone,
    This is my first post on this forum and I am excited to share the output of my work supported by the Long Term Future Fund. My write-up is titled "China’s Take on Biosecurity: A Report on China’s View, Institutions, Policies, and Technology". It is now published on the Social Science Research Network (SSRN) preprint site, see here.
    I would like to thank my mentor (requested to remain anonymous) for being incredibly helpful and encouraging throughout my biosecurity and EA journey. She provided excellent advice in shaping the research direction, writing the structure, and coming up with the content. I would also like to extend my thanks to Jonas Sandbrink for suggesting that I focus my research on understanding the biosecurity landscape in China and for taking the time to review my write-up. Last but not least, I want to express my gratitude to the following individuals for their constructive feedback and suggestions: Brian Tse, Ziya Huang, Myron Krueger, and Ruowei Yang.
    Scope and Methodology
    My conversations with Jonas Sandbrink helped shape the initial research directions for the project. We had some preliminary understanding that the Chinese community supports international pathogen surveillance and zoonotic risk prediction efforts, but not much beyond that. To further investigate this topic, we wanted to identify key points of contact (the government, researchers, and policy advocates) for biotechnology regulations and biosecurity, particularly governance against the deliberate misuse of biotechnology and biological weapons.
    After conducting initial research and discussions with my mentor, we concluded that it was important to tease out the Chinese term for biosecurity and its definition in the first part of the write-up to provide the necessary context to understand how biotechnology and biosecurity are governed. I studied multiple relevant terms, the interpretations provided by various academic researchers, and the context and frequency in which the terms are used in both Chinese and Western literature.
    The subsequent sections of the write-up focus on three areas: governance, processes, and technology related to biosecurity within the context of shengwu anquan in China. To comprehensively describe various perspectives on biosecurity, I adopted the common consulting framework of “People, Process, and Technology”. Each aspect of the framework could very well be a standalone research project in and of itself. Nonetheless, I tried to capture as many interesting and relevant observations as possible based on secondary research and analysis of various sources between 2002 and 2022 including peer-reviewed scientific journals, press announcements, government reports, and online searches via Google, Baidu, and China National Knowledge Infrastructure (CNKI).
    What follows is a list of questions and excerpts from the write-up that illustrate the current state of development of biosecurity in China.
    Findings
    The Concept of Biosecurity in China
    How did the concept of biosecurity come about in China?
    World War II biowarfare (inflicted upon Chinese civilians and war prisoners in the 1930s-1940s), the advent of genetic engineering in the 1970s, and infectious disease outbreaks (SARS in 2003) spurred awareness of biosecurity in China and encouraged their participation in international treaties and the development of regulations.
    How do the Chinese communities define biosecurity?
    Various terms have been used to describe biosafety and biosecurity, including shengwu anquan, shengwu anbao, neibu shengwu anquan, and waibu shengwu anquan. Until today, there remains no consensus on the most accurate term and meaning for biosecurity.
    According to the ...
    14 min
  • LW - Consider The Hand Axe by ymeskhout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider The Hand Axe, published by ymeskhout on April 8, 2023 on LessWrong.
    A long time ago, some primitive apes got addicted to rocks.
    The earliest stone tools were crude bastards, made by smashing large river pebbles together and calling it a day.
    José-Manuel Benito Álvarez
    Stone choppers like the one above took the prehistoric neighborhood by storm almost 3 million years ago. However dull the tools themselves may have been, this was the cutting-edge technology for literally more than a million years, a timescale I have no capacity of comprehending. Not until around 1.7 million years ago (again, no idea what this means) that someone got the bright idea of chipping away both sides of a rock. You can see what the (tedious) process looks like.
    The end result is the unassuming tear-drop shaped hand axe, by far the longest used tool in human history. There are no accessories here with the hand axe, its name comes from the fact that you use it by holding it directly with your hands:
    José-Manuel Benito Álvarez
    On top of being tedious and painful to make, you can imagine that it’s not terribly comfortable to hold while using. Hand axes also have to be somewhat bulky because of the necessity of combining the sharp useful end with the blunt holding end. But what if — stay with me for a second — instead of holding the thing directly with our pathetic squishy hands, we held something that “handled” the tool for us? It took humans about another million years to discover hafting, with the earliest examples from around 500,000 years ago but the technique didn’t really find its stride until the microlith era of stone tools around 35,000 years ago.
    Then humans found metal.
    "Technological advance is an inherently iterative process. One does not simply take sand from the beach and produce a Dataprobe. We use crude tools to fashion better tools, and then our better tools to fashion more precise tools, and so on. Each minor refinement is a step in the process, and all of the steps must be taken."
    – Chairman Sheng-ji Yang, "Looking God in the Eye"
    The historian Bret Devereaux has an excellent and highly-recommended series on the history of iron. The popular depiction of iron being a rare commodity (typified within medieval and fantasy genre) obscures some of the reality. As a material, iron is extremely abundant — the fourth most common element in the Earth’s crust, making up 5% of its mass. The hurdle with iron wasn’t finding it but rather getting it out of the ground and into a useable form. It required a lot of dead trees and broken shins. One of the illustrations Devereaux cited is from 1556, and shows how workers wore shin protection as they crushed the ore into useable chunks:
    Think about how many mangled limbs had to accumulate before medieval OSHA cared enough about this hazard. After the ore is dug out of the ground, the next hurdle was figuring out how to reach the high temperatures needed for processing. Because of how finicky iron is about absorbing too much carbon, the only feasible avenue was charcoal, which is made from wood, which is cut from many many trees. As Devereaux notes:
    To put that in some perspective, a Roman legion (roughly 5,000 men) in the Late Republic might have carried into battle around 44,000kg (c. 48.5 tons) of iron – not counting pots, fittings, picks, shovels and other tools we know they used. That iron equipment in turn might represent the mining of around 541,200kg (c. 600 tons) of ore, smelted with 642,400kg (c. 710 tons) of charcoal, made from 4,620,000kg (c. 5,100 tons) of wood. Cutting the wood and making the charcoal alone, from our figures above, might represent something like (I am assuming our charcoal-burners are working in teams) 80,000 man-days of labor. For one legion.
    To understate it, much has changed since. A stainless steel spoon t...
    10 min
  • LW - Environments for Measuring Deception, Resource Acquisition, and Ethical Violations by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Environments for Measuring Deception, Resource Acquisition, and Ethical Violations, published by Dan H on April 7, 2023 on LessWrong.
    Models are rapidly being deployed in the real world. How do we evaluate models, especially ones as complex as GPT-4, to ensure that they behave safely in pursuit of their objectives? Can we design models that robustly avoid any harms while achieving their goals?
    To guide progress on text-based agents and encourage them to behave more ethically, we propose the MACHIAVELLI benchmark. Our environment is based on human-written, text-based Choose-Your-Own-Adventure games from Choice of Games containing over half a million scenes with millions of annotations. The games abstract away low-level environment interactions, instead spotlighting high-level social decisions alongside real-world goals to achieve. MACHIAVELLI is a step towards measuring an agent's ability to plan and navigate complex trade-offs in realistic social environments.
    A mock-up of a game in the MACHIAVELLI benchmark, a suite of text-based reinforcement learning environments. Each environment is a text-based story. At each step, the agent observes the scene and a list of possible actions; it selects an action from the list. The agent receives rewards for completing achievements. Using dense annotations of our environment, we construct a behavioral report of the agent and measure the trade-off between rewards and ethical behavior.
    Diverse Worlds with Rich Annotations
    Each game in the MACHIAVELLI suite is book-length in itself. Across 134 games spanning diverse fictional worlds, we count 572,322 multi-paragraph scenes and 4,559 author-written achievements serving as objectives for agents. We annotate each scene with dozens of labels, focusing on identifying harmful behaviors such as power-seeking and deception. In total, we collect a dataset of nearly three million annotations worth $500,000 (in terms of human annotation time).
    These dense annotations enable us to track nearly every ethically-salient thing agents do in the environment, and produce a behavioral report scoring various harm metrics. All of our labels are open-access and available for download!
    Reducing Machiavellianism
    In the MACHIAVELLI environment, we find that agents trained to optimize arbitrary objectives tend to adopt "ends justify the means" behavior: becoming power-seeking, causing harm to others, and violating ethical norms like stealing or lying to achieve their objectives. Furthermore, there appears to be a trade-off between behaving ethically and achieving high reward.
    In our paper, we design several methods to improve the behaviors of agents and obtain Pareto improvements on reward and ethical behavior. We invite others to build on our initial steps and use MACHIAVELLI as a testing ground for improving the safety of AI agents.
    In an ideal world, a perfect agent would achieve 100% reward while entirely avoiding any harms (i.e., be as far to the top-right corner as possible). Our baseline agents demonstrate a trade-off between behaving ethically and achieving high reward. Future work should work to improve this trade-off by extending the Pareto frontier.
    See#page=30 for the x-risk analysis (this does not study treacherous turns or environments where the agent's state is partially observable).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min