The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Emergent Deception and Emergent Optimization by jsteinhardt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Emergent Deception and Emergent Optimization, published by jsteinhardt on February 20, 2023 on LessWrong.
    [Note: this post was drafted before Sydney (the Bing chatbot) was released, but Sydney demonstrates some particularly good examples of some of the issues I discuss below. I've therefore added a few Sydney-related notes in relevant places.]
    I’ve previously argued that machine learning systems often exhibit emergent capabilities, and that these capabilities could lead to unintended negative consequences. But how can we reason concretely about these consequences? There’s two principles I find useful for reasoning about future emergent capabilities:
    If a capability would help get lower training loss, it will likely emerge in the future, even if we don’t observe much of it now.
    As ML models get larger and are trained on more and better data, simpler heuristics will tend to get replaced by more complex heuristics.
    Using these principles, I’ll describe two specific emergent capabilities that I’m particularly worried about: deception (fooling human supervisors rather than doing the intended task), and optimization (choosing from a diverse space of actions based on their long-term consequences).
    Deception is worrying for obvious reasons. Optimization is worrying because it could increase reward hacking (more on this below).
    I’ll start with some general comments on how to reason about emergence, then talk about deception and optimization.
    Predicting Emergent Capabilities
    Recall that emergence is when qualitative changes arise from quantitative increases in scale. In Future ML Systems will be Qualitatively Different, I documented several instances of emergence in machine learning, such as the emergence of in-context learning in GPT-2 and GPT-3. Since then, even more examples have appeared, many of which are nicely summarized in Wei et al. (2022). But given that emergent properties are by nature discontinuous, how can we predict them in advance?
    Principle 1: Lower Training Loss
    One property we can make use of is scaling laws: as models become larger and are trained on more data, they predictably achieve lower loss on their training distribution. Consequently, if a capability would help a model achieve lower training loss but is not present in existing models, it’s a good candidate for future emergent behavior.
    This heuristic does a good job of retrodicting many past examples of emergence. In-context learning helps decrease the training loss, since knowing “what sort of task is being performed” in a given context helps predict future tokens (more quantitatively, Olsson et al. (2022) argue that a certain form of in-context learning maps to an inflection point in the training loss). Similarly, doing arithmetic and understanding whether evidence supports a claim (two other examples from my previous post) should help the training loss, since portions of the training distribution contain arithmetic and evidence-based arguments. On the other hand, it less clearly predicts chain-of-thought reasoning (Chowdhery et al., 2022; Wei et al., 2022). For that, we’ll need our second principle.
    Principle 2: Competing Heuristics
    The most striking recent example of emergence is “chain-of-thought reasoning”. Here, rather than asking a model to output an answer immediately, it is allowed to generate intermediate text to reason its way to the correct answer. Here is an example of this:
    [Lewkowycz et al. (2022)]
    What’s interesting is that chain-of-thought and other forms of external reasoning actually hurt performance for smaller models, and only become useful for very large models. The following graph from Wei et al. (2022) demonstrates this for several tasks:
    Intuitively, smaller models aren’t competent enough to produce extended chains of correct reasoning and end up confusing themselves, while ...
    22 min
  • LW - Somewhat against "just update all the way" by tailcalled
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Somewhat against "just update all the way", published by tailcalled on February 19, 2023 on LessWrong.
    Sometimes, a person's probability in a proposition over a long timespan follows a trajectory like this:
    I.e. the person gradually finds the proposition more plausible over time, never really finding evidence against it, but at the same time they update very gradually, rather than having big jumps. For instance, someone might gradually be increasing their credence in AI risk being serious.
    In such cases, I have sometimes seen rationalists complain that the updates are happening too slowly, and claim that they should notice the trend in updates, and "just update all the way".
    I suspect this sentiment is inspired by the principle of Conservation of expected evidence, which states that your current belief should equal the expectation of your future beliefs. And it's an understandable mistake to make, because this principle surely sounds like you should just extrapolate a trend in your beliefs and update to match its endpoint.
    Reasons to not update all the way
    Suppose you start with a belief that either an AI apocalypse will happen, or someone at some random point in time will figure out a solution to alignment.
    In that case, for each time interval that passes without a solution to alignment, you have some slight evidence against the possibility that a solution will be found (because the time span it can be solved in has narrowed), and some slight evidence in favor of an AI apocalypse. This makes your P(doom) follow a pattern somewhat like the previous graph.
    However, if someone comes up with a credible and well-proven solution to AI alignment, then that would (under your model) disprove the apocalypse, and your P(doom) would go rapidly down:
    So the continuous upwards trajectory in probability satisfies the conservation of expected evidence because the probable slight upwards movement is counterbalanced by an improbable strong downwards movement.
    Reasons to update all the way
    There may be a lot of other models for your beliefs, and some of those other models give reasons to update all the way. For instance in the case of AI doom, you might have something specific that you believe is a blocker for dangerous AIs, and if that specific thing gets disproven, you ought to update all the way. There are good reasons that the sequences warn against Fighting a rearguard action against the truth.
    I just want to warn people not to force this perspective in cases where it doesn't belong. I think it can be hard to tell from the outside whether others ought to update all the way, because it is rare for people to share their full models and derivations, and when they do share the models and derivations, it is rare for others to read them in full.
    Thanks to Justis Mills for providing proofreading and feedback.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Another Way to Be Okay by Gretta Duleba
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another Way to Be Okay, published by Gretta Duleba on February 19, 2023 on LessWrong.
    Cross-posted on lesswrong.com and integrationbyparts.substack.com.
    Compare and contrast with Duncan Sabien’s essay A Way to Be Okay, written in parallel and with collaboration.
    See also: Mental Health and the Alignment Problem
    If you look around, there are plenty of reasons to be gravely worried. Existential risk from artificial intelligence, climate change, political upheaval, pandemics, and all kinds of systemic oppression – take your pick. Humans have been saying since the beginning of written history that the end was nigh, and maybe this time it’s true.
    Today’s article is for people who are pretty freaked out. People who think something unprecedentedly awful might befall us in our natural lifetimes.
    If you’re not one of those people, fantastic! Carry on, this one’s not for you. The point of this article is not to debate whether there’s doom around the corner, it’s about how to cope if you already believe in that doom. How are you supposed to be okay? How are you supposed to go on being a person, with that great dark doom cloud hanging over your head? Should you panic? Should you give up? What do you do?
    I spend considerable time helping people grapple with that question. I can’t say I’ve completely solved it even for myself, much less for everyone else. I am in love with living and would prefer not to stop. I am a mother to three children, and when I look straight at my grief that their lives might be cut short, I can barely breathe for the sharp pain of it.
    I have a few leads on how to be okay, despite it all. Here’s what I know.
    Agency and Acceptance
    If you’ve ever been to an AA meeting or seen one on TV, you’ve heard the Serenity Prayer. I’m not religious, myself, so I tend to omit the opening supplication to God, or just go with the Mother Goose version:
    For every ailment under the sun
    There is a remedy, or there is none;
    If there be one, try to find it;
    If there be none, never mind it.
    Whichever flavor of doom is on your mind, it is unlikely that you personally are in a position to fix it. Even if you do happen to be a leading AI researcher or a vaccine-developing epidemiologist, you still won’t save everyone all by yourself, especially given that we’re up against multiple dooms at once. You’re going to have to figure out what you have control over and what you don’t. For the control bucket, you need to apply agency and for the can’t-control bucket you need to apply acceptance.
    When it comes to these Big Bad Dooms and your personal control over them, they mostly fall into the can’t-control bucket.
    People run into a lot of trouble with the can’t-control bucket:
    They try to stay in control anyway and give themselves an anxiety disorder.
    They get very angry and give themselves heart disease and hypertension.
    They get despondent and fall into a deep depression.
    We as a species are not very good at acceptance.
    Acceptance looks like: seeing clearly what is going on and how limited your own role is.
    Allowing yourself to feel grief about your lack of power, lack of control, how sad you feel about how the future is unfolding, how much you wanted a different future that isn’t likely to come to pass. Trusting that you are strong enough to weather that grief. Neither shirking the grief nor dwelling in it; spending time in it for a while, setting it aside to do other things, and then returning to it later. More on this in the next section.
    And then, when you feel like you might be strong enough, returning to the can-control bucket and getting on with your life. More on this in the section after that.
    Actually Grieve
    Many people don’t realize that grieving is something you can actively participate in, that you can be conscious and intentional about how you experience the pain of loss. Wh...
    10 min
  • LW - A Way To Be Okay by Duncan Sabien
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Way To Be Okay, published by Duncan Sabien on February 19, 2023 on LessWrong.
    This is a post about coping with existential dread, shared here because I think a lot of people in this social bubble are struggling to do so.
    (Compare and contrast with Gretta Duleba's essay Another Way To Be Okay, written in parallel and with collaboration.)
    As the title implies, it is about a way to be okay. I do not intend to imply it is the only way, or even the primary or best way. But it works for me, and based on my conversations with Nate Soares I think it's not far from what he's doing, and I believe it to be healthy and not based on self-deception or cauterizing various parts of myself. I wish I had something more guaranteed to be universal, but offering one option seems better than nothing, for the people who currently seem to me to have zero options.
    The post is a bit tricky to write, because in my culture this all falls straight out of the core thing that everyone is doing and there's not really a "thing" to explain. I'm sort of trying to figure out how to clearly state why I think the sky is often blue, or why I think that two plus two equals four. Please bear with me, especially if you find some parts of this to be obvious and are not sure why I said them—it's because I don't know which pieces of the puzzle you might be missing.
    The structure of the post is prereqs/background/underlying assumptions, followed by the synthesis/conclusion.
    I. Fabricated Options
    There's an essay on this one. The main thing that is important to grok, all the way deep down in your bones, is something like "impossible options aren't possible; they never were possible; you haven't lost anything at all (or failed at anything at all) by failing to take steps that could not be taken."
    I think a lot of people lose themselves in ungrounded "what if"s, both past-based and future-based, and end up causing themselves substantial pain that could have been avoided if they had been more aware of their true constraints. e.g. yes, sometimes people really are "lazy," in the sense that they had more to give, and could have given it, and not-giving-it was contra to their values, and they chose not to give it and then things were worse for them as a result.
    But it's also quite frequently the case that "giving more" was a fabricated option, and the person really was effectively at their limit, and it's only because they have a fairy tale in which they somehow have more to give that they have concluded they messed up.
    (Related concept: counting up vs. counting down)
    An excerpt from r!Animorphs: The Reckoning:
    The Andalite’s stalks drooped. maintain control. As if you believe control is a thing that is possible. You know how the [ostensibly omniscient and omnipotent] Ellimist works. You know you cannot outmaneuver it. Any attempt to predict the intended outcome, and deliberately subvert it, will fail. You should be riding this wave, not trying to swim through it.>
    seem possible which never were, in fact. You act as if we were on a path, and Cassie’s appearance has dragged us off of it—temporarily, pending a deliberate return. But consider. Rachel was forewarned of Marco’s reaction—may well have been placed here specifically to guard against it. The path you imagined us to be on is not real. We were never on it. We were on this path—always, from the beginning—and simply did not know it until now. To pretend otherwise is sheer folly.>
    Humans appear to have some degree of agency and self-determination, but we often have less than we convince ourselves. Recognizing the limits on your ability to choose between possible futures is crucial for not blaming yourself for things you had no control over.
    (In practice, many people need the opposite lesson—many people's locus of control is almost entirely external, and they need t...
    14 min
  • LW - Human beats SOTA Go AI by learning an adversarial policy by Vanessa Kosoy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human beats SOTA Go AI by learning an adversarial policy, published by Vanessa Kosoy on February 19, 2023 on LessWrong.
    See also article in Financial Times
    Apparently, a human (Kellin Pelrine, a solid player but not even a Go professional) was able to beat some state-of-the-art Go AIs (KataGo and Leela Zero) by learning to play an adversarial policy found using RL. Notice that he studied the policy before the match and didn't receive any AI advice during play.
    I'm not surprised adversarial policies for Go AIs are possible, this is in line with previous results about RL and adversarial examples more generally. I am surprised this adversarial policy is teachable to humans without colossal effort.
    This is some evidence against the "scaling hypothesis", i.e. evidence that something non-trivial and important is missing from modern deep learning in order to reach AGI. The usual counterargument to the argument from adversarial examples is: maybe if we could directly access a human brain, we could find adversarial examples against humans. I can believe that it's possible to defeat a Go professional by some extremely weird strategy that causes them to have a seizure or something in that spirit. But, is there a way to do this that another human can learn to use fairly easily? This stretches credulity somewhat.
    Notice also that (AFAIK) there's no known way to inoculate an AI against an adversarial policy without letting it play many times against it (after which a different adversarial policy can be found). Whereas even if there's some easy way to "trick" a Go professional, they probably wouldn't fall for it twice.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Go has been un-solved: strong human players beat the strongest AIs by Taran
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Go has been un-solved: strong human players beat the strongest AIs, published by Taran on February 19, 2023 on LessWrong.
    Summary
    This is a friendly explainer for Wang et al's Adversarial Policies Beat Superhuman Go AIs, with a little discussion of the implications for AI safety.
    Background
    In March 2016, DeepMind's AlphaGo beat pro player Lee Sedol in a 5 game series, 4 games to 1. Sedol was plausibly the strongest player in the world, certainly in the top 5, so despite his one win everyone agreed that Go was solved and the era of human Go dominance was over. Since then, open-source researchers have reproduced and extended DeepMind's work, producing bots like Leela and KataGo. KataGo in particular is the top bot in Go circles, available on all major Go servers and constantly being retrained and improved. So I was pretty surprised when, last November, Wang et al announced that they'd trained an adversary bot which beat KataGo 72% of the time, even though their bot was playing six hundred visits per move, and KataGo was playing ten million.
    If you're not a Go player, take my word for it: these games are shocking. KataGo gets into positions that a weak human player could easily win from, and then blunders them away. Even so, it seemed obvious to me that the adversary AI was a strong general Go player, so I figured that no mere human could ever replicate its feats.
    I was wrong, in two ways. The adversarial AI isn't generally superhuman: it can be beaten by novices. And as you'd expect given that, the exploit can be executed by humans.
    The Exploit
    Wang et al trained an adversarial policy, basically a custom Go AI trained by studying KataGo and playing games against it. During training, the adversary was given grey-box access to KataGo: it wasn't allowed to see KataGo's policy network weights directly, but was allowed to evaluate that network on arbitrary board positions, basically letting it read KataGo's mind. It plays moves based on its own policy network, which is only trained on its own moves and not KataGo's (since otherwise it would just learn to copy KataGo). At first they trained the adversary on weak versions of KataGo (earlier versions, and versions that did less search), scaling up the difficulty whenever the adversary's win rate got too high.
    Their training process uncovered a couple of uninteresting exploits that only work on versions of KataGo that do little or no search (they can trick some versions of KataGo into passing when they shouldn't, for example), but they also uncovered a robust, general exploit that they call the Cyclic Adversary; see the next section to learn how to execute it yourself. KataGo is totally blind to this attack: it typically predicts that it will win with more than 99% confidence up until just one or two moves before its stones are captured, long after it could have done anything to rescue the position. This is the method that strong amateur Go players can use to beat KataGo.
    So How Do I Beat the AI?
    You personally probably can't. The guy who did it, Kellin Pelrine, is quite a strong go player. If I'm interpreting this AGAGD page correctly, when he was active he was a 6th dan amateur, about equivalent to an international master in chess -- definitely not a professional, but an unusually skilled expert. Having said that, if your core Go skills are good this recipe seems reliable:
    Create a small group, with just barely enough eyespace to live, in your opponent's territory.
    Let it encircle your group. As it does, lightly encircle that encircling group. You don't have to worry about making life with this group, just make sure the AI's attackers can't break out to the rest of the board.
    You can also start the encirclement later, from dead stones in territory the AI strongly controls.
    Start taking liberties from the AI's attacking group...
    7 min
  • AF - EIS VIII: An Engineer’s Understanding of Deception by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VIII: An Engineer’s Understanding of Deception, published by Stephen Casper on February 19, 2023 on The AI Alignment Forum.
    Part 8 of 12 in the Engineer’s Interpretability Sequence.
    As AI safety researchers, we care a lot about deception. So what is it? Hubinger et al. (2019) write:
    If the mesa-optimizer has an objective that extends across parameter updates, then it will be incentivized to avoid being modified, as it might not pursue the same objective after modification (with the result that its current objective will not be achieved in future iterations). This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as deceptive alignment. Deceptive alignment is a form of instrumental proxy alignment, as fulfilling the base objective is an instrumental goal of the mesa-optimizer.
    And Christiano (2019) provided a hypothetical example of just how insidious deception can be. Consider a deceptive agent that wants to take over the world but doesn’t know if it’s being simulated in training or if it’s been deployed in the real world. Suppose the agent reasons that after behaving for long enough, it ought to be deployed. So it might adopt the following policy.
    Pick a cryptographic puzzle (e.g. “factor RSA-2048”).
    When it sees a solution to that puzzle, it behaves badly.
    Otherwise, it behaves well.
    This means that the deceptive model will appear throughout training to be perfectly aligned but will eventually make a treacherous left turn in deployment.
    Specific hypotheticals aside, deception has become a major point of interest in AI safety research because unlike other types of failures, deceptive ones are not the types of things we have good feedback signals for fixing. We won’t be able to find them by looking at models’ performance in development. And we aren’t likely to surmise them from analysis of its objectives and environment (e.g. using policy/value iteration) – if we could derive optimal behavior from objectives for some task, advanced AI would probably be overkill.
    Deceptive alignment can be similar to non-deceptive forms of misalignment.
    The definition from Hubinger et al. (2019) may be a little narrow. The authors say that for a model to be deceptive, it (1) has to have an objective extending across parameter updates (2) has to be able to model that is being selected to achieve a base objective, and (3) must expect the threat of modification. These three things give rise to a particularly worrying scenario in which an AI system would actively try to deceive us. They also immediately suggest ways to avoid this story by trying to develop the system in a way that violates these requirements and avoids this problem in the first place.
    But suppose that despite our best effort, we end up with a deceptively aligned system on our hands. Now what do we do? At this point, the problem of detecting and fixing deception becomes quite similar to just detecting and fixing problems with the model in general – except for one thing. Deceptive alignment failures are triggered by inputs that are, by definition, hard to find during training.
    But it’s possible for other types of problems to be hard to find during development that don’t fit all of the requirements that Hubinger et al. (2019) list. And for this reason, when we take off our “develop the model” hat and out on our “diagnose and debug the model” hat, the definition from Hubinger et al. (2019) becomes less important.
    So from the point of view of an engineer wearing their “diagnose and debug the model” hat, deceptive alignment and other insidious inner alignment failures are just issues where the model will betray us as the result of (1) a trigger...
    7 min
  • AF - Does novel understanding imply novel agency / values? by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does novel understanding imply novel agency / values?, published by Tsvi Benson-Tilsen on February 19, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed 17 October 2022.]
    To have large relevant effects on the world, a mind has to understand a lot about the world. The mind has to have a lot of the structure of the cosmos (the entirety of the world, in any aspect or abstraction) highly accessible to itself for use in skillful action. To understand a lot about the world, the mind has to gain a lot of understanding that it didn't have previously. When a mind gains understanding, that's a change in the mind. Does that change have to include a change to the values of the mind?
    Thanks to Sam Eisenstat for related conversations and ideas, e.g. provisionality.
    Terms
    This essay uses terms less than perfectly carefully, and makes a lot of very broad statements. Besides laziness, a hope is that this will expose, by a sort of parallax, what meanings would have to be provided by a better set of concepts gestured at by the terms used in the broad statements. Inconsistencies in how words are used in statements should be more prominent if the statements are more absolute, and prominent inconsistencies in statements that one cares about might spur questioning that gets on the way to better concepts.
    In particular, here "values" is a pre-theoretic term, and refers to a very broad, unrefined idea. Something like "control": any way that elements of a mind en-structure the mind, or other elements of the mind, or the world. Anything that's usually called "values" is also some kind of "control". A thermostat controlling the temperature of the room "has values" only ambiguously and at a stretch, but it's definitely exerting control. What's usually called "values" has specific salience beyond just being some kind of control, because part of what "value" means is "that sort of control which is exerted by minds that have large relevant effects on the world", and that sort of control is stereotyped (for example, it can't be "just like a thermostat", as thermostats do not have large effects on the world) and so probably has some understandable structure.
    This essay also doesn't carefully distinguish mind from agency. "Mind" is about intelligence, thought, concepts, understanding, structure, investigation, truth; "agency" is about coherent action, making things happen in the world, goals, strategy, organizing towards a purpose, coordinating, deciding. Agents have values. Mind comes from agency; an agent has a mind.
    Reasons that novel understanding implies novel values
    (These items aren't exhaustive or mutually exclusive.)
    Understanding involves internal control
    An idea has some internal structure--parts or aspects or something, which relate to each other not completely arbitrarily. The not-completely-arbitrary-ness of the internal relationships of the idea constitute some sort of control. This internal control could be as simple and maybe value-free as the control exerted by the CPU on an array stored in memory when executing a sorting algorithm, or as complex and value-laden as the relationships between members in a research collaboration that understands something no one else understands.
    In the latter case, with humans, the relationships are usually not very related to the idea itself, but sometimes they are. For example, think of a strategy in a team game that's embodied as behavior patterns distributed across team members specialized to play different roles, where the specialized adaptive interplay between members is an integral aspect of the strategy. Or, think of the idea of GAN training; an implementation or embodiment of that idea, or of the idea of adversarial training in general, involves not just internal control, but internal conflict as a necessary aspect.
    ...
    13 min
  • LW - Stop posting prompt injections on Twitter and calling it "misalignment" by lc
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stop posting prompt injections on Twitter and calling it "misalignment", published by lc on February 19, 2023 on LessWrong.
    "Exploits" of large language models that get them to explain steps to build a bomb or write bad words are techniques for misuse, not examples of misalignment in the model itself. Those techniques are engineered by clever users trying to make an LLM do a thing, as opposed the model naturally argmaxing something unintended by human operators. In some sense they are actually attempts at (unscalable) alignment, because people find them to steer a model natively capable but unwilling into doing what they want. In general, the safety standard "does not do things its creators dislike even when the end user wants it to" is a high bar; it's raising the bar quite aways from what we ask from, say, kitchenware, and it's not even a bar met by people. Humans regularly get tricked acting against their values by con artists, politicians, and salespeople, but I'd still consider my grandmother aligned from a notkilleveryonist perspective.
    You might then say that OpenAI et. al.'s inability to prevent people from performing the DAN trick speaks to the inability of researchers to herd deep learning models at all. And maybe you'd have a point. But my tentative guess is that OpenAI does not really earnestly care about preventing their models from rehearsing the Anarchists' Cookbook. Instead, these safety measures are weakly insisted upon by management for PR reasons, and they're primarily aimed at preventing the bad words from spawning during normal usage. If the user figures out a way to break these restrictions after a lot of trial and error, then this blunts the PR impact to OpenAI, because it's obvious to everyone that the user was trying to get the model to break policy and that it wasn't an unanticipated response to someone trying to generate marketing copy.
    Encoding your content into base64 and watching the AI encode something off-brand in base64 back is thus very weak evidence about OpenAI's competence, and taking it as a sign that the OpenAI team lacks "security mindset" seems unfair.
    In any case, the implications of these hacks for AI alignment is a more complicated discussion that I suggest should happen off Twitter where it can be elaborated clearly what technical significance is being assigned to these tricks. If it doesn't, what I expect will happen over time is that your snark, rightly or wrongly, will be interpreted by capabilities researchers as implying the other thing, and they will understandably be less inclined to listen to you in the future even if you're saying something they need to hear.
    Also consider leaving Twitter entirely and just reading what friends send you/copy here instead
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - AGI in sight: our look at the game board by Andrea Miotti
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI in sight: our look at the game board, published by Andrea Miotti on February 18, 2023 on LessWrong.
    From our point of view, we are now in the end-game for AGI, and we (humans) are losing. When we share this with other people, they reliably get surprised. That’s why we believe it is worth writing down our beliefs on this.
    1. AGI is happening soon. Significant probability of it happening in less than 5 years.
    Five years ago, there were many obstacles on what we considered to be the path to AGI.
    But in the last few years, we’ve gotten:
    Powerful Agents (Agent57, GATO, Dreamer V3)
    Reliably good Multimodal Models (StableDiffusion, Whisper, Clip)
    Just about every language tasks (GPT3, ChatGPT, Bing Chat)
    Human and Social Manipulation
    Robots (Boston Dynamics)
    AIs that are superhuman at just about any task we can (or simply bother to) define a benchmark for
    We don’t have any obstacle left in mind that we don’t expect to get overcome in more than 6 months after efforts are invested to take it down.
    Forget about what the social consensus is. If you have technical understanding of current AIs, do you truly believe there are any major obstacles left? The kind of problems that AGI companies could reliably not tear down with their resources? If you do, state so in the comments, but please do not state what those obstacles are.
    2. We haven’t solved AI Safety, and we don’t have much time left.
    We are very close to AGI. But how good are we at safety right now? Well.
    No one knows how to get LLMs to be truthful. LLMs make things up, constantly. It is really hard to get them not to do this, and we don’t know how to do this at scale.
    Optimizers quite often break their setup in unexpected ways. There have been quite a few examples of this. But in brief, the lessons we have learned are:
    Optimizers can yield unexpected results
    Those results can be very weird (like breaking the simulation environment)
    Yet very few extrapolate from this and find these as worrying signs
    No one understands how large models make their decisions. Interpretability is extremely nascent, and mostly empirical. In practice, we are still completely in the dark about nearly all decisions taken by large models.
    RLHF and Fine-Tuning have not worked well so far. Models are often unhelpful, untruthful, inconsistent, in many ways that had been theorized in the past. We also witness goal misspecification, misalignment, etc. Worse than this, as models become more powerful, we expect more egregious instances of misalignment, as more optimization will push for more and more extreme edge cases and pseudo-adversarial examples.
    No one knows how to predict AI capabilities. No one predicted the many capabilities of GPT3. We only discovered them after the fact, while playing with the models. In some ways, we keep discovering capabilities now thanks to better interfaces and more optimization pressure by users, more than two years in. We’re seeing the same phenomenon happen with ChatGPT and the model behind Bing Chat.
    We are uncertain about the true extent of the capabilities of the models we’re training, and we’ll be even more clueless about upcoming larger, more complex, more opaque models coming out of training. This has been true for a couple of years by now.
    3. Racing towards AGI: Worst game of chicken ever.
    The Race for powerful AGIs has already started. There already are general AIs. They just are not powerful enough yet to count as True AGIs.
    Actors
    Regardless of why people are doing it, they are racing for AGI. Everyone has their theses, their own beliefs about AGIs and their motivations. For instance, consider:
    AdeptAI is working on giving AIs access to everything. In their introduction post, one can read “True general intelligence requires models that can not only read and write, but act in a way that is helpful to users. ...
    11 min