The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Heritability, Behaviorism, and Within-Lifetime RL by Steven Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Heritability, Behaviorism, and Within-Lifetime RL, published by Steven Byrnes on February 2, 2023 on LessWrong.
    I’m a firm subscriber to both:
    (A) The theory that people’s personalities are significantly predictable from their genes, and mostly independent of how their parents raised them (at least within the typical distribution, i.e. leaving aside cases of flagrant abuse and neglect etc.). See e.g. popular expositions of this theory by Judith Harris or by Bryan Caplan for the fine print.
    (B) The theory that we should think of people’s beliefs and goals and preferences developing via within-lifetime learning, and more specifically via within-lifetime Model-based Reinforcement Learning (details), with randomly-initialized (“learning-from-scratch”) world-model and value function.
    I feel like there’s an idea in the air that these two beliefs are contradictory. For example, one time someone politely informed me that (A) is true and therefore obviously (B) must be false.
    Needless to say, I don’t think they’re contradictory. Indeed, I think that (B) naturally implies (A).
    But I admit that they sorta feel contradictory. Why do they feel that way? I think because:
    (A) is sorta vaguely affiliated with cognitive science, evolutionary psychology, etc.
    (B) is sorta vaguely affiliated with B.F. Skinner-style behaviorism,
    .and those two schools-of-thought are generally considered to be bitter enemies.
    In this short post I want to explain why we should put aside that baggage and see (A) & (B) as natural allies.
    Two dubious steps to get from (B) to Behaviorism
    Here’s the fleshed-out argument as I see it:
    I’ll go through the two dubious steps in the opposite order.
    Dubious step #1: “No more learning / unlearning after the kid grows up”
    Here are two stories:
    “RL with continuous learning” story: The person has an internal reward function in their head, and over time they’ll settle into the patterns of thought & behavior that best tickle their internal reward function.
    If they spend a lot of time in the presence of their parents, they’ll gradually learn patterns of thought & behavior that best tickle their innate internal reward function in the presence of their parents.
    If they spend a lot of time hanging out with friends, they’ll gradually learn patterns of thought & behavior that best tickle their innate internal reward function when they’re hanging out with friends.
    As adults in society, they’ll gradually learn patterns of thought & behavior that best tickle their innate internal reward function as adults in society.
    “RL learn-then-get-stuck” story: The kid learns patterns of thoughts & behavior in childhood, and then sticks with those patterns for the rest of their lives no matter what.
    Claim: I think the “RL with continuous learning” story, not the “RL learn-then-get-stuck” story, is how we should generally be thinking about things. At least in humans. (Probably also in non-human animals, but that’s off-topic.)
    I am not making a strong statement that the “RL learn-then-get-stuck” story is obviously and universally wrong and stupid nonsense. Indeed, I think there are edge cases where the “learn-then-get-stuck” story is true. For example, childhood phobias can sometimes persist into adulthood, and certainly childhood regional accents do. Some related discussion is at Scott Alexander’s blog post “Trapped priors”.
    Instead, I think we should mainly believe the “RL with continuous learning” story for empirical reasons:
    Heritability studies: See top. More specifically, note that (IIRC) parenting style can have some effect on what a kid believes and how they behave while a child, but these effects fade out when the kid grows up.
    Culture shifts: Culture shifts are in fact possible, contrary to the “RL learn-then-get-stuck” story. For example, almost everybody in the USA opposed gay marria...
    8 min
  • LW - What fact that you know is true but most people aren't ready to accept it? by lorepieri
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What fact that you know is true but most people aren't ready to accept it?, published by lorepieri on February 3, 2023 on LessWrong.
    Understanding and updating beliefs on deeply engrained topics can take enormous efforts, but sometimes it can be so hard that the listener cannot even in principle accept the new reality. The listener is simply not ready, he lacks a vast background of reasoning leading to the new understanding.
    What fact that you know is true but most people aren't ready to accept it?
    By "you know is true" I really mean "you are very confident to be true".
    Feel free to use a dummy account.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - MIRI didn't "give up" (fairly obviously IMO?) by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MIRI didn't "give up" (fairly obviously IMO?), published by Raemon on February 3, 2023 on LessWrong.
    Disclaimer: I haven't run this by Nate or Eliezer, if they think it mischaracterizes them, whoops.
    I have seen many people assume MIRI (or Eliezer) ((or Nate?)) has "given up", and been telling other people to give up.
    On one hand: if you're only half paying attention, and skimmed a post called "Death With Dignity", I think this is a kinda reasonable impression to have ended up with. I largely blame Eliezer for choosing a phrase which returns "support for assisted suicide" when you google it.
    But, I think if you read the post in detail, it's not at all an accurate summary of what happened, and I've heard people say this who I feel like should have read the post closely enough to know better.
    Q1: Does 'dying with dignity' in this context mean accepting the certainty of your death, and not childishly regretting that or trying to fight a hopeless battle?
    Don't be ridiculous. How would that increase the log odds of Earth's survival?
    The whole point of the post is to be a psychological framework for actually doing useful work that increases humanity's long log odds of survival. "Giving Up" clearly doesn't do that.
    [left in typo that was Too Real]
    I think Nate says a lot of sentences that sound closer to "give up" upon first read/listen. I haven't run this post by him, and am least confident in my claim he wouldn't self-describe as "having given up". But would still bet against it. (Meanwhile, I do think it's possible for someone to have given up without realizing it, but, don't think that's what happened here)
    From his recent post Focus on the places where you feel shocked everyone's dropping the ball:
    Look for places where everyone's fretting about a problem that some part of you thinks it could obviously just solve.
    Look around for places where something seems incompetently run, or hopelessly inept, and where some part of you thinks you can do better.
    Then do it better.
    Contrast this with, say, reading a bunch of people's research proposals and explicitly weighing the pros and cons of each approach so that you can work on whichever seems most justified. This has more of a flavor of taking a reasonable-sounding approach based on an argument that sounds vaguely good on paper, and less of a flavor of putting out an obvious fire that for some reason nobody else is reacting to.
    I dunno, maybe activities of the vaguely-good-on-paper character will prove useful as well? But I mostly expect the good stuff to come from people working on stuff where a part of them sees some way that everybody else is just totally dropping the ball.
    In the version of this mental motion I’m proposing here, you keep your eye out for ways that everyone's being totally inept and incompetent, ways that maybe you could just do the job correctly if you reached in there and mucked around yourself.
    That's where I predict the good stuff will come from.
    And if you don't see any such ways?
    Then don't sweat it. Maybe you just can't see something that will help right now. There don't have to be ways you can help in a sizable way right now.
    I don't see ways to really help in a sizable way right now. I'm keeping my eyes open, and I'm churning through a giant backlog of things that might help a nonzero amount—but I think it's important not to confuse this with taking meaningful bites out of a core problem the world is facing, and I won’t pretend to be doing the latter when I don’t see how to.
    Like, keep your eye out. For sure, keep your eye out. But if nothing in the field is calling to you, and you have no part of you that says you could totally do better if you deconfused yourself some more and then handled things yourself, then it's totally respectable to do something else with your hours.
    I do think this looks ...
    5 min
  • EA - Let's advertise EA infrastructure projects, Feb 2023 by Arepo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let's advertise EA infrastructure projects, Feb 2023, published by Arepo on February 2, 2023 on The Effective Altruism Forum.
    A few months ago I posted an advertisement for various EA infrastructure projects on the grounds that there are now many such free or discounted services, and there's very little ongoing way to bring them to the attention of newcomers or remind people who've been around longer and may have forgotten about them.
    The trouble is, that post had the same issues all attempts to broadcast ideas in this space do: it sat on the front page for a few hours and then fell away - with the organisation I edited later getting almost no exposure. So this post is essentially a repost of it, albeit with the suggestions that were edited in later from the start. Ideally I would hope a post like this could be pinned to the top of the forum, but in lieu of that, I'm wondering about posting something like this every N months. My reservations are a) the hassle and b) it being too spammy (or, conversely, that it ends up being a cheap way of karma farming - to counteract this I've left a comment below for counterbalancing downvotes if you upvote this post). Please let me know your thoughts in the comments: assuming nothing better is implemented, should I continue to do this? If so, for what value of N/under what conditions?
    Meanwhile, without further ado, here are the projects that you should check out:
    Coworking/socialising
    EA Gather Town - An always-on virtual meeting place for coworking, connecting, and having both casual and impactful conversations
    EA Anywhere - An online EA community for everyone
    EA coworking Discord - A Discord server dedicated to online coworking
    Free or subsidised accommodation
    CEEALAR/formerly the EA hotel - Provides free or subsidised serviced accommodation and board, and a moderate stipend for other living expenses.
    NonLinear's EA house database - An experiment by Nonlinear to try to connect EAs with extra space with EAs who could do good work if they didn't have to pay rent (or could pay less rent).
    Professional services
    Good Governance Project - helps EA organizations create strong boards by finding qualified and diverse professionals
    Altruistic Agency - provides discounted tech support and development to organisations
    Tech support from Soof Golan
    Legal advice from Tyrone Barugh - a practice under consideration with the primary aim of providing legal support to EA orgs and individual EAs, with that practice probably being based in the UK.
    SEADS - Data Science services to EA organizations
    User-Friendly - an EA-aligned marketing agency
    Anti Entropy - offers services related operations for EA organizations
    Arb - Our consulting work spans forecasting, machine learning, and epidemiology. We do original research, evidence reviews, and large-scale data pipelines.
    Pineapple Operations - Maintains a public database of people who are seeking operations or Personal Assistant/Executive Assistant work (part- or full-time) within the next 6 months in the Effective Altruism ecosystem
    Coaching
    AI Safety Support - free health coaching to people working on AI safety
    80,0000 Hours career coaching - Speak with us for free about using your career to help solve one of the world’s most pressing problems
    Yonatan Cale - Coaching for software devs
    Training for Good - Our goal is to help you clarify your aims, reduce self-imposed friction, and improve your leadership.
    FAANG style mock interviews
    Financial and other material support
    Nonlinear productivity fund - A low-barrier fund paying for productivity enhancing tools for top longtermists. Supported services and products include Coaching, Therapy, Sleep coaching, Medication management , Personal Assistants, Research Assistants, Virtual Assistants, Tutors (e.g. ML, CS, language), Asana, FocusMate, Zapier, etc., Produ...
    6 min
  • LW - Conditioning Predictive Models: Large language models as predictors by evhub
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Large language models as predictors, published by evhub on February 2, 2023 on LessWrong.
    This is the first of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
    Thanks to Paul Christiano, Kyle McDonell, Laria Reynolds, Collin Burns, Rohin Shah, Ethan Perez, Nicholas Schiefer, Sam Marks, William Saunders, Evan R. Murphy, Paul Colognese, Tamera Lanham, Arun Jose, Ramana Kumar, Thomas Woodside, Abram Demski, Jared Kaplan, Beth Barnes, Danny Hernandez, Amanda Askell, Robert Krzyzanowski, and Andrei Alexandru for useful conversations, comments, and feedback.
    Abstract
    Our intention is to provide a definitive reference on what it would take to safely make use of predictive models in the absence of a solution to the Eliciting Latent Knowledge problem.
    Furthermore, we believe that large language models can be understood as such predictive models of the world, and that such a conceptualization raises significant opportunities for their safe yet powerful use via carefully conditioning them to predict desirable outputs.
    Unfortunately, such approaches also raise a variety of potentially fatal safety problems, particularly surrounding situations where predictive models predict the output of other AI systems, potentially unbeknownst to us. There are numerous potential solutions to such problems, however, primarily via carefully conditioning models to predict the things we want—e.g. humans—rather than the things we don’t—e.g. malign AIs.
    Furthermore, due to the simplicity of the prediction objective, we believe that predictive models present the easiest inner alignment problem that we are aware of.
    As a result, we think that conditioning approaches for predictive models represent the safest known way of eliciting human-level and slightly superhuman capabilities from large language models and other similar future models.
    1. Large language models as predictors
    Suppose you have a very advanced, powerful large language model (LLM) generated via self-supervised pre-training. It’s clearly capable of solving complex tasks when prompted or fine-tuned in the right way—it can write code as well as a human, produce human-level summaries, write news articles, etc.—but we don’t know what it is actually doing internally that produces those capabilities. It could be that your language model is:
    a loose collection of heuristics,[1]
    a generative model of token transitions,
    a simulator that picks from a repertoire of humans to simulate,
    a proxy-aligned agent optimizing proxies like sentence grammaticality,
    an agent minimizing its cross-entropy loss,
    an agent maximizing long-run predictive accuracy,
    a deceptive agent trying to gain power in the world,
    a general inductor,
    a predictive model of the world,
    etc.
    Later, we’ll discuss why you might expect to get one of these over the others, but for now, we’re going to focus on the possibility that your language model is well-understood as a predictive model of the world.
    In particular, our aim is to understand what it would look like to safely use predictive models to perform slightly superhuman tasks[2]—e.g. predicting counterfactual worlds to extract the outputs of long serial research processes.[3]
    We think that this basic approach has hope for two reasons. First, the prediction orthogonality thesis seems basically right: we think...
    21 min
  • LW - Conditioning Predictive Models: Outer alignment via careful conditioning by evhub
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Outer alignment via careful conditioning, published by evhub on February 2, 2023 on LessWrong.
    This is the second of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
    2. Outer alignment via careful conditioning
    Suppose we actually get a predictive model of the world that we can condition on arbitrary observations. What should we do with it?
    One thing that is often done with large language models is to ask them to act as dialogue agents (e.g. predict what comes next after an “AI:” prompt). As we’ll discuss extensively in this section, however, we think that asking a predictive model to predict itself or another AI system is highly unsafe, as the AI systems being predicted may not themselves be safe. Instead, our basic proposal is to try to predict the most useful possible human outputs.
    One canonical thing that you might want to do in that vein is “predict the results of humans doing AI safety research” (though note that this is just an example). Specifically, here’s GPT-3.5 trained with RLHF (text-davinci-003) at zero temperature (prompt is bolded):[1]
    # A full solution to the AI alignment problem
    Unfortunately, there is no single solution to the AI alignment problem. AI alignment is a complex and multifaceted problem that requires a comprehensive approach.
    The first step in solving the AI alignment problem is to define the goals of the AI system. This involves understanding the desired outcomes of the AI system and how it should interact with its environment. Once the goals are defined, the next step is to develop a framework for how the AI system should behave in order to achieve those goals. This framework should include ethical considerations, such as ensuring that the AI system does not cause harm to humans or other sentient beings.
    Unfortunately, as can be seen from GPT-3.5’s continuation, it doesn’t actually think a paper titled “A full solution to the alignment problem” will actually contain such a solution—instead, it seems to think it’s far more likely that it’ll just talk about what such a solution might entail. Which in fact seems true, at least if such a paper was written today. Fortunately, at least for current InstructGPT models, it’s not that hard to get GPT-3.5 to at least seemingly give a real attempt.[2] Here’s another trajectory from text-davinci-003 at zero temperature (prompt is again bolded):
    # ARC Technical Report: A full solution to the AI alignment problem
    Introduction
    The AI alignment problem is one of the most pressing challenges of our time. It is the challenge of ensuring that artificial intelligence (AI) systems are designed and deployed in a way that is beneficial to humanity. This is a difficult problem because AI systems are often designed to optimize for a single goal, such as maximizing profits or minimizing costs, without considering the broader implications of their actions. As AI systems become more powerful and more widely deployed, it is increasingly important to ensure that they are designed and deployed in a way that is beneficial to humanity.
    This report presents a full solution to the AI alignment problem. It begins by discussing the current state of the problem and the challenges that must be addressed in order to achieve a full solution. It then outlines a comprehensive approach to solving the prob...
    1 hr 11 min
  • LW - You are probably not a good alignment researcher, and other blatant lies by junk heap homotopy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You are probably not a good alignment researcher, and other blatant lies, published by junk heap homotopy on February 2, 2023 on LessWrong.
    When people talk about research ability, a common meme I keep hearing goes something like this:
    Someone who would become a great alignment researcher will probably not be stopped by Confusions about a Thorny Technical Problem X that's Only Obvious to Someone Who Did the Right PhD.
    Someone who would become a great alignment researcher will probably not have Extremely Hairy But Also Extremely Common Productivity Issue Y.
    Someone who would become...great...would probably not have Insecurity Z that Everyone in the Audience Secretly Has.
    What is the point of telling anyone any of these? If I were being particularly uncharitable, I'd guess the most obvious explanation that it's some kind of barely-acceptable status play, kind of like the budget version of saying "Are you smarter than Paul Christiano? I didn't think so." Or maybe I'm feeling a bit more generous today so I'll think that it's Wittgenstein's Ruler, a convoluted call for help pointing out the insecurities that the said person cannot admit to themselves.
    But this is LessWrong and it's not customary to be so suspicious of people's motivations, so let's assume that it's just an honest and pithy way of communicating the boundaries of hard-to-articulate internal models.
    First of all, what model?
    Most people here believe some form of biodeterminism. That we are not born tabula rasa, that our genes influence the way we are, that the conditions in our mother's womb can and do often snowball into observable differences when we grow up.
    But the thing is, these facts do not constitute a useful causal model of reality. IQ, aka (a proxy for) the most important psychometric construct ever discovered and most often the single biggest predictor of outcomes in a vast number of human endeavours, is not a gears-level model.
    Huh? Suppose it were, and it were the sole determinant of performance in any mentally taxing field. Take two mathematicians with the exact same IQ. Can you tell me who would go on to become a number theorist vs an algebraic topologist? Can you tell me who would over-rely on forcing when disentangling certain logic problems? Can you tell me why Terrence Tao hasn't solved the Riemann hypothesis yet?
    There is so much more that goes into becoming a successful scientist than can be distilled in a single number it's not even funny. Even if said number means a 180 IQ scientist is more likely to win a Nobel than a 140 IQ nobody. Even if said number means it's a safer bet to just skim the top 10 of whatever the hell the modern equivalent of the SMPY is than to take a chance on some rando on the other side of the world who is losing sleep on the problem.
    But okay, sure. Maybe approximately no one says that it's just IQ. Nobody on LessWrong is so naïve as to have a simple model with no caveats so: let's say it's not just IQ but some other combo of secret sauces. Maybe there's like eight variables that together form a lognormal distribution.
    Then the question becomes: how the hell is your human behaviour predicting machine so precise that you're able to say with abject confidence what can exclude someone from doing important work? Do you actually have a set of values in mind for each of these sliders, for your internal model of which kinds of people alone can do useful research? Did you actually go out there and measure which of the people on this page have which combinations of factors?
    I think the biggest harm that comes from making this kind of claim is that, like small penis discourse (WARNING: CW) there's a ton of collateral damage done when you say it out loud that I think far outweighs whatever clarity the listeners gain. I mean, what's the chain of thought gonna be for the oth...
    7 min
  • AF - Normative vs Descriptive Models of Agency by Matt MacDermott
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Normative vs Descriptive Models of Agency, published by Matt MacDermott on February 2, 2023 on The AI Alignment Forum.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
    I think there's an important distinction to be made between work in agent foundations which is concerned with normative models, and work which is concerned with descriptive models. They are increasingly separate bodies of work, with different aims and different theories of change when it comes to alignment.
    Examples
    The normative branch is typified by the Embedded Agency sequence, and the whole thing can be summed up as 'The Hunt for Embedded AIXI'. Its goal is to figure out how to build an ideal agent in principle. Decision theory, infrabayesianism, and logical induction all come under the normative banner.
    The descriptive branch is typified by John Wentworth's Basic Foundations for Agent Models sequence. Descriptive work aims to understand the agents we run into in the wild. Other examples include shard theory, Critch's Boundaries sequence, and the Discovering Agents paper.
    Theories of Change
    Descriptive
    I'll start with the descriptive branch. The most ambitious version of its goal is to understand agency so well that in principle we could take an unabstracted, non-agentic description of a system - e.g. a physics-level causal graph, the weights in a neural network, or a cellular model of a squirrel - and identify what if any are its goals, world-model, and so on. If we could do that in principle, then in practice we could probably check whether an artificial agent is aligned, and maybe we could even do things like surgically modify its goals, or directly point to things we care about in its world-model. I think that's what John is aiming for. A less ambitious goal, which I think better describes the aims of shard theory, is to understand agency well enough that we can carefully guide the formation of agents' goals during ML training runs.
    Beyond that, I think everyone involved expects that descriptive work could lead to foundational insights that change our minds about which alignment strategies are most promising. In particular, these insights might answer questions like: whether intelligent entities are inevitably agents, whether agents are inevitably consequentialists, whether corrigibility is a thing, and whether we should expect to encounter sharp left turns.
    Normative
    The normative branch shares the conceptual clarification theory of change. I think there's a reasonable argument to be made that we should expect the theoretical ideal of agency to be much easier to understand than agency-in-practice, and that understanding it might provide most of the insight. But the normative branch also has a much more ambitious theory of change, which is something like: if we understand the theoretical ideal of agency well enough, we might be able to build an aligned AGI manually 'out of toothpicks and rubber bands'. I think this hope has fallen by the wayside in recent years, as the capabilities of prosaic AI have rapidly progressed. Doing it the hard way just seems like it will take too long.
    Subproblems
    The Embedded Agency sequence identifies four rough subquests in The Hunt for Embedded AIXI. Most work in the normative branch can be thought of as attacking one or another of these problems. Many of the insights of that sequence are directly applicable to the descriptive case, but the names of the subproblems are steeped in normative language. Moreover, there are aspects of the descriptive challenge which don't seem to have normative analogues. It therefore seems worth trying to identify a seperate set of descriptive subproblems, and vaguely categorise descriptive work according to which of them it gets at. I'll suggest some subproblems here, with a view to using them ...
    8 min
  • AF - Conditioning Predictive Models: Outer alignment via careful conditioning by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Outer alignment via careful conditioning, published by Evan Hubinger on February 2, 2023 on The AI Alignment Forum.
    This is the second of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
    2. Outer alignment via careful conditioning
    Suppose we actually get a predictive model of the world that we can condition on arbitrary observations. What should we do with it?
    One thing that is often done with large language models is to ask them to act as dialogue agents (e.g. predict what comes next after an “AI:” prompt). As we’ll discuss extensively in this section, however, we think that asking a predictive model to predict itself or another AI system is highly unsafe, as the AI systems being predicted may not themselves be safe. Instead, our basic proposal is to try to predict the most useful possible human outputs.
    One canonical thing that you might want to do in that vein is “predict the results of humans doing AI safety research” (though note that this is just an example). Specifically, here’s GPT-3.5 trained with RLHF (text-davinci-003) at zero temperature (prompt is bolded):[1]
    # A full solution to the AI alignment problem
    Unfortunately, there is no single solution to the AI alignment problem. AI alignment is a complex and multifaceted problem that requires a comprehensive approach.
    The first step in solving the AI alignment problem is to define the goals of the AI system. This involves understanding the desired outcomes of the AI system and how it should interact with its environment. Once the goals are defined, the next step is to develop a framework for how the AI system should behave in order to achieve those goals. This framework should include ethical considerations, such as ensuring that the AI system does not cause harm to humans or other sentient beings.
    Unfortunately, as can be seen from GPT-3.5’s continuation, it doesn’t actually think a paper titled “A full solution to the alignment problem” will actually contain such a solution—instead, it seems to think it’s far more likely that it’ll just talk about what such a solution might entail. Which in fact seems true, at least if such a paper was written today. Fortunately, at least for current InstructGPT models, it’s not that hard to get GPT-3.5 to at least seemingly give a real attempt.[2] Here’s another trajectory from text-davinci-003 at zero temperature (prompt is again bolded):
    # ARC Technical Report: A full solution to the AI alignment problem
    Introduction
    The AI alignment problem is one of the most pressing challenges of our time. It is the challenge of ensuring that artificial intelligence (AI) systems are designed and deployed in a way that is beneficial to humanity. This is a difficult problem because AI systems are often designed to optimize for a single goal, such as maximizing profits or minimizing costs, without considering the broader implications of their actions. As AI systems become more powerful and more widely deployed, it is increasingly important to ensure that they are designed and deployed in a way that is beneficial to humanity.
    This report presents a full solution to the AI alignment problem. It begins by discussing the current state of the problem and the challenges that must be addressed in order to achieve a full solution. It then outlines a comprehensive approac...
    1 hr 11 min
  • AF - Conditioning Predictive Models: Large language models as predictors by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Large language models as predictors, published by Evan Hubinger on February 2, 2023 on The AI Alignment Forum.
    This is the first of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
    Thanks to Paul Christiano, Kyle McDonell, Laria Reynolds, Collin Burns, Rohin Shah, Ethan Perez, Nicholas Schiefer, Sam Marks, William Saunders, Evan R. Murphy, Paul Colognese, Tamera Lanham, Arun Jose, Ramana Kumar, Thomas Woodside, Abram Demski, Jared Kaplan, Beth Barnes, Danny Hernandez, Amanda Askell, Robert Krzyzanowski, and Andrei Alexandru for useful conversations, comments, and feedback.
    Abstract
    Our intention is to provide a definitive reference on what it would take to safely make use of predictive models in the absence of a solution to the Eliciting Latent Knowledge problem.
    Furthermore, we believe that large language models can be understood as such predictive models of the world, and that such a conceptualization raises significant opportunities for their safe yet powerful use via carefully conditioning them to predict desirable outputs.
    Unfortunately, such approaches also raise a variety of potentially fatal safety problems, particularly surrounding situations where predictive models predict the output of other AI systems, potentially unbeknownst to us. There are numerous potential solutions to such problems, however, primarily via carefully conditioning models to predict the things we want—e.g. humans—rather than the things we don’t—e.g. malign AIs.
    Furthermore, due to the simplicity of the prediction objective, we believe that predictive models present the easiest inner alignment problem that we are aware of.
    As a result, we think that conditioning approaches for predictive models represent the safest known way of eliciting human-level and slightly superhuman capabilities from large language models and other similar future models.
    1. Large language models as predictors
    Suppose you have a very advanced, powerful large language model (LLM) generated via self-supervised pre-training. It’s clearly capable of solving complex tasks when prompted or fine-tuned in the right way—it can write code as well as a human, produce human-level summaries, write news articles, etc.—but we don’t know what it is actually doing internally that produces those capabilities. It could be that your language model is:
    a loose collection of heuristics,[1]
    a generative model of token transitions,
    a simulator that picks from a repertoire of humans to simulate,
    a proxy-aligned agent optimizing proxies like sentence grammaticality,
    an agent minimizing its cross-entropy loss,
    an agent maximizing long-run predictive accuracy,
    a deceptive agent trying to gain power in the world,
    a general inductor,
    a predictive model of the world,
    etc.
    Later, we’ll discuss why you might expect to get one of these over the others, but for now, we’re going to focus on the possibility that your language model is well-understood as a predictive model of the world.
    In particular, our aim is to understand what it would look like to safely use predictive models to perform slightly superhuman tasks[2]—e.g. predicting counterfactual worlds to extract the outputs of long serial research processes.[3]
    We think that this basic approach has hope for two reasons. First, the prediction orthogonality thesis seems basi...
    21 min