The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - AGI will have learnt utility functions by Beren Millidge
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum.
    This post is part of the work done at Conjecture.
    Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post.
    There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function.
    Rather, the optimization process is assumed to be 'wrapped around' some core and unchanging utility function. The capabilities core of the agent is also totally modular and disjoint from the utility function such that arbitrary planners and utility functions can be composed so long as they have the right I/O interfaces. The core 'code' of an AIXI like agent is incredibly simple and, for instance, could be implemented in this Python pseudocode :
    def action_perception_loop(): while True: observation = self.sensors.get_observation() state = self.update_state(self.current_state, observation) all_action_plans = self.generate_action_plans(state) all_trajectories = self.world_model.generate_all_trajectories(all_action_plans, state) optimal_plan, optimal_utility = self.evaluate_trajectories(all_trajectories) self.execute(optimal_plan)
    There's a couple of central elements to this architecture which must be included in any AIXI-like architecture. The AGI needs some sensorimotor equipment to both sense the world and execute its action plans. It needs a Bayesian filtering component to be able to update its representation of the world state given new observations and its current state. It needs a world model that can generate sets of action plans and then generate 'rollouts' which are simulations of likely futures given an action plan. Finally, it needs a utility function that can calculate the utility of different simulated trajectories into the future and pick the best one. Let's zoom in on this component a little more and see how the evaluate_trajectories function might look inside. It might look like this:
    Essentially, the AIXI agent just takes all trajectories and ranks them according to its utility function and then picks the best one to execute. The fundamental problem with such an architecture, which is severely underappreciated, is that it implicitly assumes a utility oracle. That is, there exists some function self.utility_function() which is built into the agent from the beginning which can assign a consistent utility value to arbitrary world-states.
    While conceptually simple, my argument is that actually designing and building such a function into an agent to achieve a specific and complex goal in the external world is incredibly difficult or impossible for agents pursuing sufficiently complex goals and operating in sufficiently complex environments. This includes almost all goals humans are likely to want to program an AGI with. This means that in practice we cannot construct AIXI-like agents that optimize for arbitrary goals in the real world, and that any agent we do build must utilize some kind of learned utility model. Specifically, this is a utility (or reward) function uθ(x) where θ is some set of parameters and x is some kind of state, where the utility function is learned by some learning process (typically supervised learning) against a dataset of state, utility pairs that are provided either by the environment or by human designers.
    What this means is that, unlike a wrapper mind, the agent’s utility function can be influe...
    26 min
  • EA - Anti Entropy: Supporting ops professionals in EA by redbermejo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anti Entropy: Supporting ops professionals in EA, published by redbermejo on January 25, 2023 on The Effective Altruism Forum.
    [‘Operations’ (in the effective altruism community) refers to everything that supports and facilitates direct work, from HR to finance to office management to recruiting]
    One of the key challenges facing the EA community is the lack of much-needed operational support - which drove us to form Anti Entropy. We believe that by providing guidance, tools, and resources to equip and empower EAs for effective operations management, we can help EA organizations to survive and thrive.
    In this post, we share our thoughts on the challenge for EA operations, what the EA community can do about this, and what Anti Entropy offers.
    Why is ops support lacking?
    We set up Anti Entropy to address a general lack of operational support in the community. What causes this situation?
    First, there are relatively few experienced ops professionals in the community. Second, good resources are limited or inadequately vetted. And third, EA founders and organizational leaders don’t always take advantage of existing operational support. For example, a newly-formed organization may not be aware that they can pay a hiring agency to help them with hiring; or they may not have the budget to use one even if they are aware of them because they didn’t anticipate this expense when they applied for funding.
    This is exacerbated by a culture of frugality and self-sufficiency in EA. Grantees feel like they need to spend their funding responsibly, which is generally good and admirable, but often they are not given clear guidance on what ‘responsible’ means and what the funders would or would not consider a ‘responsible,’ reasonable use. So they err on the side of caution and inaction.
    This leads to several problems:
    Organizations fail unnecessarily
    A lack of operational support can cause promising organizations to shut down prematurely. This is often due to practical issues rather than a lack of impact or bad ideas. This is particularly concerning for new or rapidly-growing organizations, which may struggle to keep up with the demands of fast growth. A lack of ops support can lead to bottlenecks, internal cultural issues, a loss of efficiency, and an unnecessary loss of impact.
    Stress and burnout
    Even among organizations that succeed, employees are more stressed and prone to burnout. Ops people have too much responsibility and not enough support. In some cases, employees who are not specifically trained in operations may be required to take on operational tasks in addition to their normal work, leading to overwork and burnout.
    While the need for operational support in the EA community has been recognized in the past, this has sometimes led to people taking on operational roles without the necessary training or support. Luckily, this has become less common, but there is still room for improvement.
    Misinformation
    People sometimes think, ‘I don’t need to hire an ops person to deal with taxes/write my grievance policy/deal with the bureaucracy of setting up a nonprofit - I’ll just look up how to do those things myself’. Unfortunately, this can lead to people relying on inaccurate or inappropriate information, potentially leading to legal issues.
    For example, a UK organization can’t use a US vacation policy since the laws are different. Similarly, some topics related to organizations’ financial status are very poorly understood - for example, what does it mean to ‘set up an entity’, ‘get fiscally sponsored’, or hire people ‘as an independent contractor’? When organizational leaders don’t understand these things, it can lead to them unwittingly breaking the law.
    What can I do?
    If you’re currently working in operations for an EA organization
    Check the Anti Entropy website for resources to help ...
    8 min
  • LW - My Model Of EA Burnout by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Model Of EA Burnout, published by LoganStrohl on January 25, 2023 on LessWrong.
    (Probably somebody else has said most of this. But I personally haven't read it, and felt like writing it down myself, so here we go.)
    I think that EA burnout usually results from prolonged dedication to satisfying the values you think you should have, while neglecting the values you actually have.
    Setting aside for the moment what “values” are and what it means to “actually” have one, suppose that I actually value these things (among others):
    True Values
    Abundance
    Power
    Novelty
    Social Harmony
    Beauty
    Growth
    Comfort
    The Wellbeing Of Others
    Excitement
    Personal Longevity
    Accuracy
    One day I learn about “global catastrophic risk”: Perhaps we’ll all die in a nuclear war, or an AI apocalypse, or a bioengineered global pandemic, and perhaps one of these things will happen quite soon.
    I recognize that GCR is a direct threat to The Wellbeing Of Others and to Personal Longevity, and as I do, I get scared. I get scared in a way I have never been scared before, because I’ve never before taken seriously the possibility that everyone might die, leaving nobody to continue the species or even to remember that we ever existed—and because this new perspective on the future of humanity has caused my own personal mortality to hit me harder than the lingering perspective of my Christian upbringing ever allowed. For the first time in my life, I’m really aware that I, and everyone I will ever care about, may die.
    My fear has me very focused on just two of my values: The Wellbeing Of Others and Personal Longevity. But as I read, think, and process, I realize that pretty much regardless of what my other values might be, they cannot possibly be satisfied if the entire species—or the planet, or the lightcone—is destroyed.
    [This is, of course, a version of EA that’s especially focused on the far future; but I think it’s common for a very similar thing to happen when someone transitions from “soup kitchens” to “global poverty and animal welfare”. There’s an exponential increase in stakes, accompanied by a corresponding increase in the fear of lost value.]
    So I reason that a new life strategy is called for.
    Over time, under the influence of my “Accuracy” value as well as my “Social Harmony” value (since I’m now surrounded by people who are thinking about this stuff), I come to believe that I should value the following:
    Should Values
    Impact
    Calibration
    Openness
    Collaboration
    Empiricism
    The Wellbeing Of Others
    Personal Longevity
    (The values on this new list with an asterisk beside them have a correlate on the original list (impactpower, collaborationsocial harmony, empiricismaccuracy), but these new values are routed through The New Strategy, and are not necessarily plugged into their correlates from the first list.)
    Over a couple of years, I change my career, my friend group, and my hobbies to reflect my new values. I spend as little time as possible on Things That Don’t Matter, because now I care about Impact, and designing computer games has very little Impact since it takes a lot of time and definitely doesn’t save the world (even though it’s pretty good on novelty, beauty, growth, and excitement).
    Ok, so let’s talk now about what “values” are.
    I think that in humans at least, values are drives to action. They are things that motivate a person to choose one possible action over another. If I value loyalty over honesty, I’ll readily lie to help my friend save face; if I value both about equally, I may be a little paralyzed in some situations while I consult the overall balance of my whole value system and try to figure out what to do. When I go for a hike with my field kit of watercolor paints, I tend to feel really good about that decision as I make it, as I hike and paint, and also as I look back on the experienc...
    8 min
  • LW - My Model Of EA Burnout by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Model Of EA Burnout, published by LoganStrohl on January 25, 2023 on LessWrong.
    (Probably somebody else has said most of this. But I personally haven't read it, and felt like writing it down myself, so here we go.)
    I think that EA burnout usually results from prolonged dedication to satisfying the values you think you should have, while neglecting the values you actually have.
    Setting aside for the moment what “values” are and what it means to “actually” have one, suppose that I actually value these things (among others):
    True Values
    Abundance
    Power
    Novelty
    Social Harmony
    Beauty
    Growth
    Comfort
    The Wellbeing Of Others
    Excitement
    Personal Longevity
    Accuracy
    One day I learn about “global catastrophic risk”: Perhaps we’ll all die in a nuclear war, or an AI apocalypse, or a bioengineered global pandemic, and perhaps one of these things will happen quite soon.
    I recognize that GCR is a direct threat to The Wellbeing Of Others and to Personal Longevity, and as I do, I get scared. I get scared in a way I have never been scared before, because I’ve never before taken seriously the possibility that everyone might die, leaving nobody to continue the species or even to remember that we ever existed—and because this new perspective on the future of humanity has caused my own personal mortality to hit me harder than the lingering perspective of my Christian upbringing ever allowed. For the first time in my life, I’m really aware that I, and everyone I will ever care about, may die.
    My fear has me very focused on just two of my values: The Wellbeing Of Others and Personal Longevity. But as I read, think, and process, I realize that pretty much regardless of what my other values might be, they cannot possibly be satisfied if the entire species—or the planet, or the lightcone—is destroyed.
    [This is, of course, a version of EA that’s especially focused on the far future; but I think it’s common for a very similar thing to happen when someone transitions from “soup kitchens” to “global poverty and animal welfare”. There’s an exponential increase in stakes, accompanied by a corresponding increase in the fear of lost value.]
    So I reason that a new life strategy is called for.
    Over time, under the influence of my “Accuracy” value as well as my “Social Harmony” value (since I’m now surrounded by people who are thinking about this stuff), I come to believe that I should value the following:
    Should Values
    Impact
    Calibration
    Openness
    Collaboration
    Empiricism
    The Wellbeing Of Others
    Personal Longevity
    (The values on this new list with an asterisk beside them have a correlate on the original list (impactpower, collaborationsocial harmony, empiricismaccuracy), but these new values are routed through The New Strategy, and are not necessarily plugged into their correlates from the first list.)
    Over a couple of years, I change my career, my friend group, and my hobbies to reflect my new values. I spend as little time as possible on Things That Don’t Matter, because now I care about Impact, and designing computer games has very little Impact since it takes a lot of time and definitely doesn’t save the world (even though it’s pretty good on novelty, beauty, growth, and excitement).
    Ok, so let’s talk now about what “values” are.
    I think that in humans at least, values are drives to action. They are things that motivate a person to choose one possible action over another. If I value loyalty over honesty, I’ll readily lie to help my friend save face; if I value both about equally, I may be a little paralyzed in some situations while I consult the overall balance of my whole value system and try to figure out what to do. When I go for a hike with my field kit of watercolor paints, I tend to feel really good about that decision as I make it, as I hike and paint, and also as I look back on the experienc...
    8 min
  • LW - How-to Transformer Mechanistic Interpretability—in 50 lines of code or less! by StefanHex
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How-to Transformer Mechanistic Interpretability—in 50 lines of code or less!, published by StefanHex on January 24, 2023 on LessWrong.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
    What if I told you that in just one weekend you can get up to speed doing practical Mechanistic Interpretability research on Transformers? Surprised? Then this is your tutorial!
    I'll give you a view to how I research Transformer circuits in practice, show you the tools you need, and explain my thought process along the way. I focus on the practical side to get started with interventions; for more background see point 2 below.
    Prerequisites:
    Understanding the Transformer architecture: Know what the residual stream is, how attention layers and MLPs work, and how logits & predictions work. For future sections familiarity with multi-head attention is useful. Here’s a link to Neel’s glossary which provides excellent explanations for most terms I might use!If you're not familiar with Transformers you can check out Step 2 (6) on Neel's guide or any of the other explanations online, I recommend Jay Alammar's The Illustrated Transformer and/or Milan Straka's lecture series.
    Some overview of Mechanistic Interpretability is helpful: See e.g. any of Neel's talks, or look at the results in the IOI paper / walkthrough.
    Basic Python: Familiarity with arrays (as in NumPy or PyTorch, for indices) is useful; but explicitly no PyTorch knowledge required!
    No hardware required, free Google Colab account works fine for this. Here's a notebook with all the code from this tutorial! PS: Here's a little web page where you can run some of these methods online! No trivial inconveniences!
    Step 0: Setup
    Open a notebook (e.g. Colab) and install Neel Nanda’s TransformerLens (formerly known as EasyTransformer).
    Step 1: Getting a model to play with
    That’s it, now you’ve got a GPT2 model to play with! TransformerLens supports most relevant open source transformers. Here’s how to run the language model
    Let’s have a look at the internal activations: TransformerLens can give you a dictionary with almost all internal activations you ever care about (referred to as “cache”):
    Here you will find things like the attention pattern blocks.0.attn.hook_pattern, the residual stream before and after each layer blocks.1.hook_resid_pre, and more!
    You can also access all the weights & parameters of the model in model.named_parameters(). Here you will find weight matrices & biases of every MLP and Attention layer, as well as the embedding & unembedding matrices. I won’t focus on these in this guide but they’re great to look at! (Exercise: What can the unembedding biases unembed.b_U tell you about common tokens?)
    Step 2: Let's start analyzing a behavior!
    Let’s go and find some induction heads! I’ll make up an example: Her name was Alex Hart. When Alex, with likely completion Hart. TransformerLens has a little tool to plot a tokenized prompt, model predictions, and associated logits:
    I find it is useful to spend a few minutes thinking about which information is needed to solve the task: The model needs to
    Realize the last token, Alex, is a repetition of a previous occurrence
    The model needs to copy the last name from after the previous Alex occurrence to the last token as prediction
    Method 1: Residual stream patching
    The number 1 thing I try when I want to reverse engineer a new behavior is to find where in the network the information is “traveling”.
    In transformers, the model keeps track of all information in the residual stream. Attention heads & MLPs read from the residual stream, perform some computation or information moving, and write their outputs back into the residual stream. I think of this stream as having a couple of “lanes” corresponding to each token position. Over the course of the model...
    25 min
  • LW - Some Thoughts on AI Art by abramdemski
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Thoughts on AI Art, published by abramdemski on January 25, 2023 on LessWrong.
    I was recently talking with a Daniel Kokotajlo about AI art. It turned out that he and I initially disagreed about ethical questions, but by the end of the conversation, I had somewhat won him over to my position.
    I have the vague impression that a lot of people (on the technology side) haven't thought through these issues so much, or (like me) have only recently thought these issues through (as a result of artists making a lot of noise about it!).
    So I thought I would write a post. Maybe it will be persuasive to some readers.
    Is this the most important conversation to be having about AI?
    No. Copyright-adjacent issues with AI art are less important than AI-induced unemployment, which is in turn less important than the big questions about the fate of the human race.
    However, it's possible that copyright-adjacent issues around intellectual property and AI will be one of the first major issues thrusting AI into the political sphere, in which case this discussion may help to shape public policy around AI for years to come.
    The basic issues.
    Large language models such as GPT, and AI image generators such as DALL-E, Imagen, Stable Diffusion, etc etc are (very often) trained on copyrighted works without the permission of the copyright holder. This hasn't proven to be a legal problem, yet, but "legal" doesn't mean "ethical".
    When models like GPT and DALL-E started coming out, I recall having the thought: oh, it's nice how these models don't really need to worry about copyright, because (I thought) deep learning turns out to generalize quite well, which means deep-learning-based systems aren't liable to regurgitate copyrighted material.
    This turns out to be simply false; these systems are in fact quite liable to reproduce, or very nearly reproduce, copyrighted material when prompted in the right way.
    Whether or not copyrighted material is precisely reproduced, or nearly reproduced, or not reproduced at all, there is, in any case, an argument to be made that these AI systems (if/when they charge for use) are turning a profit based on copyrighted material in an illegitimate way.
    After all: the purpose of copyright law is, to a very large extent, to preserve the livelihood of intellectual property creators, who would otherwise have limited ability to profit from their own works due to the ease of reproducing it once made. Modern AI systems are threatening this, whether or not they technically violate copyright.
    But I want to firmly distinguish between a few different issues:
    AI systems training on copyrighted data without the consent of the copyright holder. This is the main issue I will discuss.
    AI systems being capable of reproducing copyrighted works exactly or almost exactly. This is a consequence of the first bullet point, plus properties of modern ML systems, plus the absence of safeguards specifically preventing this from happening.
    AI systems imitating work in a more general sort of way, such as copying the style of specific artists who never consented to their work being used as training data. This is one of the main reasons to think that training on copyrighted work (without permission) has occurred, in cases where there isn't much public information about what data was used to train an AI. It is also one of the main reasons (I have seen) that artists want these systems to stop training on copyrighted works.
    AI putting artists and writers out of work. This is not the main topic of the post, but is an obvious underlying reason why people might be upset.
    Some initial arguments.
    It's not illegal.
    Artists who take a position against AI art will sometimes describe the situation as follows: AI programmers steal our art, and use it to train AIs, which can then steal our artistic style, and the...
    13 min
  • LW - Some Thoughts on AI Art by abramdemski
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Thoughts on AI Art, published by abramdemski on January 25, 2023 on LessWrong.
    I was recently talking with a Daniel Kokotajlo about AI art. It turned out that he and I initially disagreed about ethical questions, but by the end of the conversation, I had somewhat won him over to my position.
    I have the vague impression that a lot of people (on the technology side) haven't thought through these issues so much, or (like me) have only recently thought these issues through (as a result of artists making a lot of noise about it!).
    So I thought I would write a post. Maybe it will be persuasive to some readers.
    Is this the most important conversation to be having about AI?
    No. Copyright-adjacent issues with AI art are less important than AI-induced unemployment, which is in turn less important than the big questions about the fate of the human race.
    However, it's possible that copyright-adjacent issues around intellectual property and AI will be one of the first major issues thrusting AI into the political sphere, in which case this discussion may help to shape public policy around AI for years to come.
    The basic issues.
    Large language models such as GPT, and AI image generators such as DALL-E, Imagen, Stable Diffusion, etc etc are (very often) trained on copyrighted works without the permission of the copyright holder. This hasn't proven to be a legal problem, yet, but "legal" doesn't mean "ethical".
    When models like GPT and DALL-E started coming out, I recall having the thought: oh, it's nice how these models don't really need to worry about copyright, because (I thought) deep learning turns out to generalize quite well, which means deep-learning-based systems aren't liable to regurgitate copyrighted material.
    This turns out to be simply false; these systems are in fact quite liable to reproduce, or very nearly reproduce, copyrighted material when prompted in the right way.
    Whether or not copyrighted material is precisely reproduced, or nearly reproduced, or not reproduced at all, there is, in any case, an argument to be made that these AI systems (if/when they charge for use) are turning a profit based on copyrighted material in an illegitimate way.
    After all: the purpose of copyright law is, to a very large extent, to preserve the livelihood of intellectual property creators, who would otherwise have limited ability to profit from their own works due to the ease of reproducing it once made. Modern AI systems are threatening this, whether or not they technically violate copyright.
    But I want to firmly distinguish between a few different issues:
    AI systems training on copyrighted data without the consent of the copyright holder. This is the main issue I will discuss.
    AI systems being capable of reproducing copyrighted works exactly or almost exactly. This is a consequence of the first bullet point, plus properties of modern ML systems, plus the absence of safeguards specifically preventing this from happening.
    AI systems imitating work in a more general sort of way, such as copying the style of specific artists who never consented to their work being used as training data. This is one of the main reasons to think that training on copyrighted work (without permission) has occurred, in cases where there isn't much public information about what data was used to train an AI. It is also one of the main reasons (I have seen) that artists want these systems to stop training on copyrighted works.
    AI putting artists and writers out of work. This is not the main topic of the post, but is an obvious underlying reason why people might be upset.
    Some initial arguments.
    It's not illegal.
    Artists who take a position against AI art will sometimes describe the situation as follows: AI programmers steal our art, and use it to train AIs, which can then steal our artistic style, and the...
    13 min
  • LW - Thoughts on the impact of RLHF research by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the impact of RLHF research, published by paulfchristiano on January 25, 2023 on LessWrong.
    In this post I’m going to describe my basic justification for working on RLHF in 2017-2020, which I still stand behind. I’ll discuss various arguments that RLHF research had an overall negative impact and explain why I don’t find them persuasive.
    I'll also clarify that I don't think research on RLHF is automatically net positive; alignment research should address real alignment problems, and we should reject a vague association between "RLHF progress" and "alignment progress."
    Background on my involvement in RLHF work
    Here are some background views about alignment I held in 2015 and still hold today. I expect disagreements about RLHF will come down to disagreements about this background:
    The simplest plausible strategies for alignment involve humans (maybe with the assistance of AI systems) evaluating a model’s actions based on how much we expect to like their consequences, and then training the models to produce highly-evaluated actions. (This is in contrast with, for example, trying to formally specify the human utility function, or notions of corrigibility / low-impact / etc, in some way.)
    Simple versions of this approach are expected to run into difficulties, and potentially to be totally unworkable, because:
    Evaluating consequences is hard.
    A treacherous turn can cause trouble too quickly to detect or correct even if you are able to do so, and it’s challenging to evaluate treacherous turn probability at training time.
    It’s very unclear if those issues are fatal before or after AI systems are powerful enough to completely transform human society (and in particular the state of AI alignment). Even if they are fatal, many of the approaches to resolving them still have the same basic structure of learning from expensive evaluations of actions.
    In order to overcome the fundamental difficulties with RLHF, I have long been interested in techniques like iterated amplification and adversarial training. However, prior to 2017 most researchers I talked to in ML (and many researchers in alignment) thought that the basic strategy of training AI with expensive human evaluations was impractical for more boring reasons and so weren't interested in these difficulties. On top of that, we obviously weren’t able to actually implement anything more fancy than RLHF since all of these methods involve learning from expensive feedback. I worked on RLHF work to try to facilitate and motivate work on fixes.
    The history of my involvement:
    My first post on this topic was in 2015.
    When I started full-time at OpenAI in 2017 it seemed to me like it would be an impactful project; I considered doing a version with synthetic human feedback (showing that we could learn from a practical amount of algorithmically-defined feedback) but my manager Dario Amodei convinced me it would be more compelling to immediately go for human feedback. The initial project was surprisingly successful and published here.
    I then intended to implement a version with language models aiming to be complete in the first half of 2018 (aiming to build an initial amplification prototype with LMs around end of 2018; both of these timelines were about 2.5x too optimistic). This seemed like the most important domain to study RLHF and alignment more broadly. In mid-2017 Alec Radford helped me do a prototype with LSTM language models (prior to the release of transformers); the prototype didn’t look promising enough to scale up.
    In mid-2017 Geoffrey Irving joined OpenAI and was excited about starting with RLHF and then going beyond it using debate; he also thought language models were the most important domain to study and had more conviction about that. In 2018 he started a larger team working on fine-tuning on language models, w...
    15 min
  • LW - Thoughts on the impact of RLHF research by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the impact of RLHF research, published by paulfchristiano on January 25, 2023 on LessWrong.
    In this post I’m going to describe my basic justification for working on RLHF in 2017-2020, which I still stand behind. I’ll discuss various arguments that RLHF research had an overall negative impact and explain why I don’t find them persuasive.
    I'll also clarify that I don't think research on RLHF is automatically net positive; alignment research should address real alignment problems, and we should reject a vague association between "RLHF progress" and "alignment progress."
    Background on my involvement in RLHF work
    Here are some background views about alignment I held in 2015 and still hold today. I expect disagreements about RLHF will come down to disagreements about this background:
    The simplest plausible strategies for alignment involve humans (maybe with the assistance of AI systems) evaluating a model’s actions based on how much we expect to like their consequences, and then training the models to produce highly-evaluated actions. (This is in contrast with, for example, trying to formally specify the human utility function, or notions of corrigibility / low-impact / etc, in some way.)
    Simple versions of this approach are expected to run into difficulties, and potentially to be totally unworkable, because:
    Evaluating consequences is hard.
    A treacherous turn can cause trouble too quickly to detect or correct even if you are able to do so, and it’s challenging to evaluate treacherous turn probability at training time.
    It’s very unclear if those issues are fatal before or after AI systems are powerful enough to completely transform human society (and in particular the state of AI alignment). Even if they are fatal, many of the approaches to resolving them still have the same basic structure of learning from expensive evaluations of actions.
    In order to overcome the fundamental difficulties with RLHF, I have long been interested in techniques like iterated amplification and adversarial training. However, prior to 2017 most researchers I talked to in ML (and many researchers in alignment) thought that the basic strategy of training AI with expensive human evaluations was impractical for more boring reasons and so weren't interested in these difficulties. On top of that, we obviously weren’t able to actually implement anything more fancy than RLHF since all of these methods involve learning from expensive feedback. I worked on RLHF work to try to facilitate and motivate work on fixes.
    The history of my involvement:
    My first post on this topic was in 2015.
    When I started full-time at OpenAI in 2017 it seemed to me like it would be an impactful project; I considered doing a version with synthetic human feedback (showing that we could learn from a practical amount of algorithmically-defined feedback) but my manager Dario Amodei convinced me it would be more compelling to immediately go for human feedback. The initial project was surprisingly successful and published here.
    I then intended to implement a version with language models aiming to be complete in the first half of 2018 (aiming to build an initial amplification prototype with LMs around end of 2018; both of these timelines were about 2.5x too optimistic). This seemed like the most important domain to study RLHF and alignment more broadly. In mid-2017 Alec Radford helped me do a prototype with LSTM language models (prior to the release of transformers); the prototype didn’t look promising enough to scale up.
    In mid-2017 Geoffrey Irving joined OpenAI and was excited about starting with RLHF and then going beyond it using debate; he also thought language models were the most important domain to study and had more conviction about that. In 2018 he started a larger team working on fine-tuning on language models, w...
    15 min
  • AF - Thoughts on the impact of RLHF research by Paul Christiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the impact of RLHF research, published by Paul Christiano on January 25, 2023 on The AI Alignment Forum.
    In this post I’m going to describe my basic justification for working on RLHF in 2017-2020, which I still stand behind. I’ll discuss various arguments that RLHF research had an overall negative impact and explain why I don’t find them persuasive.
    I'll also clarify that I don't think research on RLHF is automatically net positive; alignment research should address real alignment problems, and we should reject a vague association between "RLHF progress" and "alignment progress."
    Background on my involvement in RLHF work
    Here are some background views about alignment I held in 2015 and still hold today. I expect disagreements about RLHF will come down to disagreements about this background:
    The simplest plausible strategies for alignment involve humans (maybe with the assistance of AI systems) evaluating a model’s actions based on how much we expect to like their consequences, and then training the models to produce highly-evaluated actions. (This is in contrast with, for example, trying to formally specify the human utility function, or notions of corrigibility / low-impact / etc, in some way.)
    Simple versions of this approach are expected to run into difficulties, and potentially to be totally unworkable, because:
    Evaluating consequences is hard.
    A treacherous turn can cause trouble too quickly to detect or correct even if you are able to do so, and it’s challenging to evaluate treacherous turn probability at training time.
    It’s very unclear if those issues are fatal before or after AI systems are powerful enough to completely transform human society (and in particular the state of AI alignment). Even if they are fatal, many of the approaches to resolving them still have the same basic structure of learning from expensive evaluations of actions.
    In order to overcome the fundamental difficulties with RLHF, I have long been interested in techniques like iterated amplification and adversarial training. However, prior to 2017 most researchers I talked to in ML (and many researchers in alignment) thought that the basic strategy of training AI with expensive human evaluations was impractical for more boring reasons and so weren't interested in these difficulties. On top of that, we obviously weren’t able to actually implement anything more fancy than RLHF since all of these methods involve learning from expensive feedback. I worked on RLHF work to try to facilitate and motivate work on fixes.
    The history of my involvement:
    My first post on this topic was in 2015.
    When I started full-time at OpenAI in 2017 it seemed to me like it would be an impactful project; I considered doing a version with synthetic human feedback (showing that we could learn from a practical amount of algorithmically-defined feedback) but my manager Dario Amodei convinced me it would be more compelling to immediately go for human feedback. The initial project was surprisingly successful and published here.
    I then intended to implement a version with language models aiming to be complete in the first half of 2018 (aiming to build an initial amplification prototype with LMs around end of 2018; both of these timelines were about 2.5x too optimistic). This seemed like the most important domain to study RLHF and alignment more broadly. In mid-2017 Alec Radford helped me do a prototype with LSTM language models (prior to the release of transformers); the prototype didn’t look promising enough to scale up.
    In mid-2017 Geoffrey Irving joined OpenAI and was excited about starting with RLHF and then going beyond it using debate; he also thought language models were the most important domain to study and had more conviction about that. In 2018 he started a larger team working on fine-tuning on langu...
    15 min