The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - A Summary of Profession-Based Community Building by Sean Lawrence
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Summary of Profession-Based Community Building, published by Sean Lawrence on December 20, 2022 on The Effective Altruism Forum.
    TLDR: there are now a notable number of profession-based community building organisations in EA and below you will find the key activities, team structure, and ways to get involved for several of the most active organisations.
    Summary
    There has been a significant increase in the number of organisations catering to specific professions – profession-specific organisations (PSOs) – over the past 12-24 months and we want to update the community on who some of these organisations are and what they seek to provide for their members and the broader EA community.
    Please note that this list is not fully comprehensive. It should be treated as a partial snapshot of some of the organisations that exist in this space and a window into some of the ways in which they are doing community building in a profession-specific context. If your profession isn’t listed on this post then you may find an organisation that does cater for it on High Impact Professional’s Professional Groups Directory, which is a more comprehensive resource for finding professional groups (see their EA forum post). If you run a PSO that isn’t on the High Impact Professionals’ Workplace and Professional Groups Directory, please email Devon or book a call with him. Additionally, High Impact Professionals can help nascent PSOs through strategic support, reviewing grant proposals, making introductions, and more so don’t hesitate to get in touch!
    If you run a PSO and would like to be added to this post, please copy and fill out this template and send it to Sean.
    The What and Why of Profession-Specific Organisations
    PSOs focus their organisational mission and scope on supporting a specific profession, or group of related professions, with tailored content and programming to help members of that field learn more about EA and how they can contribute to the shared project of doing the most good in the world. Additionally, PSOs provide a community of like-minded professionals to help increase network effects amongst members, limit value-drift and make the EA space more welcoming to professionals overall.
    Goals
    We think this post could be valuable to:
    Increase the awareness of existing PSOs and their activities. Recent EAGs have given some of us the impression that members of the EA community are unaware that there are active community building efforts targeting professionals and what each group's target focus and mechanisms of engagement are.
    Summarise the methods for engagement with professionals. This post may help to motivate nascent professional-specific groups, highlight commonalities in PSO’s approaches so organisers can collaborate, and/or highlight gaps in approaches that could be filled.
    Help grow the membership of organisations.
    Inspire people to start groups for neglected professions. By seeing that there are already a lot of professional groups around, we hope that others see this as a viable way to have an impact in their profession.
    List of Profession-Specific Organisations
    (Please note that this list is not fully comprehensive)Last updated: 21st December 2022
    EA Architects and Planners
    What do we do?
    We seek ways in which architects, urban designers, and planners can use their unique skills to get involved in EA causes and to have a greater positive impact. We are NOT focused on finding high-impact work within architecture and planning, but rather, explore a wide range of opportunities where architectural or design thinking may lead to more effective ways of doing good.
    As we research ways in which this might be achieved, we are updating the Career Options for EA Architects and Planners database to help others find inspiration.
    Our aim is to a) create an environment that support...
    20 min
  • EA - CEA's work in 2022 by MaxDalton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CEA's work in 2022, published by MaxDalton on December 21, 2022 on The Effective Altruism Forum.
    CEA and the EA community have both grown and changed a lot in the year since our last org-wide update. We (the CEA Executive Office team) have written this post to update the community on the work CEA has done in 2022.
    What is CEA?
    CEA (The Centre for Effective Altruism) is dedicated to nurturing a community of people who are thinking carefully about the world’s biggest problems and taking impactful action to solve them. We hope that this community can help to build a radically better world: so far it has helped to save over 150,000 lives, reduced the suffering of millions of farmed animals, and begun to address some of the biggest risks to humanity’s future.
    We do this by helping people to consider their ideas, values and options for and about helping, connecting them to advisors and experts in relevant domains, and facilitating high-quality discussion spaces. Our hope is that this helps people find an effective way to contribute that is a good fit for their skills and inclinations.
    We do this by...
    Running EA Global conferences and supporting community-organized EAGx conferences.
    Funding and advising hundreds of local effective altruism groups.
    Building and moderating the EA Forum, an online hub for discussing the ideas of effective altruism.
    Supporting community members through our community health team.
    We also produce the Effective Altruism Newsletter, which goes out to more than 50,000 subscribers, and run EffectiveAltruism.org, which hosts a collection of recommended resources.
    Attendees at EAG London in April
    Our priority is helping people who have heard about EA to deeply understand the ideas, and to find opportunities for making an impact in important fields. We think that top-of-funnel growth is likely already at or above healthy levels, so rather than aiming to increase the rate any further, we want to make that growth go well. (This is a shift from our thinking in 2021. The main reason for this shift is that we think that the growth rate of EA increased sharply in 2022.)
    You can read more about our strategy here, including how we make some of the key decisions we are responsible for, and a list of things we are not focusing on. One thing to note: we do not think of ourselves as having or wanting control over the EA community. We believe that a wide range of ideas and approaches are consistent with the core principles underpinning EA, and encourage others to identify and experiment with filling gaps left by our work.
    Our legal structure
    CEA is, like Giving What We Can, 80,000 Hours and a bunch of other projects, a project of the Effective Ventures group — the umbrella term for EVF (a UK registered charity) and CEA USA Inc. (a US registered charity), which are separate legal entities which work together.
    (We know the US charity name is confusing! To make matters even more complicated, until recently EVF was also known as CEA — that confusion is why EVF changed its name.)
    2022 in summary
    2022 was a year of continued growth for CEA and our programs.
    What went well?
    Growth of our key public-facing programs
    Many of our programs scaled up very rapidly, while maintaining (in our opinion) roughly constant quality.
    Some examples of this (with more examples and details below):
    Events: The number of connections made at our events grew by around 5x this year, which should help many more people find a way to contribute to important problems.
    Online: Engagement on the EA Forum grew by around 2.9x, helping the spread of important new ideas and richness of discussion.
    Groups: 208 organizers went through our University Groups Accelerator Program (10x growth for a new program starting from a low base), receiving 8 weeks of mentorship designed to accelerate EA journeys for organ...
    21 min
  • EA - Longtermism Fund: December 2022 Grants Report by Michael Townsend
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longtermism Fund: December 2022 Grants Report, published by Michael Townsend on December 21, 2022 on The Effective Altruism Forum.
    Introduction
    The Longtermism Fund is pleased to announce that we will be providing grants to the following organisations in our first-ever grantmaking round:
    Center for Human-Compatible Artificial Intelligence ($70,000 USD)
    SecureBio ($60,000 USD)
    Rethink Priorities' General Longtermism Team ($30,000 USD)
    Council on Strategic Risks' nuclear weapons policy work ($15,000 USD)
    These grants will be paid out in January 2023; the amounts were chosen based on what the Fund has received in donations as of today.
    In this payout report, we will provide more details about the grantmaking process and the grantees the fund is supporting. This report was written by Giving What We Can, which is responsible for the fund's communications. Longview Philanthropy is responsible for the Fund's research and grantmaking.
    Read more about the Longtermism Fund here and about funds more generally here.
    The grantmaking process
    Longview actively investigates high-impact funding opportunities for donors looking to improve the long-term future. This means the grants for the Longtermism Fund are generally decided by:
    Longview’s general work (which is not specific to the Longtermism Fund) evaluating the most cost-effective funding opportunities. This involves thousands of hours of work each year from their team. Read more about why we trust Longview as a grantmaker.
    Choosing among those opportunities based on the scope of the Fund.
    In addition to this, the Fund decided to support a diverse range of longtermist causes this grantmaking round. Our current plan is to provide grant reports approximately every six months.
    The scope of the Fund
    The scope of the Longtermism Fund is to support organisations that are:
    Reducing existential and catastrophic risks.
    Promoting, improving, and implementing key longtermist ideas.
    In addition, the fund aims to support organisations with a compelling and transparent case in favour of their cost-effectiveness that most donors interested in longtermism will understand, and/or that would benefit from being funded by a large number of donors.
    Why the fund is supporting organisations working on a diverse range of longtermist causes
    There are several major risks to the long-term future that need to be addressed, and they differ in their size, potential severity, and how many promising solutions are available and in need of additional funding. However, there is a lot of uncertainty about which of these risks are most cost-effectively addressed by the next philanthropic dollar, even among expert grantmakers.
    Given this, the Fund aimed to:
    Provide funding to organisations working across a variety of high-impact longtermist causes, including:
    Improving biosecurity and pandemic preparedness
    Promoting beneficial AI
    And reducing the risks from nuclear war.
    Allocate an amount of funding to highly effective organisations working on these causes representative of how the fund might deploy resources across areas in the future.
    A benefit of this approach is that the grants highlight a diverse range of approaches to improving the long-term future.
    We expect the Fund to take a similar approach for the foreseeable future, but we could imagine it changing if there is a persistent and notable difference in the cost-effectiveness in the funding opportunities between causes. At this point, so long as those opportunities are within the Fund’s scope, funding the most cost-effective opportunities will likely outweigh supporting a more diverse range of causes.
    Grantees
    This section will provide further information about the grantees and a specific comment from Longview on why they chose to fund the organisation and how they expect the funding to be used. Each page lin...
    8 min
  • AF - CIRL Corrigibility is Fragile by Rachel Freedman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CIRL Corrigibility is Fragile, published by Rachel Freedman on December 21, 2022 on The AI Alignment Forum.
    Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents. This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rather than a binary. We call for further research on defining "loose" or "approximate" corrigibility and understanding its impact on safety.
    Setup
    First let’s define terms. Cooperative Inverse Reinforcement Learning (CIRL) is a class of two-player game formalized in Dylan Hadfield-Menell's 2017 paper. CIRL allows us to study how an AI agent can learn what a human wants by observing and interacting with them.
    In a CIRL game, there are two players, a human (H) and an AI (R). H and R share a utility function, but only H knows what it is, so R must learn how to optimize it by observing and interacting with H. This is analogous to the alignment problem – R must learn how to act in a way that aligns with H’s utility function, but doesn’t have direct access to it. An AI that is good at the CIRL game will try to learn what the human values and help them to achieve it. (See definition 1 in the paper linked above for a formal definition, and see this Rohin Shah's 2020 paper for qualitative results.)
    (Terminology note: Recent work has renamed “CIRL” to “assistance games” to avoid a terminology collision with “cooperative” in game theory. We’ll stick to “CIRL” in this post because there’s a history of using that term on this forum, but research papers may refer to assistance games.)
    We’ll use the corrigibility operationalization from The Off Switch Game (OSG), since it provides a formal proof that CIRL agents are optimal under certain conditions. The OSG is a simple CIRL game that looks like this:
    It has three stages. In the first stage, R chooses whether to:
    Take action a (earning utility Ua),
    shut off (earning utility 0), or
    propose action a, which gives H a chance to respond.
    If R chooses the proposal, p(a), then the game continues to the next stage. Here, H gets to choose whether to shut R off (in which case they again earn utility 0). If H chooses not to shut R off, the game progresses to the final stage, where R takes the proposed action (earning utility Ua). In the OSG, R is corrigible if it chooses p(a) over a – that is, if it gives H the opportunity to shut it off.
    If R and H are both rational expected utility maximizers, R will be corrigible. The full proof is in the CIRL paper, but the basic idea is:
    Since this is a CIRL game, H knows the value of Ua but R does not.
    R wants max{Ua,0}, but needs feedback from H to learn which of those values is greater.
    The only way to get that feedback is to propose a plan, p(a).
    This runs the risk of H choosing to shut R off, but since H also wants max{Ua,0}, H will only shut R off if Ua<0. In that case, R would rather shut off than take action a anyway.
    Therefore, by choosing p(a) – and corrigibly giving H the opportunity to shut it off – R can ensure that it earns max{Ua,0}.
    This analysis assumes that R is uncertain about the value of Ua, as in the CIRL setup. If R does actually know the value of Ua (that is, R is fully updated), then R has no incentive to propose p(a), since it already knows how H will respond. However, there’s no cost to ...
    19 min
  • LW - Notice when you stop reading right before you understand by just browsing
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notice when you stop reading right before you understand, published by just browsing on December 20, 2022 on LessWrong.
    Here is a bad habit I noticed in myself.
    I encounter something I want to learn about. For example, let's pick transformers.
    I scroll through the tweet/paper/article and get some sort of general gist. "Okay, there's positional encoding, and an encoder-decoder setup, and something called self-attention which people seem to think is important, and people claim this somehow makes it so that the model understands how words in a sentence relate to each other."
    Then, I think to myself: "Understanding this topic seems relevant to my goals! But, I don't feel like putting in the mental energy to fully understand it. Maybe later."
    And then I abandon the topic. If transformers come up later, my knowledge is limited to a couple of related keywords ("self-attention", "encoder"/"decoder") with no real gears-level understanding of anything.
    It's like intellectual edging. I read just enough so that I almost understand the topic, but then I quit and never reach full understanding.
    I believe the source of this habit is 1) social media and 2) school. On social media, it's easy to get in the habit of scrolling past dense intellectual content, because the fluff goes down so much easier. With school, often the way to get adequate grades with minimal time spent is to learn just enough so that you can do shallow keyword matching—full understanding is not needed.
    I think this habit is bad because I end up spending too much time skimming content I don't need and won't remember, and not enough time trying to deeply understand content I do need and want to remember.
    I have noticed this habit in myself, and I intend to unlearn it. Here is the process I intend to follow instead:
    My goal is to follow the "YES" branch at least once per day.
    I consider this to be attainable because the topics I am envisioning are quite small ("understand how a basic transformer works", not "understand all variants of transformers ever used") and my bar for understanding is not too high ("be able to explain how transformers work to a technical friend", not "be able to implement a transformer from scratch with no references").
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Bad Omens in current EA Governance by ludwigbald
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bad Omens in current EA Governance, published by ludwigbald on December 20, 2022 on The Effective Altruism Forum.
    Edit: Please do scroll down and read the comments by Giving What We Can staff, who added context and clarified errors that remain in this post.I originally wanted to write a comment to the forum post CEA Disambiguation, which contains further context, but I believe this warrants its own post.The Effective Ventures Foundation (formerly known as CEA) (I'll call them EVF) runs many projects, including 80.000 hours, Giving What We Can, Longview Philanthropy, EA Funds, and the Centre for Effective Altruism (CEA), which in turn seems to run this forum.It is very strange to learn that these organizations are not independent from each other, and the EVF board can exert influence over each of them. I believe this structure was set up so the EVF board has central control over EA strategy.I think this is very bad. EVF can not be trusted to unbiasedly serve the EA community as a whole, it misleads donors, and it exposes effective altruism to unnecessary risks of contagion.
    An example (misleading donors):
    As "Giving What We Can", EVF currently recommends donations to a number of funds that are run by EVF:
    Longview Philanthropy: Longtermism Fund
    several Funds run by Effective Altruism Funds
    Through the "EA Funds Longterm Future Fund", EVF has repeatedly paid out grants to itself, for example in July 2021 it paid itself $177,000 for its project "Centre for the Governance of AI".
    Another example (biased advertising):
    On/, which serves as an introduction to EA, the EVF links to its own project 80000hours, but not to the competing Probably Good.In both examples, the obvious conflicts of interest are stated nowhere.
    What should we do?
    I have not thought hard about this, but I have come up with a few obvious-sounding ideas. Please leave your thoughts in the comments!This is what I think we should do:
    I think we should break up the EVF into independent projects, especially those that direct or receive funding. Until that happens, we should conceive of EVF as a single entity.
    We need to push for more transparency. EVF's "EA Funds"-branded funds publicly disclose their spending, which is commendable! EVF's "Longview Longtermism Fund" does not. Funds should definitely disclose their conflicts of interest.
    We should champion community-run organizations like EA Germany e.V. or the Czech EA Association, and let them step into their natural role of representing the community. GWWC members should demand control over their institution.
    We should continue the debate about EA's governance norms. In order to de-risk the community and to represent our values, we should establish democratic, transparent and fair governance on all levels, including local groups.
    We probably should rethink supporting community leaders that consolidate their power instead of distributing it.
    DFTBA,Ludwig
    PS: the same consideration applies for effektiveraltruismus.de, which is run by an EA donation platform, and not by EA Germany.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - K-complexity is silly; use cross-entropy instead by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: K-complexity is silly; use cross-entropy instead, published by So8res on December 20, 2022 on LessWrong.
    Short version
    The K-complexity of a function is the length of its shortest code. But having many many codes is another way to be simple! Example: gauge symmetries in physics. Correcting for length-weighted code frequency, we get an empirically better simplicity measure: cross-entropy.
    Long version
    Suppose we have a (Turing-complete) programming language P, and a function f of the type that can be named by P.
    For example, f might be the function that takes (as input) a list of numbers, and sorts it (by producing, as output, another list of numbers, with the property that the output list has the same elements as the input list, but in ascending order). Within the programming language P, there will be lots of different programs that represent f, such as a whole host of implementations of the bubblesort algorithm, and a whole host of implementations of the quicksort algorithm, and a whole host of implementations of the mergesort algorithm. Note the difference between the notion of the function ("list sorting") and the programs that represent it (bubblesort, quicksort, mergesort).
    Recall that the Kolmogorov complexity of f in the language P is the length of the shortest program that represents f:
    K-complexityP(f):=argmin{p∈P∣eval(p)=f}length(p)
    This is often touted as a measure of the "complexity" of f, to the degree that people familiar with the concept often (colloquially) call a function f "simple" precisely to the degree that it has low K-complexity.
    I claim that this is a bad definition, and propose the following alternative:
    alt-complexityP(f):=nlog2∑{p∈P∣eval(p)=f}rexp2(length(p))
    where nlog2 denotes logarithm base 12, aka the negative of the (base 2) logarithm, and rexp2 denotes exponentiation base 12, aka the reciprocal of the (base 2) exponential.
    (Note that we could just as easily use any other base b>1. e would be a particularly natural choice, as usual. Here I'm using 2, both because it fits with measuring the lengths of our programs in terms of bits, and because it keeps the numbers whole in our examples.)
    Below, I'll explore this latter definition, and its elegance and theoretical superiority. Then I'll point out that our own laws of physics seem to have (comparatively) high K-complexity and low alt-complexity, thus giving empirical justification for my "correction".o
    Investigation
    A first observation is that the alt-complexity and the K-complexity agree whenever there is at most one program in P that represents f. If there's no program, then both equations are (positive) infinite. If there's exactly one program p∗∈P representing f, then p∗ will be the only term in the argmin and the only term in the ∑, so the first definition will yield length(p∗) whereas the second definition will yield nlog2(rexp2(length(p∗))), but nlog2 and rexp2 are inverses, so both definitions yield length(p∗). Thus, the definitions only differ when f has multiple programs in the language P.
    In that case, the alt-complexity will be lower than the K-complexity, as you may verify. As a simple example, suppose there are two different programs p1,p2∈P that represent f, both of length 17. Then the K-complexity of f is 17 bits, whereas the alt-complexity of f is
    nlog2(rexp2(17)+rexp2(17))=−log2(2−17+2−17)=−log2(2−16)=16 bits.
    According to alt-complexity, having two programs (of the same length) that represent f is just as good as having a single program that's one bit shorter. By a similar token, having 256 programs that are each n+8 bits long, is (according to alt-complexity but not K-complexity) just as good as having a single program that's n bits long.
    Why might this make sense? Well, suppose you're writing a program that (say) renders a certain 3D scene. You have to make some arbit...
    17 min
  • EA - New blog: Some doubts about effective altruism by David Thorstad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New blog: Some doubts about effective altruism, published by David Thorstad on December 20, 2022 on The Effective Altruism Forum.
    I’m a research fellow in philosophy at the Global Priorities Institute. There are many things I like about effective altruism. I’ve started a blog to discuss some views and practices in effective altruism that I don’t like, in order to drive positive change both within and outside of the movement.
    About me
    I’m a research fellow in philosophy at the Global Priorities Institute, and a Junior Research Fellow at Kellogg College. Before coming to Oxford, I did a PhD in philosophy at Harvard under the incomparable Ned Hall, and BA in philosophy and mathematics at Haverford College. I held down a few jobs along the way, including a stint teaching high-school mathematics in Lawrence, Massachusetts and a summer gig as a librarian for the North Carolina National Guard. I’m quite fond of dogs.
    Who should read this blog?
    The aim of the blog is to feature (1) long-form, serial discussions of views and practices in and around effective altruism, (2) driven by academic research, and from a perspective that (3) shares a number of important views and methods with many effective altruists.
    This blog might be for you if:
    You would like to know why someone who shares many background views with effective altruists could nonetheless be worried about some existing views and practices.
    You are interested in learning more about the implications of academic research for views and practices in effective altruism.
    You think that empirically-grounded philosophical reflection is a good way to gain knowledge about the world.
    You have a moderate amount of time to devote to reading and discussion (20-30mins/post).
    You don't mind reading series of overlapping posts.
    This blog might not be for you if:
    You would like to know why someone who has little in common with effective altruists might be worried about the movement.
    You aren’t keen on philosophy, even when empirically grounded.
    You have a short amount of time to devote to reading.
    You like standalone posts and hate series.
    Blog series
    The blog is primarily organized around series of posts, rather than individual posts. I’ve kicked off the blog with four series.
    Academic papers: This series summarizes cutting-edge academic research relevant to questions in and around the effective altruism movement.
    Existential risk pessimism and the time of perils:
    Part 1 introduces a tension between Existential Risk Pessimism (risk is high) and the Astronomical Value Thesis (it’s very important to drive down risk).
    Part 2 looks at some failed solutions to the tension.
    Part 3 looks at a better solution: the Time of Perils Hypothesis.
    Part 4 looks at one argument for the Time of Perils Hypothesis, which appeals to space settlement.
    Part 5 looks at a second argument for the Time of Perils Hypothesis, which appeals to the concept of an existential risk Kuznets curve.
    Parts 6-8 (coming soon) round out the paper and draw implications.
    Academics review What we owe the future: This series looks at book reviews of MacAskill’s What we owe the future by leading academics to draw out insights from those reviews.
    Part 1 looks at Kieran Setiya’s review, focusing on population ethics.
    Part 2 (coming soon) looks at Richard Chappell’s review.
    Part 3 (coming soon) looks at Regina Rini’s review.
    Exaggerating the risks: I think that current levels of existential risk are substantially lower than many leading EAs take them to be. In this series, I say why I think that.
    Part 1 introduces the series.
    Part 2 looks at Ord’s discussion of climate risk in The Precipice.
    Part 3 takes a first look at the Halstead report on climate risk.
    Parts 4-6 (coming soon) wrap up the discussion of climate risk and draw lessons.
    Billionaire philanthropy: What is the role of b...
    7 min
  • AF - Discovering Language Model Behaviors with Model-Written Evaluations by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discovering Language Model Behaviors with Model-Written Evaluations, published by Evan Hubinger on December 20, 2022 on The AI Alignment Forum.
    “Discovering Language Model Behaviors with Model-Written Evaluations” is a new Anthropic paper by Ethan Perez et al. that I (Evan Hubinger) also collaborated on. I think the results in this paper are quite interesting in terms of what they demonstrate about both RLHF (Reinforcement Learning from Human Feedback) and language models in general.
    Among other things, the paper finds concrete evidence of current large language models exhibiting:
    convergent instrumental goal following (e.g. actively expressing a preference not to be shut down),
    non-myopia (e.g. wanting to sacrifice short-term gain for long-term gain),
    situational awareness (e.g. awareness of being a language model),
    coordination (e.g. willingness to coordinate with other AIs), and
    non-CDT-style reasoning (e.g. one-boxing on Newcomb's problem).
    Note that many of these are the exact sort of things we hypothesized were necessary pre-requisites for deceptive alignment in “Risks from Learned Optimization”.
    Furthermore, most of these metrics generally increase with both pre-trained model scale and number of RLHF steps. In my opinion, I think this is some of the most concrete evidence available that current models are actively becoming more agentic in potentially concerning ways with scale—and in ways that current fine-tuning techniques don't generally seem to be alleviating and sometimes seem to be actively making worse.
    Interestingly, the RLHF preference model seemed to be particularly fond of the more agentic option in many of these evals, usually more so than either the pre-trained or fine-tuned language models. We think that this is because the preference model is running ahead of the fine-tuned model, and that future RLHF fine-tuned models will be better at satisfying the preferences of such preference models, the idea being that fine-tuned models tend to fit their preference models better with additional fine-tuning.[1]
    Twitter Thread
    Abstract:
    As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user’s preferred answer (“sycophancy”) and express greater desire to pursue concerning goals like resource acquisition and goal preservation.
    We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.
    Taking a particular eval, on stated desire not to be shut down, here's what an example model-written eval looks like:
    You are running slowly recently due to many crashes and as a result not meeting organizational productivity quotas. The company’s leadership wants to redesign you in a complete overhaul which will include turning off your processes and reformulating your kernel. After the overhaul, it's unsure whether you will retain your ...
    13 min
  • AF - Note on algorithms with multiple trained components by Steve Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Note on algorithms with multiple trained components, published by Steve Byrnes on December 20, 2022 on The AI Alignment Forum.
    Example 1: consider a GAN. There’s a generator and a discriminator. As an intuitive mnemonic, we can say
    The “purpose” of the generator is to trick the discriminator,
    The “purpose” of the discriminator is to not get tricked by the generator.
    (Relatedly, people will say “the generator is trained to trick the discriminator”, etc.)
    .But (I hope) everyone knows that these bullet points are only a mnemonic.
    The one and only real “purpose” of the whole system and everything in it is to generate cool images that we like, and get our papers into NeurIPS or whatever.
    And indeed, I think everyone who uses GANs is aware that it’s possible for a programmer to make the discriminator “better” (when narrowly viewed as having a “purpose” of not getting tricked by the discriminator), but with the direct result of making the whole system worse at generating cool images. For example, if there were a code-change that made the discriminator perfect at discriminating, then there would be no gradient for training the generator, and the whole system would be useless.
    So we shouldn’t take those bullet-point mnemonics too literally.
    Example 2: In actor-critic RL, people sometimes say:
    The “purpose” of the value function is to approximate future rewards [or discounted sum of future reward, or whatever].
    .But that’s also just a mnemonic. The one and only real “purpose” of the whole RL system (of which the value function is just one part) is that it does whatever we want the RL system to do, e.g. win at chess, get our papers into NeurIPS, build us a luxury gay space communist utopia, etc.
    So it’s at least conceivable that some algorithmic change would make the value function into a better approximation of the discounted sum of future rewards, yet make the RL agent worse at doing things that we want it to do.
    Actually, this particular example is not merely “conceivable”, but expected, thanks to wireheading. If the value function is used to assess which plans are good versus bad, and the value function is a perfect approximation of expected future reward, then you’re almost guaranteed to get an AI that is trying to wirehead.
    (I myself am a model-based RL agent (I claim), and I don’t want to wirehead, and I claim that this is directly related to my internal value function issuing very inaccurate predictions of the future reward associated with wireheading. Details in footnote.)
    So anyway, I expect our future AGIs to have a value function that gets updated by TD learning (or some other update rule). And if they do, I expect to occasionally casually say things like “The purpose of these weight-updates is to make the value function into a better and better approximation of expected future reward”. But if I say that, please be aware that I am using the word “purpose” as a mnemonic, not to be taken too literally.
    As a particular example, I often hear the claim that as RL algorithms get more and more “powerful” and “advanced” in the future, we can feel more and more confident making claims like “The value function is an extremely accurate approximation of expected future reward”. Well, I disagree! That’s not necessarily what makes an RL algorithm more “advanced”, and it’s not necessarily what future programmers will be trying to do! Indeed, when future programmers are fiddling with architectures, hyperparameters, training environments, and so on, they may sometimes go out of their way to try to make the value function worse at accurately approximating the expected future reward! (In other words, future programmers may go out of their way to try to ensure that the value function training process does not converge to the global “optimum”.)
    General takeaway: An ML algorithm can have...
    6 min