The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Chilean AIS Hackathon Retrospective by Agustín Covarrubias
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chilean AIS Hackathon Retrospective, published by Agustín Covarrubias on May 9, 2023 on The Effective Altruism Forum.
    TL;DR
    We hosted an AI Safety “Thinkathon” in Chile. We had participation from 40 students with differing skill levels and backgrounds, with groups totalling 13 submissions.
    We see potential in:
    Similar introductory events aiming for a broad audience
    Collaborating more often with student organizations
    Leveraging remote help from external mentors
    We experimented with an alternative naming, having remote mentors, different problem sources, and student organization partnerships, with varying results.
    We could have improved planning and communicating the difficulty of challenges.
    Introduction
    In February, we ran the first AI Safety Hackathon in Chile (and possibly in all of South America). This post provides some details about the event, a teaser of some resulting projects and our learnings throughout.
    Goals and overview of the event
    The hackathon was meant to kick-start our nascent AI Safety Group at UC Chile, generating interest in AI Safety and encouraging people to register for our AGI Safety Fundamentals course group.
    It ran between the 25th and the 28th of February, the first two days being in-person events and the other two serving as additional time for participants to work on their proposals, with some remote assistance on our part. Participants formed teams of up to four people, and could choose to assist either virtually (through Discord) or in-person (on the first two days).
    We had help from Apart Research and partial funding from AI Alignment Awards.
    Things we experimented with
    Aiming for a broad audience, we named the event “Thinkathon” (instead of hackathon) and provided plenty of introductory material alongside the proposed problems.
    We think this was the right choice, as the desired effect was reflected on the participant demographics (see below).
    We could have been better at preparing participants. Some participants suggested we could have done an introductory workshop.
    We incorporated the two problems from the AI Alignment Awards (Goal Misgeneralization and the Shutdown problem), alongside easier, self-contained problems aimed at students with different backgrounds (like policy or psychology).
    We think most teams weren't prepared to tackle the AI Alignment Awards challenges. Most teams (77%) chose them initially regardless of their experience, getting stuck quickly.
    This might have worked better by communicating difficulty more clearly, as well as emphasizing that aiming for incremental progress rather than a complete solution is a better strategy for a beginner's hackathon.
    As we don't know many people with previous experience in AIS in Chile, we got help from external mentors, which connected remotely to help participants.
    We think this was a good decision, as participants rated mentor support highly (see below).
    We collaborated actively with two student governments from our university (the Administration and Economics Student Council and the Engineering Student Council). They helped with funding, logistics and outreach.
    We think this was an excellent choice, as they provided a much broader platform for outreach and crucial logistics help.
    We had a great time working with them, and they were eager to work with us again!
    Things that went well
    40 people attended in-person and 10 people remotely (through Discord), we were surprised by both the high number of attendants and the preference for in-person participation.
    We had a total of 13 submitted proposals, much higher than expected.
    While all proposals were incremental contributions, most were of high quality.
    Skill level and majors varied significantly, going from relatively advanced CS students to freshmen from other fields (like economics). We were aiming for diversity, so this is a w...
    10 min
  • LW - Most people should probably feel safe most of the time by Kaj Sotala
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Most people should probably feel safe most of the time, published by Kaj Sotala on May 9, 2023 on LessWrong.
    There is an idea that I’ve sometimes heard around rationalist and EA circles, that goes something like “you shouldn’t ever feel safe, because nobody is actually ever safe”. I think there are at least two major variations of this:
    You shouldn’t ever feel safe, because something bad could happen at any time. To think otherwise is an error of rationality.
    You shouldn’t ever feel safe, because AI timelines might be short and we might be about to die soon. Thus, to think that you’re safe is making an error of rationality.
    I’m going to argue against both of these. If you already feel like both of these are obviously wrong, you might not need the rest of this post.
    Note that I only intend to dispute the intellectual argument that these are making. It’s possible to accept on an intellectual level that it would make sense to feel safe most of the time, but still not feel safe. That kind of emotional programming requires different kinds of tools to deal with. I’m mostly intending to say that if you feel safe, you don’t need to feel bad about that. You don’t need to make yourself feel unsafe; for most people, it’s perfectly rational to feel safe.
    I do expect some of the potential readers of this post to live in a very unsafe environment - e.g. parts of current-day Ukraine, or if they live together with someone abusive - where they are actually in constant danger. For them, it may make sense to feel unsafe all the time. (If you are one of them, I genuinely hope things get better for you soon.) But these are clearly situations where something has gone badly wrong; the feeling that one has in those situations shouldn’t be something that one was actively striving for. I think that any reader who doesn’t live in an actively horrendous situation would do better to feel safe most of the time. (Short timelines don't count as a horrendous situation, for reasons that I'll get into.)
    As I interpret it, the core logic in both of the “you shouldn’t ever feel safe” claims goes as follows:
    To feel safe implies a belief that nothing bad is going to happen to you
    But something bad can happen to you at any time, even when you don’t expect it. In the case of AI, we even have reasons to put a significant probability on this in fact happening soon.
    Thus, feeling safe requires having an incorrect belief, and the rational course of action is to not feel safe.
    One thing that you might notice from looking at this argument is that one could easily construct an exactly opposite one as well.
    To feel unsafe implies a belief that things aren’t going to go well for you.
    But things can go well for you, even when you don’t expect it. In the case of AI, we even have reasons to put a significant probability on things going well.
    Thus, feeling unsafe requires having an incorrect belief, and the rational course of action is to feel safe.
    That probably looks obviously fallacious - just because things can go well, doesn’t mean that it would be warranted to always feel safe. But why then would it be warranted to feel unsafe in the case where things just can go badly?
    To help clarify our thinking, let's take a moment to look at how the US military orients to the question of being safe or not. More specifically, to the question of whether a given military unit is reasonably safe or whether it should prepare for an imminent battle.
    Readiness Condition levels are a series of standardized levels that a unit’s commander uses to adjust the unit’s readiness to move and fight. Here’s an abridged summary of them:
    REDCON-1. Full alert; unit ready to move and fight. The unit’s equipment and NBC alarms are stowed, soldiers at observation posts are pulled in. All personnel are alert and mounted on vehicles. Weapons are mann...
    16 min
  • LW - Yoshua Bengio argues for tool-AI and to ban "executive-AI" by habryka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yoshua Bengio argues for tool-AI and to ban "executive-AI", published by habryka on May 9, 2023 on LessWrong.
    Yoshua Bengio wrote a blogpost yesterday in which he argues for developing "scientist AI", which seems in-structure very similar to historical Tool-AI proposals.
    For the (IMO) best response to this kind of proposal see Gwern's: Why Tool AIs Want to Be Agent AIs.
    Below I copies the blogpost in-full, since all of it seems pretty relevant.
    AI Scientists: Safe and Useful AI?
    Published 7 May 2023 by yoshuabengio
    There have recently been lots of discussions about the risks of AI, whether in the short term with existing methods or in the longer term with advances we can anticipate. I have been very vocal about the importance of accelerating regulation, both nationally and internationally, which I think could help us mitigate issues of discrimination, bias, fake news, disinformation, etc. Other anticipated negative outcomes like shocks to job markets require changes in the social safety net and education system. The use of AI in the military, especially with lethal autonomous weapons has been a big concern for many years and clearly requires international coordination.
    In this post however, I would like to share my thoughts regarding the more hotly debated question of long-term risks associated with AI systems which do not yet exist, where one imagines the possibility of AI systems behaving in a way that is dangerously misaligned with human rights or even loss of control of AI systems that could become threats to humanity. A key argument is that as soon as AI systems are given goals – to satisfy our needs – they may create subgoals that are not well-aligned with what we really want and could even become dangerous for humans.
    Main thesis: safe AI scientists
    The bottom line of the thesis presented here is that there may be a path to build immensely useful AI systems that completely avoid the issue of AI alignment, which I call AI scientists because they are modeled after ideal scientists and do not act autonomously in the real world, only focusing on theory building and question answering. The argument is that if the AI system can provide us benefits without having to autonomously act in the world, we do not need to solve the AI alignment problem. This would suggest a policy banning powerful autonomous AI systems that can act in the world (“executives” rather than “scientists”) unless proven safe. However, such a solution would still leave open the political problem of coordinating people, organizations and countries to stick to such guidelines for safe and useful AI.
    The good news is that current efforts to introduce AI regulation (such as the proposed bills in Canada and the EU, but see action in the US as well) are steps in the right direction.
    The challenge of value alignment
    Let us first recap the objective of AI alignment and the issue with goals and subgoals. Humanity is already facing alignment problems: how do we make sure that people and organizations (such as governments and corporations) act in a way that is aligned with a set of norms acting as a proxy for the hard-to-define general well-being of humanity? Greedy individuals and ordinary corporations may have self-interests (like profit maximization) that can clash with our collective interests (like preserving a clean and safe environment and good health for everyone). Politics, laws, regulations and international agreements all imperfectly attempt to deal with this alignment problem. The widespread adoption of norms which support collective interests is enforced by design in democracies, to an extent, due to limitations on the concentration of power by any individual person or corporation.
    It is further aided by our evolved tendency to adopt prevailing norms voluntarily if we recognise their general valu...
    13 min
  • LW - When is Goodhart catastrophic? by Drake Thomas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When is Goodhart catastrophic?, published by Drake Thomas on May 9, 2023 on LessWrong.
    Thanks to Aryan Bhatt, Eric Neyman, and Vivek Hebbar for feedback.
    This post gets more math-heavy over time; we convey some intuitions and overall takeaways first, and then get more detailed. Read for as long as you're getting value out of things!
    TLDR
    How much should you optimize for a flawed measurement? If you model optimization as selecting for high values of your goal V plus an independent error X, then the answer ends up being very sensitive to the distribution of the error X: if it’s heavy-tailed you shouldn't optimize too hard, but if it’s light-tailed you can go full speed ahead.
    Related work
    Why the tails come apart by Thrasymachus discusses a sort of "weak Goodhart" effect, where extremal proxy measurements won't have extremal values of your goal (even if they're still pretty good). It implicitly looks at cases similar to a normal distribution.
    Scott Garrabrant's taxonomy of Goodhart's Law discusses several ways that the law can manifest. This post is about the "Regressional Goodhart" case.
    Scaling Laws for Reward Model Overoptimization (Gao et al., 2022) considers very similar conditioning dynamics in real-world RLHF reward models. In their Appendix A, they show a special case of this phenomenon for light-tailed error, which we'll prove a generalization of in the next post.
    Defining and Characterizing Reward Hacking (Skalse et al., 2022) shows that under certain conditions, leaving any terms out of a reward function makes it possible to increase expected proxy return while decreasing expected true return.
    How much do you believe your results? by Eric Neyman tackles very similar phenomena to the ones discussed here, particularly in section IV; in this post we're interested in characterizing that sort of behavior and when it occurs. We strongly recommend reading it first if you'd like better intuitions behind some of the math presented here - though our post was written independently, it's something of a sequel to Eric's.
    Motivation/intuition
    Goodhart's Law says
    When a measure becomes a target, it ceases to be a good measure.
    When I (Drake) first heard about Goodhart's Law, I internalized something like "if you have a goal, and you optimize for a proxy that is less than perfectly correlated with the goal, hard enough optimization for the proxy won't get you what you wanted." This was a useful frame to have in my toolbox, but it wasn't very detailed - I mostly had vague intuitions and some idealized fables from real life.
    Much later, I saw some objections to this frame on Goodhart that actually used math. The objection went something like:
    Let's try to sketch out an actual formal model here. What's the simplest setup of "two correlated measurements"? We could have a joint normal distribution over two random variables, U and V, with zero mean and positive covariance. You actually value V, but you measure a proxy U. Then we can just do the math: if I optimize really hard for U, and give you a random datapoint with U=1012 or something, how much V do you expect to get?
    If we look at the joint distribution of U and V, we'll see a distribution with elliptical contour lines, like so:
    Now, the naïve hope is that expected V as a function of observed U would go along the semi-major axis, shown in red below:
    But actually we'll get the blue line, passing through the points at which the ellipses are tangent to the V-axis.
    Importantly, though, we're still getting a line: we get linearly more value V for every additional unit of U we select for! Applying 99th percentile selection on U isn't going to be as good as 99th percentile selection on V, but it's still going to give us more V than any lower percentile selection on U. The proxy is inefficient, but it's not doomed.
    Lately, however, ...
    14 min
  • AF - When is Goodhart catastrophic? by Drake Thomas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When is Goodhart catastrophic?, published by Drake Thomas on May 9, 2023 on The AI Alignment Forum.
    Thanks to Aryan Bhatt, Eric Neyman, and Vivek Hebbar for feedback.
    This post gets more math-heavy over time; we convey some intuitions and overall takeaways first, and then get more detailed. Read for as long as you're getting value out of things!
    TLDR
    How much should you optimize for a flawed measurement? If you model optimization as selecting for high values of your goal V plus an independent error X, then the answer ends up being very sensitive to the distribution of the error X: if it’s heavy-tailed you shouldn't optimize too hard, but if it’s light-tailed you can go full speed ahead.
    Related work
    Why the tails come apart by Thrasymachus discusses a sort of "weak Goodhart" effect, where extremal proxy measurements won't have extremal values of your goal (even if they're still pretty good). It implicitly looks at cases similar to a normal distribution.
    Scott Garrabrant's taxonomy of Goodhart's Law discusses several ways that the law can manifest. This post is about the "Regressional Goodhart" case.
    Scaling Laws for Reward Model Overoptimization (Gao et al., 2022) considers very similar conditioning dynamics in real-world RLHF reward models. In their Appendix A, they show a special case of this phenomenon for light-tailed error, which we'll prove a generalization of in the next post.
    Defining and Characterizing Reward Hacking (Skalse et al., 2022) shows that under certain conditions, leaving any terms out of a reward function makes it possible to increase expected proxy return while decreasing expected true return.
    How much do you believe your results? by Eric Neyman tackles very similar phenomena to the ones discussed here, particularly in section IV; in this post we're interested in characterizing that sort of behavior and when it occurs. We strongly recommend reading it first if you'd like better intuitions behind some of the math presented here - though our post was written independently, it's something of a sequel to Eric's.
    Motivation/intuition
    Goodhart's Law says
    When a measure becomes a target, it ceases to be a good measure.
    When I (Drake) first heard about Goodhart's Law, I internalized something like "if you have a goal, and you optimize for a proxy that is less than perfectly correlated with the goal, hard enough optimization for the proxy won't get you what you wanted." This was a useful frame to have in my toolbox, but it wasn't very detailed - I mostly had vague intuitions and some idealized fables from real life.
    Much later, I saw some objections to this frame on Goodhart that actually used math. The objection went something like:
    Let's try to sketch out an actual formal model here. What's the simplest setup of "two correlated measurements"? We could have a joint normal distribution over two random variables, U and V, with zero mean and positive covariance. You actually value V, but you measure a proxy U. Then we can just do the math: if I optimize really hard for U, and give you a random datapoint with U=1012 or something, how much V do you expect to get?
    If we look at the joint distribution of U and V, we'll see a distribution with elliptical contour lines, like so:
    Now, the naïve hope is that expected V as a function of observed U would go along the semi-major axis, shown in red below:
    But actually we'll get the blue line, passing through the points at which the ellipses are tangent to the V-axis.
    Importantly, though, we're still getting a line: we get linearly more value V for every additional unit of U we select for! Applying 99th percentile selection on U isn't going to be as good as 99th percentile selection on V, but it's still going to give us more V than any lower percentile selection on U. The proxy is inefficient, but it's not doomed.
    Late...
    14 min
  • LW - A technical note on bilinear layers for interpretability by Lee Sharkey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A technical note on bilinear layers for interpretability, published by Lee Sharkey on May 8, 2023 on LessWrong.
    Summary
    In this short theoretical note (now on Arxiv) I examine bilinear layers, which are MLP layers that take the form
    MLPBilinear(x)=(W1x)⊙(W2x).
    When used in language models, they perform better than standard MLPs with elementwise activation functions (but appear very slightly below state of the art).
    Despite their competitiveness, they are mathematically much easier to analyze: Although they are nonlinear functions of their input, bilinear layers can be expressed using only linear operations and third order tensors.
    Because they can be linearized, we can extend 'A Mathematical Framework for Transformer Circuits' (Elhage et al. 2022) beyond attention-only transformers to transformers with both attention and MLP layers.
    In a similar way to how the analysis of Elhage et al. (2022) helped to reveal QK- and OV-circuits, induction heads, and virtual attention heads, the analyzability of bilinear layers may lend them to deeper safety insights by allowing us to talk more formally about circuits in large language models.
    Additionally, and more speculatively, bilinear layers might offer an alternative path for mechanistic interpretability through understanding the mechanisms of feature construction instead of having to enumerate and understand a (potentially exponentially) large number of features in large models.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Inference Speed is Not Unbounded by OMN
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inference Speed is Not Unbounded, published by OMN on May 8, 2023 on LessWrong.
    [The intro of this post has been lightly edited since it was first posted to address some comments. I have also changed the title to better reflect my core argument. My apologies if that is not considered good form.]
    This post will be a summary of some of my ideas on what intelligence is, the processes by which it’s created, and a discussion of the implications. Although I prefer to remain pseudonymous, I do have a PhD in Computer Science and I’ve done AI research at both Amazon and Google Brain. I spent some time tweaking the language in order to minimize how technical you need to be to read it.
    There is a recurring theme I've seen in discussions about AI where people express incredulity about neural networks as a method for AGI since they require so much "more data" than humans to train. On the other hand, I see some people discussing superintelligences that make impossible inferences given virtually no input data, positing AI that will instantly do inconceivable amounts of processing. Both of these very different arguments are making statements about learning speed, and in my opinion mischaracterize what learning actually looks like.
    My basic argument is that the there are probably mathematical limits on how fast it is possible to learn. This means, for instance, that training an intelligent system will always take more data and time than might initially seem necessary. What I’m arguing is that intelligence isn’t magic - the inferences a system makes have to come from somewhere. They have to be built, and they have to be built sequentially. The only way you get to skip steps, and the only reason intelligence exists at all, is that it is possible to reuse knowledge that came from somewhere else.
    Three Apples and a Blade of Grass
    Because I think it makes a good jumping off point, I’m going to start by framing this around a recent discussion I saw around a years-old quote from Yudowsky about superintelligence:
    A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis . by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
    The linked post does a good job tearing this down. It correctly points out that there are basically an infinite number of possible universes, and three frames of an apple dropping are not nearly enough to conclude you exist in ours. I might even argue that the author still overstates the degree to which three images could reduce the number of universes in consideration; for instance the changing patterns on a falling apple don’t actually tell you it’s in a 3D world, that would require you to understand how light interacts with objects.
    Still, I do think the author successfully explains why, at a literal level, EY’s statement is wrong.
    But at a deeper level, this entire framing still feels very off to me, as if even asking that question is making a category error. It feels like everyone is asking what the number three ate for breakfast. It suggests that one could have a system that is simultaneously superintelligent but has absolutely no knowledge about the world at all.
    Knowing what we now know about intelligence, I just don’t think that’s possible. And I don’t just mean it’s impractical, or we just aren't capable of building an AI like that. I mean that I believe with very high confidence that such a thing would be a mathematical impossibility.
    I think there’s a human tendency to want a certain type of structure to intelligence, and I see this assumed a lot in places like this forum (I was a lurker before I made this account). There’s a desire to see intelligence as synonymous with learning, where existing knowledge is something comple...
    26 min
  • LW - Understanding mesa-optimization using toy models by tilker
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding mesa-optimization using toy models, published by tilker on May 7, 2023 on LessWrong.
    Overview
    Solving the problem of mesa-optimization would probably be easier if we understood how models do search internally
    We are training GPT-type models on the toy task of solving mazes and studying them in both a mechanistic interpretability and behavioral context.
    This post lays out our model training setup, hypotheses we have, and the experiments we are performing and plan to perform. Experimental results will be forthcoming in our next post.
    We invite members of the LW community to challenge our hypotheses and the potential relevance of this line of work. We will follow up soon with some early results. Our main source code is open source, and we are open to collaborations.
    Introduction
    Some threat models of misalignment presuppose the existence of an agent which has learned to perform a search over actions to effectively achieve goals. Such a search process might involve exploring different sequences of actions in parallel and evaluating the best sequence of actions to achieve some goal.
    To deepen our understanding of what it looks like when models are actually performing search, we chose to train simple GPT-2 like models to find the shortest paths through mazes. Maze-solving models provide a tractable and interesting object of study, as the structure of both the problem and solutions is extensively studied. This relative simplicity makes identifying and understanding search through the lens of mechanistic and behavioral experiments much more concrete than working with pre-trained LLMs and more feasible in the context of limited computational resources.
    Connections to mesa-optimization
    Mesa-optimizers are learned optimizers for an objective that can be distinct from the base-objective. Inner misalignment can occur when the AI system develops an internal optimization process that inadvertently leads to the pursuit of an unintended goal. In the context of search, the propensity for mesa-optimization may be increased as the system explores various future states, potentially identifying alternative objectives that appear at least as rewarding or efficient in achieving the desired outcome. Existing literature on search has highlighted the potential for unintended consequences of search in ML systems.
    One lens of viewing the problem of mesa-optimization is that the behavior of a system changes in an undesirable way upon a distributional shift, and we believe that mazes provide a number of mechanisms to create such distributional shifts.
    Training setup
    We first aim to train a transformer model to predict the shortest path between a given start and end position in a maze.
    The maze exists as a 2D grid, with each position on the grid encoded as a single token. For example, a 5x5 maze has 25 coordinates that have corresponding tokens in the vocabulary.
    To the transformer, the maze is described as:
    An adjacency list containing all connections between pairs of positions: for example, (0,0) <--> (0,1).
    A "wall" in the maze is merely a missing connection between positions in the maze, but otherwise not explicitly stated.
    The start and end positions are coordinates on the maze grid, such as (3,3) and (4,0), respectively.
    A training example contains a maze (as an adjacency list), start and end coordinates, and a path consisting of position tokens
    We use an autoregressive decoder-only transformer model (implemented using TransformerLens), which (at inference) makes predictions one token at a time based on previously generated tokens. Our transformer models incorporate layer normalization and MLP layers by default.
    One training sample consists of a maze, as well as a unique path connecting randomly selected origin and target coordinates (circle and cross). The solved maze shown abo...
    18 min
  • AF - Announcing “Key Phenomena in AI Risk” (facilitated reading group) by Nora Ammann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing “Key Phenomena in AI Risk” (facilitated reading group), published by Nora Ammann on May 9, 2023 on The AI Alignment Forum.
    TLDR: “Key Phenomena in AI risk” is a 7 week-long, facilitated reading group. It is aimed at people interested in conceptual AI alignment research, in particular from fields such as philosophy, systems research, biology, cognitive and social sciences.
    The program will run between July and August 2023. Sign up here by May 28th.
    What?
    The “Key Phenomena in AI risk” reading curriculum provides an extended introduction to some key ideas in AI risk, in particular risks from misdirected optimization or 'consequentialist cognition'. As such, it aims to remain largely agnostic of solution paradigms.
    See here for a short overview of the curriculum; here for a more extensive summary; and here for the full curriculum.
    This is a 7-week long program, which consists of a weekly 90’ facilitated call to discuss the week’s key phenomena and readings, as well as individual time for reading (min. 2h, more if you would like to explore the optional readings).
    The courses are virtual and free of charge.
    For Who?
    The curriculum is primarily aimed at people interested in conceptual research in AI risk and alignment.
    It is designed to be accessible to audiences in, among others, philosophy (of agency, knowledge, power, etc.) and systems research (e.g. biological, cognitive, information-theoretic, social systems, etc.).
    When?
    The reading groups will be taking place in July and August 2023.
    We expect to run 2-6 groups à 4-8 participants (including 1 facilitator). Each group will be led by a facilitator with substantive knowledge of AI risk.
    Overview of the curriculum
    Week 0 is dedicated to getting to know each other and clarifying how the program will work.
    Week 1 focuses on why important features of generally intelligent 'consequentialist cognition' might be algorithmically realizable, and its potential implications.
    Week 2 focuses on why it can be hard to direct such intelligence in a safe and beneficial direction.
    Week 3 discusses instrumental convergence in goal-oriented systems. .
    Week 4 discusses risks from systems that seek predictive omniscience.
    Week 5 discusses some factors on why surveilling (or oversight of) these artificial systems may be fraught with differential advantage for deceptive tendencies.
    Week 6 discusses why even an incoherent aggregation of optimizing systems could still impose a (misaligned) optimizing pressure in the world.
    Here is a longer summary. Here is the full curriculum.
    The curriculum has been developed by TJ (Research Scholar FHI) with inputs from Nora Ammann, Sahil Kulshrestha, and Tsvi Benson-Tilsen. The program is operationally supported by PIBBSS.
    The curriculum was initially developed as part of the PIBBSS summer research fellowship,. but we realized that it might be of interest and useful to people outside of the fellowship program, too.
    We are orienting to this present round of reading groups as a way to test whether it’s worth continuing to run them on a more regular basis, as well so to help us improve the program.
    Sign up
    Sign up here by May 28th.
    About the application
    The application consists of one stage, where we ask you to fill in a form with
    Your CV
    Your motivation for participating in the program
    Your prior exposure to AI risk/alignment to date
    We select people based on our best understanding of their motivation to contribute to AI alignment and how much they would counterfactually benefit from participating in the program.
    If you have any questions, feel free to leave a comment below or contact us at [email protected]
    If you are keen to facilitate one of the reading group, also reach out.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Predictable updating about AI risk by Joe Carlsmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predictable updating about AI risk, published by Joe Carlsmith on May 8, 2023 on The Effective Altruism Forum.
    (Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.)
    "This present moment used to be the unimaginable future."
    Stewart Brand
    1. Introduction
    Here’s a pattern you may have noticed. A new frontier AI, like GPT-4, gets released. People play with it. It’s better than the previous AIs, and many people are impressed. And as a result, many people who weren’t worried about existential risk from misaligned AI (hereafter: “AI risk”) get much more worried.
    Now, if these people didn’t expect AI to get so much better so soon, such a pattern can make sense. And so, too, if they got other unexpected evidence for AI risk – for example, concerned experts signing letters and quitting their jobs.
    But if you’re a good Bayesian, and you currently put low probability on existential catastrophe from misaligned AI (hereafter: “AI doom”), you probably shouldn’t be able to predict that this pattern will happen to you in the future. When GPT-5 comes out, for example, it probably shouldn’t be the case that your probability on doom goes up a bunch. Similarly, it probably shouldn’t be the case that if you could see, now, the sorts of AI systems we’ll have in 2030, or 2050, that you’d get a lot more worried about doom than you are now.
    But I worry that we’re going to see this pattern anyway. Indeed, I’ve seen it myself. I’m working on fixing the problem. And I think we, as a collective discourse, should try to fix it, too. In particular: I think we’re in a position to predict, now, that AI is going to get a lot better in the coming years. I think we should worry, now, accordingly, without having to see these much-better AIs up close. If we do this right, then in expectation, when we confront GPT-5 (or GPT-6, or Agent-GPT-8, or Chaos-GPT-10) in the flesh, in all the concreteness and detail and not-a-game-ness of the real world, we’ll be just as scared as we are now.
    This essay is about what “doing this right” looks like. In particular: part of what happens, when you meet something in the flesh, is that it “seems more real” at a gut level. So the essay is partly a reflection on the epistemology of guts: of visceral vs. abstract; “up close” vs. “far away.” My views on this have changed over the years: and in particular, I now put less weight on my gut’s (comparatively skeptical) views about doom.
    But the essay is also about grokking some basic Bayesianism about future evidence, dispelling a common misconception about it (namely: that directional updates shouldn’t be predictable in general), and pointing at some of the constraints it places on our beliefs over time, especially with respect to stuff we’re currently skeptical or dismissive about. For example, at least in theory: you should never think it >50% that your credence on something will later double; never >10% that it will later 10x, and so forth. So if you’re currently e.g. 1% or less on AI doom, you should think it’s less than 50% likely that you’ll ever be at 2%; less than 10% likely that you’ll ever be at 10%, and so on. And if your credence is very small, or if you’re acting dismissive, you should be very confident you’ll never end up worried. Are you?
    I also discuss when, exactly, it’s problematic to update in predictable directions. My sense is that generally, you should expect to update in the direction of the truth as the evidence comes in; and thus, that people who think AI doom unlikely should expect to feel less worried as time goes on (such that consistently getting more worried is a red flag). But in the case of AI risk, I think at least some non-crazy views should actually expect to get more worried over time, even while being fairly non-worried now. In particular, i...
    1 hr 1 min