The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Coordinate-Free Interpretability Theory by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coordinate-Free Interpretability Theory, published by johnswentworth on September 14, 2022 on The AI Alignment Forum.
    Some interpretability work assigns meaning to activations of individual neurons or small groups of neurons. Some interpretability work assigns meaning to directions in activation-space. These are two different ontologies through which to view a net’s internals. Probably neither is really the “right” ontology, and there is at least one other ontology which would strictly outperform both of them in terms of yielding accurate interpretable structure.
    One of the core problems of interpretability (I would argue the core problem) is that we don’t know what the “right” internal ontology is for a net - which internal structures we should assign meaning to. The goal of this post is to ask what things we could possibly assign meaning to under a maximally-general ontology constraint: coordinate freedom.
    What Does Coordinate Freedom Mean?
    Let’s think of a net as a sequence of activation-states xi, with the layer i layer i+1 function given by xi+1=fi(xi).
    We could use some other coordinate system to represent each xi. For instance, we could use (high dimensional) polar coordinates, with ri=xi and ϕi a high-dimensional angle (e.g. all but one entry of a unit vector). Or, we could apply some fixed rotation to xi, e.g. in an attempt to find a basis which makes things sparse. In general, in order to represent xi in some other coordinate system, we apply a reversible transformation x′i=gi(xi), where x′i is the representation under the new coordinate system. In order to use these new coordinates while keeping the net the same overall, we transform the layer transition functions:
    f′i−1=gi∘fi−1
    f′i=fi∘g−1i
    In English: we transform into the new coordinate system when calculating the layer state xi, and undo that transformation when computing the next layer state xi+1. That way, the overall behavior remains the same while using new coordinates in the middle
    The basic idea of coordinate freedom is that our interpretability tools should not depend on which coordinate system we use for any of the internal states. We should be able to transform any layer to any coordinate system, and our interpretability procedure should still assign the same meaning to the same (transformed) internal structures.
    What Kind Of Coordinate Free Internal Structure Is Even Possible?
    Here’s one example of a coordinate free internal structure one could look for: maybe the layer ii+1 function can be written as
    fi(xi)=F(G(xi))
    for some low-dimensional G. For instance, maybe xi and xi+1 are both 512-dimensional, but xi+1 can be calculated (to reasonable precision) from a 22-dimensional summary G(xi). We call this a low-dimensional “factorization” of fi.
    (Side note: I’m assuming throughout this post that everything is differentiable. Begone, pedantic mathematicians; you know what you were thinking.)
    This kind of structure is also easy to detect in practice: just calculate the singular vector decomposition of the jacobian dfidxi at a bunch of points, and see whether the jacobian is consistently (approximately) low rank. In other words, do the obvious thing which we were going to do anyway.
    Why is this structure coordinate free? Well, no matter how we transform xi and xi+1, so long as the coordinate changes are reversible, the transformed function f′i will still factor through a low-dimensional summary. Indeed, it will factor through the same low-dimensional summary, up to isomorphism. We can also see the corresponding fact in the first-order approximation: we can multiply the jacobian on the left and right by any invertible matrix, its rank won’t change, and low-rank components will be transformed by the transformation matrices.
    . and as far as local structure goes (i.e. first-order approximation near any...
    10 min
  • LW - Deep Q-Networks Explained by Jay Bailey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deep Q-Networks Explained, published by Jay Bailey on September 13, 2022 on LessWrong.
    Note: This is a long post. The post is structured in such a way that not everyone needs to read everything - Sections 1 and 2 are skippable background information, and Sections 4 and 5 go into technical detail that not everybody wants or needs to know. Section 3 on its own is sufficient to gain a high-level understanding of DQN if you already know how reinforcement learning works.This post was made as my final project for the AGI Safety Fundamentals course.This post aims to provide a distillation of Deep Q-Networks, neural networks trained via Deep Q-Learning. The algorithm, usually just referred to as DQN, is the algorithm that first put deep reinforcement learning on the map. The 2013 paper is also the first paper in the Key Papers section of the excellent Spinning Up In Deep RL resource. As a result, anyone who wants to understand reinforcement learning is likely to start here.
    Since reading and replicating papers can be difficult, this resource aims to make the learning curve a little easier for those who want to improve their ML knowledge past an initial deep learning course.
    This post will be focused mainly on the 2015 paper, which adds some extra improvements to the 2013 paper, as well as giving more information that’s helpful for replication. This is a long post, but it’s broken up into sections so that you can only read the sections you’re interested in. It starts with background information, then goes over DQN at three levels of detail, each of which builds off the previous one.
    You should probably have a basic understanding of supervised learning in order to get the most out of this post. You should understand what a loss function is, what gradient descent is conceptually, and be familiar with the concept of a neural network.
    Supervised Learning With Neural Networks (Section 1) - Background information on deep learning. This can be safely skipped if you already understand how supervised learning problems like image classifiers work.
    Reinforcement Learning (Section 2) - Background information on RL. This can be safely skipped if you’re already familiar with the basics of reinforcement learning, like states, actions, and why reinforcement learning is harder than supervised learning.
    DQN Overview (Section 3) - This section presents DQN at a high level, without the math. I recommend everyone read this section. You can stop there, or continue on to Section 4 if you want to understand the math behind it.
    DQN Technical Explanation (Section 4) - This section builds off of Section 3, and includes the algorithms and equations that make DQN’s work. You can skip this section if you just want a high-level overview of what DQN is - the what, but not the how.
    Replicating DQN (Section 5) - This section builds off of Section 4, and includes tips and tricks to make DQN work in code. I recommend only reading this section if you’re interested in replicating this or similar papers yourself.
    Further Improvements (Section 6) - This section talks about improvements that have been made in the DQN line of research since the original DQN papers were released. If this topic interests you, you can skip Sections 4 and 5 and still understand this section.
    Regardless of how far you get, feedback is certainly welcome - part of the purpose of this post is to test my fit for doing more distilling work, so if you found this helpful (or unhelpful) I'd love to hear about it!
    Supervised Learning With Neural Networks (Section 1)
    To understand supervised learning, let’s start with a toy problem that’s often used to compare algorithms and make sure they work before applying them to more challenging domains. MNIST is a dataset composed of tens of thousands of hand-written digits from 0 to 9. Each piece of data co...
    37 min
  • EA - Roodman's Thoughts on Biological Anchors by lukeprog
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Roodman's Thoughts on Biological Anchors, published by lukeprog on September 14, 2022 on The Effective Altruism Forum.
    A new review of Ajeya Cotra's Forecasting TAI with biological anchors (see also update here), written by David Roodman in April 2020, has been added to the folder of public reviews for Cotra's report.
    Roodman's summary:
    I think my main critical reaction is about the draft report’s ecumenical approach. It puts non-zero weight on several different frameworks which, conditional on the various parameter choices favored in the report, contradict one another. This mixing of distributions expresses a kind of radical uncertainty: not only are we unsure about the parameter values within each framework; we’re also unsure about which framework is most right.
    This set-up is pragmatic and humble, but. I think in principle the ecumenism discards useful information, by not imposing the restriction that the various frameworks agree. In principle, they are all measuring the same thing. In pure Bayesian reasoning, if one has several uncertain measurements of the same value, each represented by a probability distribution, then one combines these primary measurements by multiplying them pointwise and rescaling the result to have total integral one. This contrasts with the pointwise averaging performed in the draft report, which is the mathematical expression of ecumenism.
    In Bayesian reasoning, if two distributions for the same parameter are normal, then their combination is too; its mean is the average of the two primary means, weighting by the respective precisions (inverse variances). Weirdly, if the two primary means are far apart, so that the two distributions hardly overlap, then their combination can pop up in the no-man’s-land between them. The intuition is that the combined distribution centers on the least unlikely estimate given what we know.
    I make that mathematical point less to argue for a mechanical implementation of Bayesian mixing of different perspectives than to advocate for an informal didactic that aims at unification. What is the least implausible way to reconcile the large disagreements between different frameworks? Could answers to that question help us settle on a single, favored framework, perhaps one that synthesizes ideas from more than one?
    That impulse ultimately led me to favor a single framework that fuses elements from several in the draft report. The idea is to model two training levels at once, of parameters and hyperparameters. Training of parameters corresponds to the training of a single neural network, or the learning a sentient organism undergoes during maturation. Hyperparameter training corresponds to the design space exploration that AI researchers engage in and, in the biological realm, to evolution. Each parameter training run may involve huge numbers of small parameter updates; each in turn serves a single hyperparameter training step.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The Defender’s Advantage of Interpretability by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Defender’s Advantage of Interpretability, published by Marius Hobbhahn on September 14, 2022 on LessWrong.
    TL;DR: In this post, I want to argue why Interpretability & Transparency tools have a defender’s advantage if they are used correctly, i.e. they improve alignment much more than new capabilities and therefore mitigate risks of dual use. I draw parallels from biosecurity researchers who have thought about the risks of dual-use and defender’s advantages in more detail and I think that the AI safety community can learn a lot from them. Lastly, I want to point out that not all interpretability tools have a clear defender’s advantage and some interpretability research might still carry a lot of risks when used incorrectly.
    I’d like to thank Lee Sharkey and Simon Grimm for their feedback on this post.
    Introduction
    Most technology is dual-use in some way--a knife can be used as a household appliance or as a weapon. However, different technologies have different propensities to be used for good or bad, e.g. more research into walls will likely benefit the defender more than the attacker while more research into the capabilities of viruses benefits attackers more than defenders.
    I feel like we, the AI safety community, have not thought enough about which approaches have a clear defender’s advantage or how we could steer existing approaches to have more of a defender's advantage. To my (very limited) understanding, the biosecurity community has thought a bit more about these kinds of dual-use trade-offs. Therefore, we could probably learn some things from them.
    In this post, I want to briefly look at some of the possible lessons from biosecurity and see if we can translate them to AI safety. Then I want to argue why interpretability is one of the approaches that plausibly has a defender’s advantage.
    I’m certainly not the first person to have come to the conclusion that interpretability is important for alignment. Chris Olah has made the case for interpretability for years. Neel Nanda has provided a long theory of impacts of interpretability research. Quintin Pope has made the case for optimism about interpretability. Evan Hubinger has provided 11 proposals to build safe AI that are all essentially something+interpretability, has developed an interpretability tech tree and summarized transformer circuits. ARC is working on ELK (and related topics) that certainly read to me as if they are intended to prevent deceptive alignment. There are many further good posts on aspects of interpretability (see e.g. here, here, here, or here).
    The reason why I add this post to the long list of posts arguing for the importance of interpretability is that I feel like the “defender’s advantage” framework allows for an easy way to decide which kind of interpretability research will help more with alignment than with capabilities and thus alleviates one major concern that some people have against it (personal conversations, not sure if someone wrote this down).
    Lessons from Biosecurity
    Most of the following comes from personal discussions with biosecurity researchers or podcasts like “Hear This Idea”s interview with Kevin Esvelt and Jonas Sandbrink. I’m not a biosecurity researcher myself and the following is likely to lack nuance.
    Gain-of-function(Enhancement of potential pandemic pathogens) = bad: More specifically, approaches that require us to build a new capability in order to learn how to safeguard against it makes offensive scaling easier than defensive scaling, e.g. the new capability enables the attacker to do more new things than the defender. Firstly, you have created a deadly virus with certainty but the development of the vaccine is uncertain--you stacked the odds against you. Secondly, even if your vaccine (or other defense) is successful, it’s likely easy to modify the new ...
    11 min
  • AF - When is intent alignment sufficient or necessary to reduce AGI conflict? by JesseClifton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When is intent alignment sufficient or necessary to reduce AGI conflict?, published by JesseClifton on September 14, 2022 on The AI Alignment Forum.
    In this post, we look at conditions under which Intent Alignment isn't Sufficient or Intent Alignment isn't Necessary for interventions on AGI systems to reduce the risks of (unendorsed) conflict to be effective. We then conclude this sequence by listing what we currently think are relatively promising directions for technical research and intervention to reduce AGI conflict.
    Intent alignment is not always sufficient to prevent unendorsed conflict
    In the previous post, we outlined possible causes of conflict and directions for intervening on those causes. Many of the causes of conflict seem like they would be addressed by successful AI alignment. For example: if AIs acquire conflict-prone preferences from their training data when we didn’t want them to, that is a clear case of misalignment. One of the suggested solutions: improving adversarial training and interpretability, just is alignment research, albeit directed at a specific type of misaligned behavior. We might naturally ask, does all work to reduce conflict risk follow this pattern? That is, is intent alignment sufficient to avoid unendorsed conflict?
    Intent Alignment isn't Sufficient is a claim about unendorsed conflict. We’re focusing on unendorsed conflict because we want to know whether technical interventions on AGIs to reduce the risks of conflict make a difference. These interventions mostly make sense for preventing conflict that isn’t desired by the overseers of the systems. (If the only conflict between AGIs is endorsed by their overseers, then conflict reduction is a problem of ensuring that AGI overseers aren’t motivated to start conflicts.)
    Let H be a human principal and A be its AGI agent. “Unendorsed” conflict, in our sense, is conflict which would not have been endorsed on reflection by H at the time A was deployed. This notion of “unendorsed” is a bit complicated. In particular, it doesn’t just mean “not endorsed by a human at the time the agent decided to engage in conflict”. We chose it because we think it should include the following cases:
    H makes a mistake in designing A such that conflict at a later time is rational by H’s lights. For example, H could’ve built A so that it can solve Prisoner’s Dilemmas using open-source game theory but failed to do so. Then A finds itself in a one-shot Prisoner’s Dilemma (as evaluated by H’s preferences), and defects. Even though H would approve of A’s decision to defect at the time, they would not endorse their own mistake of locking in a suboptimal architecture for A.
    H’s interactions with A lead to H’s preferences being corrupted, and then H approves of a decision they would not have approved of if their preferences hadn’t been corrupted.
    We’ll use Evan Hubinger’s decomposition of the alignment problem. In Evan’s decomposition, an AI is aligned with humans (i.e., doesn’t take any actions we would consider bad/problematic/dangerous/catastrophic) if it is intent-aligned and capability robust. (An agent is capability robust if it performs well by its own lights once it is deployed.) So the question for us is: What aspects of capability robustness determine whether unendorsed conflict occurs, and will these be present by default if intent alignment succeeds?
    Let’s decompose conflict-avoiding “capability robustness” into the capabilities necessary and sufficient for avoiding unendorsed conflict into two parts:
    Cooperative capabilities. An agent is cooperatively capable to the extent that it’s able to avoid conflict that is costly by its lights, when it is rational to do so.
    Understanding H’s cooperation-relevant preferences. This means that the agent understands H’s preferences sufficiently well that it would...
    18 min
  • AF - When would AGIs engage in conflict? by JesseClifton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When would AGIs engage in conflict?, published by JesseClifton on September 14, 2022 on The AI Alignment Forum.
    Here we will look at two of the claims introduced in the previous post: AGIs might not avoid conflict that is costly by their lights (Capabilities aren’t Sufficient) and conflict that is costly by our lights might not be costly by the AGIs’ (Conflict isn’t Costly).
    Explaining costly conflict
    First we’ll focus on conflict that is costly by the AGIs’ lights. We’ll define “costly conflict” as (ex post) inefficiency: There is an outcome that all of the agents involved in the interaction prefer to the one that obtains. This raises the inefficiency puzzle of war: Why would intelligent, rational actors behave in a way that leaves them all worse off than they could be?
    We’ll operationalize “rational and intelligent” actors as expected utility maximizers. We believe that the following taxonomy of the causes of inefficient outcomes between expected utility maximizers is exhaustive, except for a few implausible edge cases. (We give the full taxonomy, and an informal argument that it is exhaustive, in the appendix.) That is, expected utility maximization can lead to inefficient outcomes for the agents only if one of the following conditions (or an implausible edge case) holds. This taxonomy builds on Fearon’s (1995) influential “rationalist explanations for war”.
    Private information and incentives not to disclose. Here, “private information” means information about one’s willingness or ability to engage in conflict — e.g., how costly one considers conflict to be, or how strong one’s military is — about which other agents are uncertain. This uncertainty creates a risk-reward tradeoff: For example, Country A might think it’s sufficiently likely that Country B will give up without much of a fight that it’s worthwhile in expectation for A to fight B, even if they’ll end up fighting a war if they are wrong.
    In these cases, removing uncertainty — e.g., both sides learning exactly how willing the other is to fight — opens up peaceful equilibria. This is why conflict due to private information requires “incentives not to disclose”. Whether there are incentives to disclose will depend on a few factors.
    First, the technical feasibility of different kinds of verifiable disclosure matters. For example, if I have an explicitly-specified utility function, how hard is it for me to prove to you how much my utility function values conflict relative to peace?
    Second, different games will create different incentives for disclosure. Sometimes the mere possibility of verifiable disclosure ends up incentivizing all agents to disclose all of their private information (Grossman 1981; Milgrom 1981). But in other cases, more sophisticated disclosure schemes are needed. For example: Suppose that an agent has some vulnerability such that unconditionally disclosing all of their private information would place them at a decisive disadvantage. The agents could then make copies of themselves, allow these copies to inspect one another, and transmit back to the agents only the private information that’s necessary to reach a bargain. (See the appendix and DiGiovanni and Clifton (2022) for more discussion of conditional information revelation and other conditions for the rational disclosure of conflict-relevant private information.)
    For the rest of the sequence, we’ll use “informational problem” as shorthand for this mechanism of conflict.
    Inability to credibly commit to peaceful settlement. Agents might fight even though they would like to be able to commit to peace. The Prisoner’s Dilemma is the classic example: Both prisoners would like to be able to write a binding contract to cooperate, but if they can’t, then the game-theoretically rational thing to do is defect.
    Similarly sometimes one agent will ...
    28 min
  • EA - Does beating yourself up about being unproductive accomplish anything? What should I ask self-compassion researcher Kristin Neff when I interview her? by Robert Wiblin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does beating yourself up about being unproductive accomplish anything? What should I ask self-compassion researcher Kristin Neff when I interview her?, published by Robert Wiblin on September 14, 2022 on The Effective Altruism Forum.
    Next week for The 80,000 Hours Podcast I'm interviewing Kristin Neff — an academic psychologist who has pioneered research into, and is strongly associated with, the idea of 'self-compassion'.
    We will discuss how much guilt/shame/negative self-talk help get things done, and if not much, how to reduce them.
    While many people believe that feeling bad about themselves when they're unproductive helps them achieve more, and that without this behaviour they'd become lazy and self-indulgent, Kristin argues that the evidence shows these behaviours to be bad for long-term productivity.
    Just as a good line manager doesn't denigrate a staff member who is struggling, and a good parent won't belittle a child who is procrastinating, we ought not treat ourselves so harshly. We should speak to ourselves like a good manager or friend or parent would: with kindness and understanding as well as a firm commitment to what's in our own long-term best interest.
    If Kristin is right, 'beating yourself up' is not only harmful to your own well-being, in typical cases it accomplishes very little.
    What should I ask her?
    Places to learn more about Kristen's work:
    A good interview she did with Spencer Greenberg on the Clearer Thinking Podcast that addresses mental health issues common in the EA community
    Wikipedia
    Her website
    Her book Self Compassion: The Proven Power of Being Kind to Yourself
    Dr. Kristin Neff | The Science of Self-Compassion | Talks at Google
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Some ideas for epistles to the AI ethicists by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some ideas for epistles to the AI ethicists, published by Charlie Steiner on September 14, 2022 on The AI Alignment Forum.
    Some papers, or ideas for papers, that I'd loved to see published in ethics journals like Minds and Machines or Ethics and Information Technology. I'm probably going to submit one of these to a 2023 AI ethics conference myself.
    Why should we do this? Because we want today's grad students to see that the ethical problems of superhuman AI are a cool topic that they can publish a cool paper about. And we want to (marginally) raise the waterline for thinking about future AI, nudging the AI ethics discourse towards more matured views of the challenges of AI.
    Secondarily, it would be good to leverage the existing skillsets of some ethicists for AI safety work, particularly those already working on AI governance. And having an academic forum where talking about AI safety is normalized bolsters other efforts to work on AI safety in academia.
    The Ideas:
    Explain the basic ideas of AI safety, and why to take them seriously.
    Iason Gabriel already had a pretty good paper like this. But it's plausible that, right now, what the ethics discourse needs is more basic explanations of why AI safety is a thing at all.
    This paper might start out by making a case that superhuman AI is going to change the world, likely in the next 10-60 years (definitely unintuitive to many, but there are AI Impacts surveys and recent results to illustrate the point). Then the basic arguments that superhuman AI will not be automatically benevolent (easy rhetorical trick is to call it "superhuman technology," everyone knows technology is bad). Then the basic arguments that to get things to go well, the AI has to know a whole lot about what humans want (and use that knowledge the way we want).
    One issue with this might be that it presents the problem, but doesn't really point people towards solutions (this may be a problem that can be solved with quick citations). It also doesn't really motivate why this is an ethics problem. It also doesn't explain why we want the solution to the key "ethics-genre" problems to use a technical understanding of the AI, rather than a human- or society-centric view.
    A more specific defense of the validity of transformative-AI-focused thinking as a valid use of ethicists' time.
    The core claim is that getting AIs to want want good things and not bad things is an unsolved ethics problem. Ethics, not engineering, because the question isn't "how do we implement some obvious standard," the question is "what is even a good standard in the first place?"
    But almost as important are secondary claims about what actual progress on this question looks like. The end goal is a standard that is connected to technical picture of how the AI will learn this information about humans, and how it will use it to make decisions.
    So the overall thrust is "given that AI safety is important, there is a specific sort of ethics-genre reasoning that is going to be useful, and here are some gestures towards what it might look like."
    You can put more than one of these ideas into a paper if you want. This particular idea feels to me like it could benefit from being paired with another topic before or after it.
    Dunking on specific mistakes, like talking about "robots" rather than "optimization processes," should probably be done with care and tact.
    A worked example of "dual use" ethics - a connection between thinking about present-day problems and superhuman AI.
    I expect most of the examples to be problems that sound relevant to the modern day, but that sneakily contain most of the alignment problem.
    E.g. Xuan's AI that takes actions in response to laws that we really want to follow the spirit of the law. Although that's a bit too futuristic, actually, because we don't have much present-day ...
    7 min
  • LW - Dan Luu on Futurist Predictions by RobertM
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dan Luu on Futurist Predictions, published by RobertM on September 14, 2022 on LessWrong.
    Epistemic status: perspective derived from following Dan Luu's output for the last 5 years or so. Trying to vaguely gesture at a few things at once. Please ask questions if you find something confusing.
    Dan Luu has written a interesting post analysing the track record of futurists' predictions. The motivation:
    I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.
    The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists1) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibility of predictions that use similar methodologies.
    I'm primarily going to address the appendix, particularly the section on Holden Karnofsky's analysis on the same subject, but the article is interesting reading and I'd recommend going through the whole thing. (I think Dan is evaluating forecasting track records pretty differently from how I would, and I haven't actually dug into any of the other analysis. On priors I'd expect it to be similar to his analysis of Holden's work.)
    Karnofsky's evaluation of Kurzweil being "fine" to "mediocre" relies on these two analyses done on LessWrong and then uses a very generous interpretation of the results to conclude that Kurzweil's predictions are fine. Those two posts rate predictions as true, weakly true, cannot decide, weakly false, or false. Karnofsky then compares the number of true + weakly true to false + weakly false, which is one level of rounding up to get an optimistic result; another way to look at it is that any level other than "true" is false when read as written. This issue is magnified if you actually look at the data and methodology used in the LW analyses.
    The specific claim I have an issue with here is "another way to look at it is that any level other than "true" is false when read as written". Depending on how you want to evaluate it it, it's either technically true but irrelevant, or not even wrong.
    In the second post, the author, Stuart Armstrong indirectly noted that there were actually no predictions that were, by strong consensus, very true when he noted that the "most true" prediction had a mean score of 1.3 (1 = true, 2 = weakly true ... , 5 = false) and the second highest rated prediction had a mean score of 1.4. Although Armstrong doesn't note this in the post, if you look at the data, you'll see that the third "most true" prediction had a mean score of 1.45 and the fourth had a mean score of 1.6, i.e., if you round to the nearest prediction score, only 3 out of 105 predictions score "true" and 32 are >= 4.5 and score "false". Karnofsky reads Armstrong's as scoring 12% of predictions true, but the post effectively makes no comment on what fraction of predictions were scored true and the 12% came from summing up the total number of each rating given.
    I'm not going to say that taking the mean of each question is the only way one could aggregate the numbers (taking the median or modal values could also be argued for, as well as some more sophisticated scoring function, an extremizing function, etc.), but summing up ...
    9 min
  • EA - Announcing an Empirical AI Safety Program by Josh1
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing an Empirical AI Safety Program, published by Josh1 on September 13, 2022 on The Effective Altruism Forum.
    Last summer, the Center for AI Safety ran the ML Safety Scholars program (MLSS) to help students interested in AI Safety gain technical skills and learn about empirical safety topics. This fall, we are running an abridged version of this program called Introduction to ML Safety.
    Apply to be a participant by September 21st.Apply to be a facilitator by September 18th.Website (for advertising): mlsafety.org/intro-to-ml-safety
    About the Course
    Introduction to ML Safety is an 8-week course that aims to introduce students with a deep learning background to empirical AI Safety research. The program is designed and taught by Dan Hendrycks, a UC Berkeley ML PhD and director of the Center for AI Safety, and provides an introduction to robustness, alignment, monitoring, systemic safety, and conceptual foundations for existential risk.
    Each week, students will be assigned readings, lecture videos, and required homework assignments. The materials are publicly available at course.mlsafety.org.
    There are two tracks:
    Introductory track: for people who are new to AI Safety. This track aims to familiarize students with the AI X-risk discussion alongside empirical research directions.
    Advanced track: for people who already have a conceptual understanding of AI X-risk and want to learn more about existing empirical safety research so they can start contributing.
    Introduction to ML Safety will be virtual by default, though students can participate in person if a section is facilitated at their local university.
    Time Commitment
    The program will last for 8 weeks, beginning on September 26th and ending on November 18th. Participants are expected to commit at least 5 hours per week. This includes ~1 hour of recorded lectures (which will take more than one hour to digest), ~1-2 hours of readings, ~1-2 hours of written assignments, and 1 hour of discussion.
    We understand that 5 hours is a large time commitment, so to make our program more inclusive and remove any financial barriers, we will provide a $1000 stipend upon completion of the course. For logistical reasons, we can only pay participants who are eligible to work in the US. Non-US residents are welcome to apply and participate but we cannot offer them a stipend. We will make an effort to provide international students in the US with stipends but we cannot guarantee that we will be able to.
    Eligibility
    Anyone can apply as long as they have the following prerequisites:
    Deep learning: you can gauge the background knowledge required by skimming the week 1 slides: deep learning review.
    Linear algebra or introductory statistics (e.g., AP Statistics)
    Multivariate differential calculus
    If you are not sure whether you meet these criteria, err on the side of applying.
    Facilitating a section
    To be a facilitator, you must have a strong background in deep learning and AI Safety. For the latter, taking AGISF (AGI Safety Fundamentals) or participating in a similar program is sufficient. Much of the course content is not covered in AGISF, so if you are not familiar with it, you will have to learn it in advance each week.
    The time commitment for running one cohort is ~2 hours per week depending on prior familiarity with the material. 1 hour of discussion and 1 hour of prep (discussion times are flexible).
    We will pay facilitators a stipend corresponding to ~$30 per hour (subject to legal constraints).
    You can apply to be a facilitator here (applications are due by September 18th).
    Questions
    Questions about the program should be posted as comments on this post. For questions that are only relevant to you, please reach out to [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please ...
    4 min