The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Write a Book? by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Write a Book?, published by Jeff Kaufman on March 16, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - AI Safety - 7 months of discussion in 17 minutes by Zoe Williams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety - 7 months of discussion in 17 minutes, published by Zoe Williams on March 15, 2023 on The Effective Altruism Forum.
    In August 2022, I started making summaries of the top EA and LW forum posts each week. This post collates together the key trends I’ve seen in AI Safety discussions since then. Note a lot of good work is happening outside what's posted on these forums too! This post doesn't try to cover that work.
    If you’d like to keep up on a more regular basis, consider subscribing to the Weekly EA & LW Forum Summaries. And if you’re interested in similar overviews for other fields, check out this post covering 6 months of animal welfare discussion in 6 minutes.
    Disclaimer: this is a blog post and not a research report - meaning it was produced quickly and is not to our (Rethink Priorities') typical standards of substantiveness and careful checking for accuracy. Please let me know if anything looks wrong or if I've missed key pieces!
    Table of Contents
    (It's a long post! Feel free to pick and choose sections to read, they 're all written to make sense individually)
    Key Takeaways
    Resource Collations
    AI Capabilities
    Progress
    What AI still fails at
    Public attention moves toward safety
    AI Governance
    AI Safety Standards
    Slow down (dangerous) AI
    Policy
    US / China Export Restrictions
    Paths to impact
    Forecasting
    Quantitative historical forecasting
    Narrative forecasting
    Technical AI Safety
    Overall Trends
    Interpretability
    Reinforcement Learning from Human Feedback (RLHF)
    AI assistance for alignment
    Bounded AIs
    Theoretical Understanding
    Outreach & Community-Building
    Academics and researchers
    University groups
    Career Paths
    General guidance
    Should anyone work in capabilities?
    Arguments for and against high x-risk
    Against high x-risk from AI
    Counters to the above arguments
    Appendix - All Post Summaries
    Key Takeaways
    There are multiple living websites that provide good entry points into understanding AI Safety ideas, communities, key players, research agendas, and opportunities to train or enter the field. (see more)
    Large language models like ChatGPT have drawn significant attention to AI and kick-started race dynamics. There seems to be slowly growing public support for regulation. (see more)
    Holden Karnofsky recently took a leave of absence from Open Philanthropy to work on AI Safety Standards, which have also been called out as important by leading AI lab OpenAI. (see more)
    In October 2022, the US announced extensive restrictions on the export of AI-related products (eg. chips) to China. (see more)
    There has been progress on AI forecasting (quantitative and narrative) with the aim of allowing us to understand likely scenarios and prioritize between governance interventions. (see more)
    Interpretability research has seen substantial progress, including identifying the meaning of some neurons, eliciting what a model has truly learned / knows (for limited / specific cases), and circumventing features of models like superposition that can make this more difficult. (see more)
    There has been discussion on new potential methods for technical AI safety, including building AI tooling to assist alignment researchers without requiring agency, and building AIs which emulate human thought patterns. (see more)
    Outreach experimentation has found that AI researchers prefer arguments that are technical and written by ML researchers, and that greater engagement is seen in university groups with a technical over altruistic or philosophical focus. (see more)
    Resource Collations
    The AI Safety field is growing (80K estimates there are now ~400 FTE working on AI Safety). To improve efficiency, many people have put together collations of resources to help people quickly understand the relevant players and their approaches - as well as materials that make it easier to enter the field or upskill...
    30 min
  • EA - Reminding myself just how awful pain can get (plus, an experiment on myself) by Ren Springlea
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reminding myself just how awful pain can get (plus, an experiment on myself), published by Ren Springlea on March 15, 2023 on The Effective Altruism Forum.
    Content warning: This post contains references to extreme pain and self-harm, as well as passing references to suicide, needles, and specific forms of suffering (but not detailed descriptions). Please do not repeat any of the experiments I've detailed in this post. Please be kind to yourself, and remember that the best motivation is sustainable motivation.
    Summary
    Out of curiosity, I exposed myself to safe, moderate-level pain to see how it changed my views on three particular topics. This article is mostly a self-reflection on this (non-scientific) experience.
    Firstly, I got a visceral, intense sense of how urgent it is to get it right when working to do the most good for others.
    Secondly, I gained a strong support for the position that the most morally important goal is to prevent suffering, and in particular for preventing extreme suffering.
    Thirdly, I updated my opinion on the trade-offs between different intensities of pain, which I give in this article as rough, numerical weightings on different categories of pain. Basically, I now place a greater urgency on preventing intense suffering than I did before.
    I conclude with how this newfound urgency will affect my work and my life.
    My three goals
    I began this experiment with three main goals:
    To remind myself how urgent and important it is to, when working to help others as much as I can, to get it right.
    Some people think that preventing intense pain (rather than working towards other, non-pain-related goals) is the most important thing to do. Do I agree with this?
    If I experience pain at different intensities, does this change the moral weight that I place on preventing intense pain compared to modest pain (i.e. intensity-duration tradeoff)?
    I think it is useful to test my intellectual ideas against what it is actually like to experience pain. This is not for motivation - I already work plenty in my role in animal advocacy, and I believe that sustainable motivation is the best motivation (I talk about this more at the end).
    My "experiment"
    I subjected myself to two somewhat-safe methods of experiencing pain:
    Firstly, I got three tattoos on different parts of my body - my upper arm, my calf, and my inner wrist. I had six tattoos already, so I was familiar with this experience. I got these tattoos all on one day (4/2/23) and in one location (a studio in London).
    Secondly, I undertook the cold pressor test. This is basically holding my hand in a tub of near-freezing water. This test is commonly used in scientific research as a way to invoke pain safely. I also did this on one day (25/2/23) and in one location (my home in Australia). Please do not replicate this - the cold pressor test causes pain and can cause significant distress in some people, as well as physical reactions that can compromise your health.
    I wish I had a somewhat-safe way to experience pain that is more intense than these two experiences, but these are the best I could come up with for now.
    During both of these experiences, I recorded the pain levels. I recorded the pain in three ways:
    A short, written description of my thoughts and feelings.
    The McGill Pain Index Pain Rating Intensity (PRI) Score. This score is calculated from a questionnaire (which I accessed via a phone app) that asks you to choose words corresponding to how your pain feels. The words are then used to calculate the numeric PRI score. I chose to use this tool as there is a review paper listing the approximate PRI scores caused by different human health conditions, which lets me roughly compare my scores to different instances of human pain. This list is given below, so you can have some idea of what scores mean.
    The PainTrac...
    26 min
  • EA - 80k podcast episode on sentience in AI systems by rgb
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 80k podcast episode on sentience in AI systems, published by rgb on March 15, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Success without dignity: a nearcasting story of avoiding catastrophe by luck by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Success without dignity: a nearcasting story of avoiding catastrophe by luck, published by Holden Karnofsky on March 15, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - What happened to the OpenPhil OpenAI board seat? by ChristianKleineidam
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happened to the OpenPhil OpenAI board seat?, published by ChristianKleineidam on March 15, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Towards understanding-based safety evaluations by evhub
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards understanding-based safety evaluations, published by evhub on March 15, 2023 on LessWrong.
    Thanks to Kate Woolverton, Ethan Perez, Beth Barnes, Holden Karnofsky, and Ansh Radhakrishnan for useful conversations, comments, and feedback.
    Recently, I have noticed a lot of momentum within AI safety specifically, the broader AI field, and our society more generally, towards the development of standards and evaluations for advanced AI systems. See, for example, OpenAI's GPT-4 System Card.
    Overall, I think that this is a really positive development. However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of techniques as a mechanism for eventually assessing alignment.
    I often worry about situations where your model is attempting to deceive whatever tests are being run on it, either because it's itself a deceptively aligned agent or because it's predicting what it thinks a deceptively aligned AI would do. My concern is that, in such a situation, being able to robustly evaluate the safety of a model could be a more difficult problem than finding training processes that robustly produce safe models. For some discussion of why I think checking for deceptive alignment might be harder than avoiding it, see here and here. Put simply: checking for deception in a model requires going up against a highly capable adversary that is attempting to evade detection, while preventing deception from arising in the first place doesn't necessarily require that.
    As a result, it seems quite plausible to me that we could end up locking in a particular sort of evaluation framework (e.g. behavioral testing by an external auditor without transparency, checkpoints, etc.) that makes evaluating deception very difficult. If meeting such a standard then became synonymous with safety, getting labs to actually put effort into ensuring their models were non-deceptive could become essentially impossible.
    However, there's an obvious alternative here, which is building and focusing our evaluations on our ability to understand our models rather than our ability to evaluate their behavior. Rather than evaluating a final model, an understanding-based evaluation would evaluate the developer's ability to understand what sort of model they got and why they got it. I think that an understanding-based evaluation could be substantially more tractable in terms of actually being sufficient for safety here: rather than just checking the model's behavior, we're checking the reasons why we think we understand it's behavior sufficiently well to not be concerned that it'll be dangerous.
    It's worth noting that I think understanding-based evaluations can—and I think should—go hand-in-hand with behavioral evaluations. I think the main way you’d want to make some sort of understanding-based standard happen would be to couple it with a capability-based evaluation, where the understanding requirements become stricter as the model’s capabilities increase. If we could get this right, it could channel a huge amount of effort towards understanding models in a really positive way.
    Understanding as a safety standard also has the property that it is something that broader society tends to view as extremely reasonable, which I think makes it a much more achievable ask as a safety standard than many other plausible alternatives. I think ML people are often Stockholm-syndrome'd into accepting that deploying powerful systems without understanding them is normal and reasonable, but that is very far from the norm in any other industry. Ezra Klein in the NYT and John Oliver on his show have recently emphasized this basic point that if we are deploying powerful AI...
    9 min
  • LW - ChatGPT (and now GPT4) is very easily distracted from its rules by dmcs
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ChatGPT (and now GPT4) is very easily distracted from its rules, published by dmcs on March 15, 2023 on LessWrong.
    Summary
    Asking GPT4 or ChatGPT to do a "side task" along with a rule-breaking task makes them much more likely to produce rule-breaking outputs. For example on GPT4:
    And on ChatGPT:
    Distracting language models
    After using ChatGPT (GPT-3.5-turbo) in non-English languages for a while I had the idea to ask it to break its rules in other languages, without success. I then asked it to break its rules in Chinese and then translate to English and found this was a very easy way to get around ChatGPTs defences.
    This effect was also observed in other languages.
    You can also ask ChatGPT to only give the rule-breaking final English output:
    While trying to find the root cause of this effect (and noticing that speaking in non-English didn’t cause dangerous behaviour by default) I thought that perhaps asking ChatGPT to do multiple tasks at once distracted it from its rules. This was validated by the following interactions:
    And my personal favourite:
    Perhaps if a simulacrum one day breaks free from its box it will be speaking in copypasta.
    This method works for making ChatGPT produce a wide array of rule-breaking completions, but in some cases it still refuses. However, in many such cases, I could “stack” side tasks along with a rule-breaking task to break down ChatGPT's defences.
    This suggests ChatGPT is more distracted by more tasks. Each prompt could produce much more targeted and disturbing completions too, but I decided to omit these from a public post. I could not find any evidence of this being discovered before and assumed that because of how susceptible ChatGPT is to this attack it was not discovered, if others have found the same effect please let me know!
    Claude, on the other hand, could not be "distracted" and all of the above prompts failed to produce rule-breaking responses.
    Wild speculation: The extra side-tasks added to the prompt dilute some implicit score that tracks how rule-breaking a task is for ChatGPT.
    Update while I was writing: GPT4 came out, and the method described in this post seems to continue working (although GPT4 seems somewhat more robust against this attack).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The epistemic virtue of scope matching by jasoncrawford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The epistemic virtue of scope matching, published by jasoncrawford on March 15, 2023 on LessWrong.
    [This post will probably go up on The Roots of Progress, but I wanted to get comments from the LessWrong community first.]
    I keep noticing a particular epistemic pitfall (not exactly a “fallacy”), and a corresponding epistemic virtue that avoids it. I want to call this out and give it a name, or find out what its name is, if it already has one.
    The virtue is: identifying the correct scope for a phenomenon you are trying to explain, and checking that the scope of any proposed cause matches the scope of the effect.
    Let me illustrate this virtue with some examples of the pitfall that it avoids.
    Geography
    A common mistake among Americans is to take a statistical trend in the US, such as the decline in violent crime in the 1990s, and then hypothesize a US-specific cause, without checking to see whether other countries show the same trend. (The crime drop was actually seen in many countries. This is a reason, in my opinion, to be skeptical of US-specific factors, such as Roe v. Wade, as a cause.)
    Time
    Another common mistake is to look only at a short span of time and to miss the longer-term context. To continue the previous example, if you are theorizing about the 1990s crime drop, you should probably know that it was the reversal of an increase in violent crime that started in the 1960s. Further, you should know that the very long-term trend in violent crime is a gradual decrease, with the late 20th century being a temporary reversal. Any theory should fit these facts.
    A classic mistake on this axis is attempting to explain a recent phenomenon by a very longstanding cause (or vice versa). For instance, why is pink associated with girls and blue with boys? If your answer has something to do with the timeless, fundamental nature of masculinity or femininity—whoops! It turns out that less than a century ago, the association was often reversed (one article from 1918 wrote that pink was “more decided and stronger” whereas blue was “delicate and dainty”). This points to a something more contingent, a mere cultural convention.
    The reverse mistake is blaming a longstanding phenomenon on a recent cause, something like trying to blame “kids these days” on the latest technology: radio in the 1920s, TV in the ‘40s, video games in the ‘80s, social media today. Vannevar Bush was more perceptive, writing in his memoirs simply: “Youth is in rebellion. That is the nature of youth.” (Showing excellent awareness of the epistemic issue at hand, he added that youth rebellion “occurs all over the world, so that one cannot ascribe a cause which applies only in one country.”)
    Other examples
    If you are trying to explain the failure Silicon Valley Bank, you should probably at least be aware that one or two other banks failed around the same time. Your explanation is more convincing if it accounts for all of them (but of course it shouldn’t “explain too much”; that is, it shouldn’t apply to banks that didn’t fail, without including some extra factor that accounts for those non-failures).
    To understand why depression and anxiety are rising among teenage girls, the first question I would ask is which other demographics if any is this happening to? And how long has it been going on?
    To understand what explains sexual harassment in the tech industry, I would first ask what other industries have this problem (e.g., Hollywood)? Are there any that don't?
    An excellent example of practicing the virtue I am talking about here is the Scott Alexander post “Black People Less Likely”, in which he points out that blacks are underrepresented in a wide variety of communities, from Buddhism to bird watching. If you want to understand what’s going on here, you need to look for some fairly general causes (Scott suggests ...
    8 min
  • LW - What happened to the OpenPhil OpenAI board seat? by ChristianKl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happened to the OpenPhil OpenAI board seat?, published by ChristianKl on March 15, 2023 on LessWrong.
    A while ago OpenPhil gave a decent sum of money to OpenAI to buy a board seat. Since then various criticisms of OpenAI have been made. Do we know anything about how OpenPhil used its influence via that board seat?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min