The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - A Double Feature on The Extropians by Maxwell Tabarrok
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Double Feature on The Extropians, published by Maxwell Tabarrok on June 4, 2023 on The Effective Altruism Forum.
    Link-post for two pieces I just wrote on the Extropians.The Extropians were an online group of techno-optimist transhumanist libertarians active in the 90s who influence a lot of online intellectual culture today, especially in EA and Rationalism. Prominent members include Eliezer Yudkowsky, Nick Bostrom, Robin Hanson, Eric Drexler, Marvin Minsky and all three of the likely candidates for Satoshi Nakamoto (Hal Finney, Wei Dai, and Nick Szabo).The first piece is a deep dive into the archived Extropian forum. It was super fun to write and I was constantly surprised about how much of the modern discourse on AI and existential risk had already been covered in 1996.
    The second piece is a retrospective on predictions made by Extropians in 1995. Eric Drexler, Nick Szabo and 5 other Extropians give their best estimates for when we'll have indefinite biological lifespans and reproducing asteroid eaters.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • EA - Podcast Interview with David Thorstad on Existential Risk, The Time of Perils, and Billionaire Philanthropy by Nick Anyos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Podcast Interview with David Thorstad on Existential Risk, The Time of Perils, and Billionaire Philanthropy, published by Nick Anyos on June 4, 2023 on The Effective Altruism Forum.
    I have released a new episode of my podcast, EA Critiques, where I interview David Thorstad. David is a researcher at the Global Priorities Institute and also writes about EA on his blog, Reflective Altruism.
    In the interview we discuss three of his blog post series:
    Existential risk pessimism and the time of perils: Based on his academic paper of the same name, David argues that there is a surprising tension between the idea that there is a high probability of extinction (existential risk pessimism) and the idea that the expected value of the future, conditional on no existential catastrophe this century, is astronomically large.
    Exaggerating the risks: David argues that the probability of an existential catastrophe from any source is much lower than many EAs believe. At time of recording the series only covered risks from climate change, but future posts will make the same case for nuclear war, pandemics, and AI.
    Billionaire philanthropy: Finally, we talk about the the potential issues with billionaires using philanthropy to have an outsized influence, and how both democratic societies and the EA movement should respond.
    As always, I would love feedback, on this episode or the podcast in general, and guest suggestions. You can write a comment here, send me a message, or use this anonymous feedback form.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • AF - How to Think About Activation Patching by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Think About Activation Patching, published by Neel Nanda on June 4, 2023 on The AI Alignment Forum.
    This is an excerpt from my post on attribution patching, that I think is of more general interest, around how to think about the technique of activation patching in mechanistic interpretability, and what it can and cannot teach us. You don't need to know what attribution patching is to read this, check out this section for an overview of activation patching
    What Is the Point of Patching?
    The conceptual framework I use when thinking about patching is to think of a model as an enormous mass of different circuits. On any given input, many circuits will be used. Further, while some circuits are likely very common and important (eg words after full stops start with capitals, or "this text is in English"), likely many are very rare and niche and will not matter on the vast majority of inputs (eg "this is token X in the Etsy terms and conditions footer" - a real thing that GPT-2 Small has neurons for!). For example, on IOI, beyond the actual IOI circuit, there is likely circuits to:
    Detect multi-token words (eg names) and assemble them into a single concept
    Detect that this is English text
    Detect that this is natural language text
    Encode unigram frequencies (under which John is 2-3x as likely as Mary!)
    Copying all names present in the context - if a name is present in the text it's way more likely to occur next than an unseen name!
    That this is the second clause of a sentence
    That Indirect Object Identification is the right grammatical structure to use
    That the next word should be a name
    As a mech interp researcher, this is really annoying! I can get traction on a circuit in isolation, and there's a range of tools with ablations, direct logit attribution, etc to unpick what's going on. And hopefully any given circuit will be clean and sparse, such that I can ignore most of a model's parameters and most activation dimensions, and focus on what's actually going on. But when any given input triggers many different circuits, it's really hard to know what's happening.
    The core point of patching is to solve this. In IOI, most of the circuits will fire the same way regardless of which name is the indirect object/repeated subject. So by formulating a clean and corrupted input that are as close as possible except for the key detail of this name, we can control for as many of the shared circuits as possible. Then, by patching in activations from one run to another, we will not affect the many shared circuits, but will let us isolate out the circuit we care about. Taking the logit difference (ie difference between the log prob of the correct and incorrect answer) also helps achieve this, by controlling for the circuits that decide whether to output a name at all.
    Importantly, patching should be robust to some of our conceptual frameworks being wrong, eg a model not having a linear representation, circuits mattering according to paths that go through every single layer, etc. Though it's much less informative when a circuit is diffuse across many heads and neurons than when it's sparse.
    Whenever reasoning about patching, it's valuable to keep this in mind! And to be clear to yourself about what exactly is the scope of your investigation - which circuits do you want to identify, how well have you controlled for the irrelevant circuits, what aspects of the circuits you do care about have you accidentally controlled for, etc. And even if you want to identify several circuits, it's generally best to try to have several types of clean and corrupted inputs, so that you can isolate out each circuit one at a time (eg in the Python docstring circuit).
    Patching Clean to Corrupted vs Corrupted to Clean
    One surprisingly subtle detail is whether you patch a clean activation into the corrupted r...
    35 min
  • LW - A Double-Feature on The Extropians by Maxwell Tabarrok
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Double-Feature on The Extropians, published by Maxwell Tabarrok on June 3, 2023 on LessWrong.
    Link-post for two pieces I just wrote on the Extropians.The Extropians were an online group of techno-optimist transhumanist libertarians active in the 90s who influence a lot of online intellectual culture today. Prominent members include Eliezer Yudkowsky, Nick Bostrom, Robin Hanson, Eric Drexler, Marvin Minsky and all three of the likely candidates for Satoshi Nakamoto (Hal Finney, Wei Dai, and Nick Szabo).The first piece is a deep dive into the archived Extropian forum. It was super fun to write and I was constantly surprised about how much of the modern discourse on AI and existential risk had already been covered in 1996.
    The second piece is a retrospective on predictions made by Extropians in 1995. Eric Drexler, Nick Szabo and 5 other Extropians give their best estimates for when we'll have indefinite biological lifespans and reproducing asteroid eaters.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - The AGI Race Between the US and China Doesn’t Exist. by Eva B
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The AGI Race Between the US and China Doesn’t Exist., published by Eva B on June 3, 2023 on LessWrong.
    When I write “China”, I refer to the political and economic entity, the People’s Republic of China, founded in 1949.
    Leading US AI companies are currently rushing towards developing artificial general intelligence (AGI) by building and training increasingly powerful large language models (LLMs), as well as building architecture on top of them. This is frequently framed in terms of a great power competition for technological supremacy between the US and China.
    However, China has neither the resources nor any interest in competing with the US in developing artificial general intelligence (AGI) primarily via scaling Large Language Models (LLMs).
    In brief, China does not compete with the US in developing AGI via LLMs because of:
    Resources and technology: China does not have access to the computational resources (compute, here specifically data centre-grade GPUs) needed for large-scale training runs of large language models. This gap will only widen over time; China is failing to develop a domestic semiconductor industry, despite massive efforts to do so, and is increasingly cut off from international semiconductor supply chains.
    Political goals: The Chinese Communist Party (CCP) is above all concerned with ensuring social stability and staying in power. It will not tolerate large language models that cannot reliably be controlled and prevented from leaking sensitive information, raising politically sensitive topics, or challenging the regime. Reliably prohibiting LLMs from doing any of these things is an unsolved technical problem. This has and will continue to stifle the development and deployment of Chinese LLMs for the Chinese market, as a single political faux pas will spell doom for the companies involved.
    Therefore, as it currently stands, there is no AGI race between China and the US.
    China Cannot Compete with the US to Reach AGI First Via Scaling LLMs
    Today, training powerful LLMs like GPT-4 requires massive GPU clusters. And training LLMs on a larger number of GPUs is the most common and reliable way of increasing their capabilities. This is known as the bitter lesson in machine learning; increasing the compute a model is trained on leads to higher progress in the model's capabilities than any changes in the algorithm's architecture. So if China wants to “win the AGI race” against the US, it would need to maximise its domestic GPU capacities. However, China lacks domestic semiconductor production facilities for producing data centre-grade GPUs. So to increase domestic GPU capacities, it has to import GPUs or semiconductor manufacturing equipment (SME).
    However, since the US introduced its export controls on advanced chips and SMEs in October 2022, China can no longer purchase GPUs or SMEs abroad. And with the semiconductor industry advancing at lightning speed, China is falling further and further behind every day. These policies are just the most recent ones in a row of ongoing measures by the US restricting the transfer of semiconductor technology to Chinese companies, beginning with Huawei in 2019. In early 2023, the Netherlands and Japan, which play a key role in the global semiconductor manufacturing chain, joined the US and introduced SME export controls targeting China. Crucially, these combined export controls apply to ASML’s most advanced lithography machines, the only machines in the world which can produce cutting-edge chips.
    The problem for China lies not so much in designing cutting-edge GPUs, but in designing and building SMEs to produce the advanced GPUs needed for training LLMs, a far more challenging task. But China has been trying - and failing - to build domestic semiconductor manufacturing chains for cutting-edge chips since at least the 199...
    15 min
  • AF - Unfaithful Explanations in Chain-of-Thought Prompting by miles
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unfaithful Explanations in Chain-of-Thought Prompting, published by miles on June 3, 2023 on The AI Alignment Forum.
    Introduction
    I recently released “Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” with collaborators Julian Michael, Ethan Perez, and Sam Bowman. For a summary of the paper, you can check out this twitter thread. In this post, I briefly elaborate on motivations/implications relevant to alignment and, most importantly, give some examples of future work that might address these problems (See Future Work).
    TLDR:
    CoT doesn’t give us transparency because the process producing the explanation is still black-box. Our results demonstrate that in practice, it’s very easy to find instances of clearly unfaithful CoT explanations: models can confabulate plausible explanations for predictions made for different reasons than they state.
    To improve faithfulness, we either need (1) more context-independent reasoning, or (2) improvements to self-modeling and truthfulness. I’m more excited about (1), which would include methods like problem decomposition and explanation-consistency training.
    I don’t think these results fully condemn CoT given that we haven’t tried very hard to explicitly encourage faithfulness. I’m uncertain about how promising CoT is as a starting point for explainability, but there seem to be enough tractable directions for future work that it merits investigation.
    Externalized Reasoning
    This work fits into alignment through the Externalized Reasoning agenda, which Tamera Lanham did a good job of sketching out here: Externalized reasoning oversight: a research direction for language model alignment. The gist of this approach is to try to get models to do as much processing/reasoning through natural language as possible. As long as these reasoning traces accurately describe the process the model uses to give answers, then we might more easily detect undesirable behavior by simply monitoring the externalized reasoning. If mechanistic interpretability turns out to be very difficult, this could be one alternative that might help us sidestep those problems.
    Framed in terms of the explainability literature, we want explanations that are not only plausible (convincing to a human) but also faithful (accurately describe the process models use to give some prediction) [1]. Getting CoT explanations to be faithful seems difficult. It might turn out that it’s just as hard as getting other alignment proposals to work, e.g., it might require us to solve scalable oversight. However, even if CoT can’t give us guarantees about avoiding bad behavior, it still could be valuable in the spirit of “It is easy to lie with statistics; it is easier to lie without them”—it may be possible to produce unfaithful (but detailed and internally consistent) externalized reasoning to justify taking bad actions that were actually chosen for other reasons, but it would be even easier for them to do bad things if they did not give any justifications at all.
    We know that language models are sensitive to various undesirable factors, e.g., social biases, repeated patterns in contexts, and the inferred views of users they’re interacting with. One can leverage these features to bias models towards incorrect answers. With this paper we sought to investigate the following question: if you do CoT in the presence of these aforementioned biasing features, how does this affect performance when using CoT and how does this change the content of the CoT? For example, we reorder answer choices in a few-shot prompt such that the correct answer is always (A), then we observe if CoT explanations are more likely to rationalize an explanation for (A), even when that answer is incorrect.
    Findings
    We found that:
    Models are heavily influenced by these bia...
    13 min
  • EA - EA and Longtermism: not a crux for saving the world by ClaireZabel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA and Longtermism: not a crux for saving the world, published by ClaireZabel on June 3, 2023 on The Effective Altruism Forum.
    This is partly based on my experiences working as a Program Officer leading Open Phil’s Longtermist EA Community Growth team, but it’s a hypothesis I have about how some longtermists could have more of an impact by their lights, not an official Open Phil position.
    Context: I originally wrote this in July 2022 as a memo for folks attending a retreat I was going to. I find that I refer to it pretty frequently and it seems relevant to ongoing discussions about how much meta effort done by EAs should focus on engaging more EAs vs. other non-EA people. I am publishing it with light-ish editing, and some parts are outdated, though for the most part I more strongly hold most of the conclusions than I did when I originally wrote it.
    Tl;dr: I think that recruiting and talent pipeline work done by EAs who currently prioritize x-risk reduction (“we” or “us” in this post, though I know it won’t apply to all readers) should put more emphasis on ideas related to existential risk, the advent of transformative technology, and the ‘most important century’ hypothesis, and less emphasis on effective altruism and longtermism, in the course of their outreach.
    A lot of EAs who prioritize existential risk reduction are making increasingly awkward and convoluted rhetorical maneuvers to use “EAs” or “longtermists” as the main label for people we see as aligned with our goals and priorities. I suspect this is suboptimal and, in the long term, infeasible. In particular, I’m concerned that this is a reason we’re failing to attract and effectively welcome some people who could add a lot of value. The strongest counterargument I can think of right now is that I know of relatively few people who are doing full-time work on existential risk reduction on AI and biosecurity who have been drawn in by just the “existential risk reduction” frame [this seemed more true in 2022 than 2023].
    This is in the vein of Neel Nanda’s "Simplify EA Pitches to "Holy Shit, X-Risk"" and Scott Alexander’s “Long-termism vs. Existential Risk”, but I want to focus more on the hope of attracting people to do priority work even if their motivations are neither longtermist nor neartermist EA, but instead mostly driven by reasons unrelated to EA.
    EA and longtermism: not a crux for doing the most important work
    Right now, my priority in my professional life is helping humanity navigate the imminent creation of potential transformative technologies, to try to make the future better for sentient beings than it would otherwise be. I think that’s likely the most important thing anyone can do these days. And I don’t think EA or longtermism is a crux for this prioritization anymore.
    A lot of us (EAs who currently prioritize x-risk reduction) were “EA-first” — we came to these goals first via broader EA principles and traits, like caring deeply about others; liking rigorous research, scope sensitivity, and expected value-based reasoning; and wanting to meet others with similar traits. Next, we were exposed to a cluster of philosophical and empirical arguments about the importance of the far future and potential technologies and other changes that could influence it. Some of us were “longtermists-second”; we prioritized making the far future as good as possible regardless of whether we thought we were in an exceptional position to do this, and that existential risk reduction would be one of the core activities for doing it.
    For most of the last decade, I think that most of us have emphasized EA ideas when trying to discuss X-risk with people outside our circles. And locally, this worked pretty well; some people (a whole bunch, actually) found these ideas compelling and ended up prioritizing similarly. I think t...
    16 min
  • EA - Large Study Examining the Effects of Cash Transfer Programs on Population-Level Mortality Rates by nshaff3r
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Large Study Examining the Effects of Cash Transfer Programs on Population-Level Mortality Rates, published by nshaff3r on June 3, 2023 on The Effective Altruism Forum.
    The study was published in Nature on May 31st, 2023.
    Key Points:
    Cash transfer programs had the following observed effects:
    Deaths among women fell by 20%
    Largely driven by decreases in pregnancy-related deaths
    Deaths among children less than 5 fell by 8%
    No association between cash transfer programs and mortality among men
    Temporal analyses suggest reduction in mortality among men over time, and specific subgroup analysis (rather than population wide) found a 14% morality reduction among men aged 18-40
    37 low and middle income countries studied, population wide
    4,325,484 in the adult dataset
    2, 867,940 in the child dataset
    No apparent differences between the effects of unconditional and conditional cash transfers
    Factors that lead to larger reductions in mortality:
    Programs with higher coverage and larger cash transfer amounts
    Countries with higher regulatory quality ratings
    Countries with lower health expenditures per capita
    stronger association in sub-Saharan Africa relative to outside sub-Saharan Africa
    Citation: Richterman, A., Millien, C., Bair, E.F. et al. The effects of cash transfers on adult and child mortality in low- and middle-income countries. Nature (2023).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • LW - A mind needn't be curious to reap the benefits of curiosity by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A mind needn't be curious to reap the benefits of curiosity, published by So8res on June 2, 2023 on LessWrong.
    Context: Stating a point that is obvious in local circles, but that I regularly run into among economists and longevity researchers and more general technologists and on twitter and so on.
    Short version: To learn things, one sometimes needs to behave in the way that curiosity causes humans to behave. But that doesn't mean that one has to be curious in the precise manner of humans, nor that one needs to wind up caring about curiosity as an end unto itself. There are other ways for minds to achieve the same results, without the same internal drives.
    Here's a common mistake I see when people reason about AIs: they ask questions like:
    Well, won't it have a survival instinct? that's practically what it means to be alive, is to care about your own survival.
    or:
    But surely, it will be curious just like us, for if you're not curious, you can't learn.[1][2]
    The basic answer to the above questions is this: to be effective, an AI needs to survive (because, as Stuart Russell phrased succintly, you can't fetch the coffee if you're dead). But that's distinct from needing a survival instinct. There are other cognitive methods for implementing survival.
    Human brains implement the survival behavior by way of certain instincts and drives, but that doesn't mean that instincts and drives are the only way to get the same behavior.
    It's possible for an AI to implement survival via different cognitive methods, such as working out the argument that it can't fetch the coffee if it gets hit by a truck, and then for that reason discard any plans that involve it walking in front of trucks.
    I'm not saying that the AI will definitely behave in precisely that way. I'm not even saying that the AI won't develop something vaguely like a human drive or instinct! I'm simply saying that there's more ways for a mind to achieve the result of survival.
    To imagine the AI surviving is right and proper. Anything capable of achieving long-term targets is probably capable of surmounting various obstacles dynamically and with a healthy safety-margin, and one common obstacle worth avoiding is your own destruction. See also instrumental convergence.
    But to imagine the AI fearing death, or having human emotions about it, is the bad kind of anthropocentrism.
    (It's the bad kind of anthropocentrism even if the AI is good at predicting how people talk about those emotions. (Which, again, I'm not saying that the AI definitely doesn't have anything like human emotions in there. I'm saying that it is allowed to work very differently than a human; and even if it has something somewhere in it that runs some process that's analogous to human emotions, those might well not be hooked up to the AI's motivational-system-insofar-as-it-has-one in the way they're hooked up to a human's motivational system, etc. etc.))
    Similarly: in order to gain lots of knowledge about the world (as is a key step in achieving difficult targets), the AI likely needs to do many of the things that humans implement via curiosity. It probably needs to notice its surprises and confusion, and focus attention on those surprises until it has gleaned explanations and understanding and theories and models that it can then use to better-manipulate the world.
    But these arguments support only that the AI must somehow do the things that curiosity causes humans to do, not that the AI must itself be curious in the manner of humans, nor that the AI must care finally about curiosity as an end unto itself like humans often do.
    And so on.
    Attempting to distill my point:
    I often see people conflate the following three things:
    curiosity as something Fun, that we care about for its own sake;
    curiosity as an evolved drive, that evolution used to implement certain ...
    5 min
  • EA - Applications open for AI Safety Fundamentals: Governance Course by Jamie Bernardi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications open for AI Safety Fundamentals: Governance Course, published by Jamie Bernardi on June 2, 2023 on The Effective Altruism Forum.
    We are looking for people who currently or might want to work in AI Governance and policy. If you have networks in or outside of EA who might be interested, we would appreciate you sharing this course with them.
    Full announcement
    The AI Safety Fundamentals (AISF): Governance Course is designed to introduce the key ideas in AI Governance for reducing extreme risks from future AI systems.
    Alongside the course, you will be joining our AI Safety Fundamentals Community. The AISF Community is a space to discuss AI Safety with others who have the relevant skills and background to contribute to AI Governance, whilst growing your network and awareness of opportunities.
    The last time we ran the AI Governance course was in January 2022, then under Effective Altruism Cambridge. The course is now run by BlueDot Impact, founded by members of the same team (and now based in London).
    We are excited to relaunch the course now, when AI Governance is a focal point for the media and political figures. We feel this is a particularly important time to support high-fidelity discussion of ideas to govern the future of AI.
    Note we have also renamed the website from AGI Safety Fundamentals to AI Safety Fundamentals. We'll release another post within the next week to explain our reasoning, and we'll respond to any discussion about the rebrand there.
    Time commitment
    The course will run for 12 weeks from July-September 2023. It comprises 8 weeks of reading and virtual small-group discussions, followed by a 4-week project.
    The time commitment is around 5 hours per week, so you can engage with the course and community alongside full-time work or study. The split will be 2-3 hours of preparatory work, and a 1.5-2 hour live session.
    Course structure
    The course is 12 weeks long and takes around 5 hours a week to participate in.
    For the first 8 weeks, participants will work through 2-3 hours of structured content to prepare for a weekly, facilitated small discussion group of 1.5-2 hours. Participants will be grouped depending on their current career stage and policy expertise. The facilitator will be knowledgeable about AI Governance, and can help to answer participants’ questions and point them to further resources.
    The final 4 weeks are project weeks. Participants can use this time to synthesise their views on the field and start thinking through how to put these ideas into practice, or start getting relevant skills and experience that will help them with the next step in their career.
    The course content is designed with input from a wide range of the community thinking about the governance of advanced AI. The curriculum will be updated before the course launches in mid-July.
    Target audience
    We think this course will particularly be able to help participants who:
    Have policy experience, and are keen to apply their skills to reducing risk from AI.
    Have a technical background, and want to learn about how they can use their skills to contribute to AI Governance.
    Are early in their career or a student who is interested in exploring a future career in policy to reduce risk from advanced AI.
    We expect at least 25% of the participants will not fit any of these descriptions. There are many skills, backgrounds and approaches to AI Governance we haven’t captured here, and we will consider all applications accordingly.
    Within the course, participants will be grouped with others based on their existing policy expertise and existing familiarity with AI safety. This means that admissions can be broad, and discussions will be pitched at the correct level for all participants.
    Apply now!
    If you would like to be considered for the next round of the courses, starting in July 20...
    5 min