The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Technological developments that could increase risks from nuclear weapons: A shallow review by MichaelA
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Technological developments that could increase risks from nuclear weapons: A shallow review, published by MichaelA on February 9, 2023 on The Effective Altruism Forum.
    This is a blog post, not a research report, meaning it was produced relatively quickly and is not to Rethink Priorities' typical standards of substantiveness and careful checking for accuracy.
    Summary
    This post is a shallow exploration of some technological developments that might occur and might increase risks from nuclear weapons - especially existential risk or other risks to the long-term future. This is one of many questions relevant to how much to prioritize nuclear risk relative to other issues, what risks and interventions to prioritize within the nuclear risk area, and how that should change in future. But note that, due to time constraints, this post isn’t comprehensive and was less thoroughly researched and reviewed than we’d like.
    For each potential development, we provide some very quick, rough guesses about how much and in what ways the development would affect the odds and consequences of nuclear conflict (“Importance”), the likelihood of the development in the coming decade or decades (“Likelihood/Closeness”), and how much and in what ways thoughtful altruistic actors could influence whether and how the technology is developed and used (“Steerability”).
    These tentative bottom line beliefs are summarized in the table below:
    Radiological weaponsPure fusion bombsHigh-altitude electromagnetic pulse (HEMP)Neutron bombsAtomically precise manufacturing (APM)AI-assisted production/designOther developments in methods for production/design???Hypersonic missiles/glide vehiclesMore accurate nuclear weaponsLong-range conventional strike capabilitiesBetter detection of nuclear warhead platforms, launchers, and/or delivery vehiclesMissile defense systemsAdvances in AI capabilitiesCyberattack (or defense) capabilitiesAdvances in autonomous weaponsMore integration of AI with NC3 systemsAnti-satellite weapons (ASAT)“Space planes” and other (non-ASAT) space capabilities
    Category
    Technological Development
    Importance
    Likelihood / Closeness
    Steer-ability
    Bomb types and production methods
    Medium
    Medium/
    High
    Medium/
    Low
    Medium
    Medium/
    Low
    Medium
    Medium
    Medium/
    Low
    Medium/
    Low
    Low
    Medium/
    Low
    Medium
    Methods for production and design
    High
    Low
    Medium
    Medium/
    High
    Medium/
    Low
    Medium
    Delivery systems
    Medium/Low
    Medium/
    High
    Medium/
    Low
    Medium
    Medium
    Medium/
    Low
    Medium/Low
    Medium/
    High
    Medium/
    Low
    Detection and defense
    Medium/High
    Medium/
    High
    Medium/
    Low
    Medium
    Medium
    Medium/
    Low
    AI and cyber
    Medium/
    High
    Medium/
    High
    Medium
    Medium/
    High
    Medium/
    High
    Medium
    Medium
    Medium/
    High
    Medium
    Medium
    Medium
    Medium
    Non-nuclear warmaking advances
    Medium/
    Low
    Medium
    Medium/
    Low
    Medium/
    Low
    Medium
    Medium/
    Low
    Note that:
    Each “potential technological development” is really more like a somewhat wide area in which a variety of different types and levels of development could occur, which makes the ratings in the above table less meaningful and more ambiguous.
    “Importance” is here assessed conditional on the development occurring, so will overstate the importance of thinking about or trying to steer unlikely developments.
    In some cases (e.g, “More accurate nuclear weapons”), the “Importance” score accounts for potential risk-reducing effects as well.
    “Likelihood/Closeness” is actually inelegantly collapsing together two different things, making our ratings of developments on that criterion less meaningful. E.g., one development could be moderately likely to occur quite soon and moderately likely to occur never, while another is very likely to occur in 15-25 years but not before then.
    Some of the topics this post discusses involve or are adjacent to information hazards (especially attention hazards), as is the case with much other di...
    9 min
  • EA - Apply for Cambridge ML for Alignment Bootcamp (CaMLAB) [26 March - 8 April] by hannah
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply for Cambridge ML for Alignment Bootcamp (CaMLAB) [26 March - 8 April], published by hannah on February 9, 2023 on The Effective Altruism Forum.
    TL;DR: A two-week machine learning bootcamp this spring in Cambridge, UK, open to global applicants and aimed at providing ML skills for AI alignment. Apply by 26 February to participate or TA.
    Following a series of machine learning bootcamps earlier this year in Cambridge, Berkeley and Boston, the Cambridge AI Safety Hub is running the next iteration of the Cambridge ML for Alignment Bootcamp (CaMLAB) in spring.
    This two-week curriculum expects no prior experience with machine learning, although familiarity with Python and understanding of (basic) linear algebra is crucial.
    The curriculum, based on MLAB, provides a thorough, nuts-and-bolts introduction to the state-of-the-art in ML techniques such as interpretability and reinforcement learning. You’ll be guided through the steps of building various deep learning models, from ResNets to transformers. You’ll come away well-versed in PyTorch and useful complementary frameworks.
    From Richard Ren, an undergraduate at UPenn who participated in the January camp:
    The material from the bootcamp was well-prepared and helped me understand how to use PyTorch and einops, as well as how backpropagation and transformers work. The mentorship from the TAs and peers was excellent, and because of their support, I think the time I spent at the camp was at least 3-5x as productive as focused time I would've spent outside of the camp learning the material on my own — propelling me to be able to take graduate-level deep learning classes at my school, read AI safety papers on my own, and giving me the knowledge necessary to pursue serious machine learning research projects.
    In addition, the benefits of spending two weeks in-person with other motivated and ambitious individuals cannot be overstated: alongside the pedagogical benefits of being paired with another person each day for programming, the conversations which took place around the curriculum were a seedbed for new insights and valuable connections.
    Richard continues:
    The mentorship from the TAs, as well as the chance conversations from the people I've met, have had a serious impact on how I'll approach the career path(s) I'm interested in — from meeting an economics Ph.D. (and having my worldview on pursuing a policy career change) to talking with someone who worked at EleutherAI in the Cambridge EA office about various pathways in AI safety. I loved the people I was surrounded with — they were ambitious, driven, kind, emotionally intelligent, and hardworking.
    Feedback from the end of the previous camp showed that:
    Participants on average said they would be 93% likely to recommend the bootcamp to a friend or colleague.
    Everyone found the camp at least as good as expected, with 82% finding it better than expected, and 24% finding it much better than expected.
    94% of participants found the camp more valuable than the counterfactual use of their time, with 71% finding it much more valuable.
    In addition, first and second place in Apart Research’s January Mechanistic Interpretability Hackathon were awarded to teams formed from participants and TAs from our January bootcamp.
    Chris Mathwin, who was part of the runner-up project, writes of the bootcamp:
    A really formative experience! Great people, great content and truly great support. It was a significantly better use of my time in upskilling in this field than I would have spent elsewhere and I have continued to work with some of my peers afterwards!
    If you’re interested in participating in the upcoming round of CaMLAB, apply here. If you have substantial ML experience and are interested in being a teaching assistant (TA), apply here. You can find more details below.
    Schedule & logistics
    Th...
    10 min
  • EA - Summer Internships at Open Philanthropy - Global Health and Wellbeing (due Feb 26) by ChrisSmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summer Internships at Open Philanthropy - Global Health and Wellbeing (due Feb 26), published by ChrisSmith on February 9, 2023 on The Effective Altruism Forum.
    We’re excited to announce that the Global Health and Wellbeing Cause Prioritization team at Open Philanthropy will again be hiring several interns this summer to work with us on research to help pick new causes to support. We think this is a great way for us to grow our capacity, develop research talent, and expand our pipeline for future full-time roles. The key points are:
    Applications are due at 11:59 PM Pacific on Sunday, February 26, 2022.
    Applicants must be currently enrolled in a degree program or working in a position that offers externship/secondment opportunities.
    The internship runs from June 5 to August 11-25 (with limited adjustments based on academic calendars) and is paid ($1,900 per week) and fully remote.
    We’re open to a wide variety of backgrounds, but expect some of the strongest candidates to be enrolled in master's or doctoral programs in the social sciences.
    We aim to employ people with many different experiences, perspectives and backgrounds who share our passion for accomplishing as much good as we can. We particularly encourage applications from people of color, self-identified women, non-binary individuals, and people from low and middle income countries.
    Full details (and a link to the application) are available here and are also copied below.
    We hope that you’ll apply and share the news with others!
    About the internship
    We’re looking for students currently enrolled in degree programs (or people whose work offers externship/secondment opportunities) to apply for a research internship from June-August 2023 and help us investigate important questions and causes. We see the internship as a way to grow our capacity, develop promising research talent, and expand our recruiting pipeline for full-time roles down the line. We plan to treat interns as team members working on the team’s core priorities, while also showing them how Open Philanthropy works and helping them build skills important for cause prioritization research.
    Most projects will take the form of “opinionated, short research briefs” – synthesizing expert opinion, academic research, and prior views to get to a bottom line on an important question or promising area. Examples of projects our interns have taken on in the past include:
    assessing how greater risk of conflict from climate change should affect our social cost of carbon
    evaluating tobacco control as an area for high impact grantmaking
    helping with a major internal project focused on determining Open Philanthropy’s optimal spending path
    We expect that future interns will work on additional shallow investigations on diverse and potentially highly impactful topics — such as community health workers in low- and lower-middle income countries, the elimination of non-compete clauses from employment contracts, and vaccine adjuvants. Interns will work on multiple projects of different depths in the same way as full-time team members. Specific projects will depend on the team’s needs and the intern’s skills. Interns will report to an existing cause prioritization team member and participate in team meetings and discussions, including presenting their work to the team for feedback.
    Like the day-to-day work of our full-time Research and Strategy Fellows, the internship would require:
    Talking to global experts, reviewing reports or academic papers, and working with potential grantees to decide whether a potential cause area is important, neglected, and tractable.
    Dividing time between gathering new information and synthesizing it into concrete recommendations.
    Working to get the right answer, not to summarize others’ views. This will require making reasonable judgment calls and be...
    8 min
  • EA - Hardening pharmaceutical response to pandemics: concrete project seeks project lead by Joel Becker
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hardening pharmaceutical response to pandemics: concrete project seeks project lead, published by Joel Becker on February 9, 2023 on The Effective Altruism Forum.
    TL;DR
    We are looking for a collaborator to explore projects aiming to harden pharmaceutical infrastructure.
    We would figure out up to 3 months of full-time equivalent funding, at up to 100% of ‘EA-salary,’ with the duration of the project ranging from 3 to 12 months based on the lead's preferences. The successful candidate would be responsible for (1) refining/adapting our MVP proposal, (2) finding an external partner, and (3) securing additional funding.
    Express your interest here!
    Pharmaceutical response needs hardening
    Certain pandemic scenarios may lead to severe compromises to pharmaceutical response. Such compromises would increase the likelihood of far worse pandemic outcomes.
    Governments already expend significant resources to protect command and control, military response, and other capabilities against more traditional threats. By comparison, the pharmaceutical response capability is relatively unprotected.
    Meanwhile, we believe that there are high-impact, relatively low-cost measures that can be taken to better harden existing facilities against these threats.
    We have the beginnings of a plan
    See here.
    We have also interacted with a couple of (technologically and politically) relevant stakeholders, who have expressed interest in seeing this project move forward.
    We need someone to run with this + partner organization
    Our first blocker is a person to drive this project forward. While we can act as involved advisors, we won’t have the capacity to work on this full-time over the next 2-3 years.
    The ideal candidate would have the following attributes, in descending order of importance. (Please don’t be discouraged from submitting the interest form if you don’t fit all of the criteria!)
    Share core vision/be enthusiastic about taking responsibility for success of project,
    Experience successfully leading projects, Excellent communicator, planner, team-builder,
    Skilled at building relationships with and managing external stakeholders,
    Expertise in biosecurity or related fields, and/or
    Existing connections with relevant stakeholders.
    We are open to commitments from 1 day per week to full-time.
    Our second blocker is a partner organization with whom we can develop, iterate, and implement our ideas in the real world.
    That person could be you
    So let us know if you’re interested here!
    We also welcome suggestions about the substance of the idea, or about getting more buy-in from important stakeholders.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - On Developing a Mathematical Theory of Interpretability by Spencer Becker-Kahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Developing a Mathematical Theory of Interpretability, published by Spencer Becker-Kahn on February 9, 2023 on LessWrong.
    If the trajectory of the deep learning paradigm continues, it seems plausible to me that in order for applications of low-level interpretability to AI not-kill-everyone-ism to be truly reliable, we will need a much better-developed and more general theoretical and mathematical framework for deep learning than currently exists. And this sort of work seems difficult. Doing mathematics carefully - in particular finding correct, rigorous statements and then finding correct proofs of those statements - is slow. So slow that the rate of change of cutting-edge engineering practices significantly worsens the difficulties involved in building theory at the right level of generality. And, in my opinion, much slower than the rate at which we can generate informal observations that might possibly be worthy of further mathematical investigation.
    Thus it can feel like the role that serious mathematics has to play in interpretability is primarily reactive, i.e. consists mostly of activities like 'adding' rigour after the fact or building narrow models to explain specific already-observed phenomena.
    My impression however, is that the best applied mathematics doesn’t tend to work like this. My impression is that although the use of mathematics in a given field may initially be reactive and disunited, one of the most lauded aspects of mathematics is a certain inevitability with which our abstractions take on a life of their own and reward us later with insight, generalization, and the provision of predictions. Moreover - remarkably - often those abstractions are found in relatively mysterious, intuitive ways: i.e. not as the result of us just directly asking "What kind of thing seems most useful for understanding this object and making predictions?" but, at least in part, as a result of aesthetic judgement and a sense of mathematical taste.
    One consequence of this (which is a downside and also probably partly due to the inherent limitations of human mathematics) is that mathematics does not tend to act as an objective tool that you can bring to bear on whatever question it is that you want to think about. Instead, the very practice of doing mathematics seeks out the questions that mathematics is best placed to answer. It cannot be used to say something useful about just anything; rather it finds out what it is that it can say something about.
    Even after taking into account these limitations and reservations, developing something that I'm clumsily thinking of as 'the mathematics of (the interpretability of) deep learning-based AI' might still be a fruitful endeavour. In case it is not clear, this is roughly speaking, because a) Many people are putting a lot of hope and resources into low-level interpretability; b) Its biggest hurdles will be making it 'work' at large scale, on large models, quickly and reliably; and c) - the sentiment I opened this article with - doing this latter thing might well require much more sophisticated general theory.
    In thinking about some of these themes, I started to mull over a couple of illustrative analogies or examples. The first - and more substantive example - is algebraic topology. This area of mathematics concerns itself with certain ways of assigning mathematical (specifically algebraic) information to shapes and spaces. Many of its foundational ideas have beautiful informal intuitions behind them, such as the notion that a shape my have enough space in it to contain a sphere, but not enough space to contain the ball that that sphere might have demarcated. Developing these informal notions into rigorous mathematics was a long and difficult process and learning this material - even now when it is presented in its best form - i...
    11 min
  • AF - Conditioning Predictive Models: Deployment strategy by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Deployment strategy, published by Evan Hubinger on February 9, 2023 on The AI Alignment Forum.
    This is the sixth of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out.
    6. Deployment strategy
    Previously, we have been focusing on how to make conditioning predictive models as safe and competitive as possible. Now, we want to take a step back and discuss considerations for using conditioning predictive models to address AI existential risk and what sorts of difficulties we might run into doing so in practice.
    In particular, just as AI safety researchers naturally think of using predictive models for advancing AI safety research, AI capabilities researchers might naturally jump to using predictive models for advancing capabilities. It may not even be necessary to generate additional research to build AGI with a powerful predictive model. Simply ignoring the previously-discussed ELK-related difficulties and training a model to take actions that lead to predicted futures that a predicted human approves of may be sufficient. Either way, the existence of powerful predictive models seems likely to rapidly contract AI timelines.
    As a result, by the time predictive models can be used to predict a full solution to AI safety, the time available to do so is minimal—and as such, it is important to have fleshed out plans on how to use them safely well ahead of time.
    Dealing with other, less careful actors
    As we mentioned previously, using a predictive model to generate alignment research is only one possible use case—one that we restricted our attention to on the basis that we thought it contained the difficult aspects of using a predictive model safely. Restricting our attention to these sorts of particular conditionals—and figuring out how to do them safely—is fine if we have control over the ways in which our model will be used. If we don’t have that control, however—e.g. we are in a world where people are using predictive models in all sorts of different ways—then we have to consider what might happen when our predictive model is used in a much less careful way than described here and figure out how to either deal with or prevent that from happening.
    We think that getting other actors to use predictive models at all should be quite doable, for standard homogeneity reasons: why would a non-leading actor want to invest a ton of resources training a model in a different way than the way that the leading actor has already demonstrated successfully produces transformative AI? The problem, however, is that this same argument does not apply to what particular conditionals the non-leading actors might try, since trying a particular conditional is likely to be substantially cheaper than training an entire predictive model.
    In a multipolar world, one team using very careful conditioning to get a predictive model to generate good alignment research means that other teams will likely soon have equivalently good models and might use them less carefully—e.g. resulting in them accidentally predicting malign superintelligences. Even in a unipolar world, a member of the team that created the predictive model might try to predict their future great-grandchildren out of curiosity, or check the predicted stock prices when they plan to retire, and inadvertently become exposed to manipulative outputs.
    Since powerful predictive models can easily be used in less careful ways, any deployment strategy ...
    15 min
  • LW - Wanted: Technical animator and/or front-end developer for interactive diagrams of invention by jasoncrawford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wanted: Technical animator and/or front-end developer for interactive diagrams of invention, published by jasoncrawford on February 8, 2023 on LessWrong.
    Seeking help for a project to create interactive diagrams—an “explorable explanation”—of the history of the steam engine.
    Goal: To allow readers to explore the early evolution of one of the most impactful technologies in human history.
    Content: The content will come from Anton Howes’s extensive research uncovering the origins of the steam engine (parts 1, 2, and 3).
    We envision it covering precursors like the perpetual motion machine of Cornelis Drebbel, early thermometers and barometers, seventeenth-century experiments on atmospheric pressure, precursor devices like those shown by Salomon de Caus, our best guesses of the Kalthoff/Petty engine, the Savery engine, and the Newcomen engine and its improvements. (After this initial phase, we would then want to expand it to include the improvements by James Watt, and the rise of the high-pressure engine, with its various nineteenth-century improvements.)
    This page on the steam engine may serve as a start for eventually creating the definitive online and interactive reference work on the history of technology, informed by the latest expert research.
    Format: We have been greatly inspired by the works of Bartosz Ciechanowski, such as his mechanical watch and internal combustion engine explanations. We’re looking to create something like this. Here are more examples of locomotive valves.
    The team: Anton Howes as researcher/writer, me (Jason Crawford) as editor/producer, you as animator/developer.
    Your input: We’ll provide the written essay and some sketches and ideas for what diagrams we should create and how they should work.
    Your deliverables: The code and/or other files to implement a set of interactive and/or animated technical diagrams.
    You: A technical illustrator, animator, and/or front-end developer, or a team, who can deliver this. Must have a portfolio of relevant work to demonstrate your abilities. The ideal candidate will have an interest in engineering, invention, and the history of technology.
    How to apply: Email [email protected] with a description of your skills and examples of your work.
    We have budget for this project and are willing to pay market rates for high-quality work.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Make CoI Policies Public by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Make CoI Policies Public, published by Jeff Kaufman on February 9, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - In (mild) defence of the social/professional overlap in EA by Amber Dawn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In (mild) defence of the social/professional overlap in EA, published by Amber Dawn on February 9, 2023 on The Effective Altruism Forum.
    The EA community has both a professional aspect and a social aspect; sometimes, these overlap. In particular, there are networks of EAs in major hubs who variously date each other, are friends with each other, live together, work at the same organizations, and sometimes make grantmaking or funding decisions related to each other. (I’ll call this the ‘work/social overlap’ throughout).
    Recently, lots of people have criticised the work/social overlap in EA, blaming it for things like the misogynistic, abusive dynamics described in the recent Time article, or for the FTX collapse. In this post, I talk a bit about why I’m wary of calls to “deal with” or “address” the work/social overlap in some way (this includes, but isn’t limited to, calls to “deal with” polyamory in EA specifically).
    This is partly because I think that the work/social overlap has some strong benefits, which is why it exists, and the benefits outweigh the drawbacks (though we should certainly try to mitigate the drawbacks as best we can). If you want to get rid of something, you should first try to make sure that it’s not importantly load-bearing first; I think in fact that EA work/social overlap probably is importantly load-bearing in some ways. (I never thought that I’d be invoking a Chesterton’s fence argument in support of polyamorously dating all your coworkers :p but there we go).
    And it’s partly because I think it’s immoral and harmful to try to prevent people from consensually forming relationships with whom they want, and it’s only a little better if the mechanisms you use to attempt this are supposedly “soft” or “non-coercive”.
    In Forum tradition, disclosures and caveats first:
    I’m fully embedded in the EA work/social overlap. I work as a freelancer, mostly with EA clients, some of whom I know socially; I live in London and am friends with many other London EAs; and I’m dating two other EAs.
    I also want to acknowledge that there are downsides to the work/social overlap - the critics aren’t wrong about their points, they are just missing important other parts of the story. In particular:
    1. The work/social overlap gives rise to conflicts-of-interest (and this is more of a problem in highly poly communities, because there are just more conflict-of-interest-ish relationships)
    2. The work/social overlap means that people who are engaged with EA professionally, but not part of the social community, may miss out on opportunities. The recent ‘come hang out in the Bay Area’ push seems to be a tacit acknowledgement that this is, in fact, how it works. It’s good that Bay Area EAs tried to mitigate this by inviting non-locals to visit the Bay and get more plugged in to the in-person social community, but it would be better, in my opinion, if organizations in hubs were less clique-y and more willing to hire people who weren’t part of the social community (and indeed, people who don’t identify as EA at all).
    3. You’re putting all your eggs in one basket. If you live with EAs, date EAs, are friends with EAs, and work with EAs, it makes the thought of leaving the EA community really hard. This is bad for epistemics: for a while, I found it really hard to think clearly about where I stood on EA philosophical/ideological positions, because I felt so ‘locked in’ to the community in all parts of my life.
    (I got past this thanks to a fruitful coaching session with Tee Barnett, where he pointed out that my social relationships formed through EA are now much deeper than our EA connection, and when I actually thought about it, I realised that most of my EA friends wouldn’t ditch me or disown me if I came to disagree with EA, so leaving EA wouldn’t really mean losing access to my so...
    10 min
  • AF - EIS II: What is “Interpretability”? by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS II: What is “Interpretability”?, published by Stephen Casper on February 9, 2023 on The AI Alignment Forum.
    Part 2 of 12 in the Engineer’s Interpretability Sequence.
    A parable based on a true story
    In 2015, a Google’s image classification app classified many photos depicting black people as gorillas. Image from WSJ.
    Remember Google’s infamous blunder from 2015 in which users found that one of its vision API’s often misclassified black people as gorillas? Consider a parable of two researchers who want to understand and tackle this issue.
    Alice is an extremely skilled mechanistic interpretability researcher who spends a heroic amount of effort analyzing Google’s model. She identifies a set of neurons and weights that seem to be involved in the detection and processing of human and gorilla faces and bodies. She develops a detailed mechanistic hypothesis and writes a paper about it with 5 different types of evidence for her interpretation. Later on, another researcher who wants to test Alice’s hypothesis edits the model in a way that the hypothesis suggests would fix the problem. As it turns out, the hypothesis was imperfect, and the model now classifies many images of gorillas as humans!
    Bob knows nothing about neural networks. Instead of analyzing the network, he looks at the dataset that the model was trained on and notices a striking lack of black people (as was indeed the case in real life (Krishnan, 2020)). He suggests making the data more representative and training the model again. When this is done, it mostly fixes the problem without side effects.
    The goal of this parable is to illustrate that when it comes to doing useful engineering work with models, a mechanistic understanding may not always be the best way to go. We shouldn’t think of something called “interpretability” as being fundamentally separate from other tools that can help us accomplish our goals with models. And we especially shouldn’t automatically privilege some methods over others. In some cases, highly involved and complex approaches may be necessary. But in other cases like Alice’s, the interesting, smart, and paper-able solution to the problem might not only be harder but could also be more failure-prone. This isn’t to say that Alice’s work could never lead to more useful insights down the road. But in this particular case Alice’s smart approach was not as good as Bob’s simple one.
    Interpretability is a means to an end.
    Since I work and think about interpretability every day, I have felt compelled to adopt a definition for it. In a previous draft of this post, I proposed defining an interpretability tool as “any method by which something novel about a system can be better predicted or described.” And I think this is ok, but I have recently stopped caring about any particular definition. Instead, I think the important thing to understand is that “interpretability’ is not a term of any fundamental importance to an engineer.
    The key idea behind this post is that whatever we call “interpretability” tools are entirely fungible with other techniques related to describing, evaluating, debugging, etc.
    Does this mean that it’s the same thing as interpretability if we just calculate performance on a test set, train an adversarial example, do some model pruning, or make a prediction based on the dataset? Pretty much. For all practical intents and purposes, these things all are all of a certain common type. Consider any of the following sentences.
    This model handles 85% of the data correctly.
    This input plus whatever is in this adversarial perturbation make the model fail.
    I got rid of 90% of the weights and the model’s performance only decreased by 2%.
    The dataset has this particular bias, so the model probably will as well.
    This model seems to have a circuit composed of these neurons and these wei...
    7 min