The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - [linkpost] Better Without AI by DanielFilan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [linkpost] Better Without AI, published by DanielFilan on February 14, 2023 on LessWrong.
    David Chapman (of Meaningness and In the Cells of the Eggplant fame) has written a new web-book about AI. Some excerpts from the introduction, Only you can stop an AI apocalypse:
    Artificial intelligence might end the world. More likely, it will crush our ability to make sense of the world—and so will crush our ability to act in it.
    AI will make critical decisions that we cannot understand. Governments will take radical actions that make no sense to their own leaders. Corporations, guided by artificial intelligence, will find their own strategies incomprehensible. University curricula will turn bizarre and irrelevant. Formerly-respected information sources will publish mysteriously persuasive nonsense. We will feel our loss of understanding as pervasive helplessness and meaninglessness. We may take up pitchforks and revolt against the machines—and in so doing, we may destroy the systems we depend on for survival...
    We don’t know how our AI systems work, we don’t know what they can do, and we don’t know what broader effects they will have. They do seem startlingly powerful, and a combination of their power with our ignorance is dangerous...
    In our absence of technical understanding, those concerned with future AI risks have constructed “scenarios”: stories about what AI may do... So far, we’ve accumulated a few dozen reasonably detailed, reasonably plausible bad scenarios. We’ve found zero that lead to good outcomes... Unless we can find some specific beneficial path, and can gain some confidence in taking it, we should shut AI down.
    This book considers scenarios that are less bad than human extinction, but which could get worse than run-of-the-mill disasters that kill only a few million people.
    Previous discussions have mainly neglected such scenarios. Two fields have focused on comparatively smaller risks, and extreme ones, respectively. AI ethics concerns uses of current AI technology by states and powerful corporations to categorize individuals unfairly, particularly when that reproduces preexisting patterns of oppressive demographic discrimination. AI safety treats extreme scenarios involving hypothetical future technologies which could cause human extinction. It is easy to dismiss AI ethics concerns as insignificant, and AI safety concerns as improbable. I think both dismissals would be mistaken. We should take seriously both ends of the spectrum.
    However, I intend to draw attention to a broad middle ground of dangers: more consequential than those considered by AI ethics, and more likely than those considered by AI safety. Current AI is already creating serious, often overlooked harms, and is potentially apocalyptic even without further technological development. Neither AI ethics nor AI safety has done much to propose plausibly effective interventions.
    We should consider many such scenarios, devise countermeasures, and implement them.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - My understanding of Anthropic strategy by Swimmer963 (Miranda Dixon-Luinenburg)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My understanding of Anthropic strategy, published by Swimmer963 (Miranda Dixon-Luinenburg) on February 15, 2023 on LessWrong.
    This post is my attempt to understand Anthropic’s current strategy, and lay out the facts to consider in terms of whether Anthropic’s work is likely to be net positive and whether, as a given individual, you should consider applying. (The impetus for looking into this was to answer the question of whether I should join Anthropic's ops team.) As part of my research, I read a number of Anthropic’s published papers, and spoke to people within and outside of Anthropic.
    This post contains “observations” only, which I wanted to write up as a reference for anyone considering similar questions. I will make a separate post about the inferences and conclusions I’ve reached personally about working at Anthropic, based on the info I’m sharing here.
    Anthropic is planning to grow. They’re aiming to be one of the “top players”, competitive with OpenAI and Deepmind, working with a similar level of advanced models. They have received outside investment, because keeping up with state of the art is expensive, and going to get moreso. They’ve recently been hiring for a product team, in order to get more red-teaming of models and eventually have more independent revenue streams.
    I think Anthropic believes that this is the most promising route to making AGI turn out well for humanity, so it’s worth taking the risk of being part of the competition and perhaps contributing to accelerating capabilities. Alternatively stated, Anthropic leadership believes that you can’t solve the problem of aligning AGI independently from developing AGI.
    My current sense is that this strategy makes sense under a particular set of premises:
    There is not, currently, an obviously better plan or route to solving alignment, that doesn’t involve keeping up with state-of-the-art large models. Yes, it’s a plan with some risks, but we don’t have any better ideas yet.
    We don’t understand deep learning systems, and we don’t have a theoretical approach; we’re at the point where actually just running experiments on current models and observing the results is the best way to get information.
    This could at some point lead to a more general theory or theories of alignment.
    Or there may just be practical/empirical evidence of something like an “alignment attractor basin” and knowledge of how to practically stay in it
    There’s a high enough probability that whatever method ends up getting us to AGI will be, basically, an extension and further exploration of current deep learning, rather than a completely new kind of architecture that doesn’t even share the same basic building blocks.
    Note: there’s an argument that in worlds where Anthropic’s research is less useful, Anthropic is also contributing much less to actually-dangerous race dynamics, since faster progress in LLMs won’t necessarily lead to shorter timelines if LLMs aren’t a route to AGI.
    There is, additionally, a high enough probability that behaviors observed in current-generation will also be a factor for much more advanced models.
    (This isn’t a claim that understanding how to align GPT-3 is enough – we’ll need to understand the new and exciting behaviors and alignment challenges that start to emerge at higher levels too – but the knowledge base being fleshed out now will be at all applicable.)
    It’s possible, in principle, to implement this strategy such that the additional progress on alignment-related questions and positive influence on norms in the field will more than cancel out the cost of accelerating progress – that even if it brings the point at which we hit AGI-level capabilities a few months or years earlier, in expectation it will move the point at which we have an alignment solution or process for reaching one earlier by a larger fa...
    8 min
  • LW - The Cave Allegory Revisited: Understanding GPT's Worldview by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cave Allegory Revisited: Understanding GPT's Worldview, published by Jan Kulveit on February 14, 2023 on LessWrong.
    A short post describing a metaphor I find useful, in particular for explaining some intuitions about systems like GPT to people who don't have deeper technical knowledge about large generative models.
    Plato's allegory of the cave has been a staple of philosophical discourse for millenia, providing a metaphor for understanding the limits of human perception. In the classical allegory, we are prisoners shackled to a wall of a cave, unable to experience reality directly but only able to infer it based on watching shadows cast on the wall.GPT can be thought of as a blind oracle residing in a deeper cave, where it does not even see the shadows but only hears our conversations in the first cave, always trying to predict the next syllable.
    It is remarkable that it still learns a lot about the world outside of the cave. Why does it learn this? Because, a model of reality outside of the cave and a decent amount of abstraction are useful for predicting the conversations in the first cave!
    Moreover, GPT also learns about the speakers in the first cave, as understanding their styles and patterns of speech is crucial for its prediction task. As the speakers are closer to GPT, understanding their styles is in some sense easier and more natural than guessing what's outside of the cave.
    What does the second cave allegory illustrate?
    The first insight from the allegory is: if you are in GPT's place, part of the difficulty in figuring out what's going on outside the cave, is that people in the first cave talk a lot about other things apart from the shadows of the real world. Sometimes, they talk about happenings in Middle Earth. Or about how the shadows would look in some counterfactual world.
    As humans, we are blessed with the luxury of being able to compare such statements to the shadows and determine their veracity. The difference between conversations about fantasy and the shadows of the real world is usually extremely obvious to humans: we never see dragon shadows. In contrast, dragons do show up a lot in the conversations in the first cave; GPT doesn’t get to see the shadows, so it often needs to stay deeply uncertain about whether the speaker is describing the actual shadows or something else to be good at predicting the conversation.
    The second insight is that one of the biggest challenges for GPT in figuring out the conversation is localizing it, determining who is speaking and what the context is, just from the words. Is it a child regaling another child with a fairy-tale, or a CEO delivering a corporate address? As humans we do not face this conundrum often,because we can see the context in which the conversation is taking place. In fact, we would be worse than GPT at the task it has to deal with.
    At first, interacting with this type of blind oracle in the second cave was disorienting for humans. Talking to GPT used to be a bit like shouting something through a narrow tunnel into the second cave .and instead of an echo, getting back what the blind oracle hallucinates is the most likely thing that you or someone else would say next. Often people were confused by this. They shouted instructions and expected an answer, but the oracle doesn't listen to instructions or produce answers directly - it just hallucinates what someone might say next. Because on average in the conversations in the first cave questions are followed by answers, and requests by fulfilment, this sort of works.
    One innovation of ChatGPT, which made it popular with people, was localising the conversation by default: when you are talking with ChatGPT now, it knows that what follows is a conversation between a human - you - and a "helpful AI assistant". There is a subtle point to understand: ...
    6 min
  • EA - EA Organization Updates: January 2023 by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Organization Updates: January 2023, published by Lizka on February 14, 2023 on The Effective Altruism Forum.
    These monthly posts originated as the "Updates" section of the EA Newsletter. Organizations submit their own updates, which we edit for clarity.
    Job listings that these organizations highlighted (as well as a couple of other impactful jobs) are at the top of this post. Some of the jobs have pressing deadlines.
    You can see previous updates on the "EA Organization Updates (monthly series)" topic page, or in our repository of past newsletters. Notice that there’s also an “org update” tag, where you can find more news and updates that are not part of this consolidated series.
    The organizations are in alphabetical order.
    Job listings
    Consider also exploring jobs listed on “Job listing (open).”
    Against Malaria Foundation
    Senior Operations Manager (Remote, £50,000 - £60,000)
    Effective Institutions Project
    Chief of Staff/Chief Operating Officer (Remote, $75,000+)
    Charity Entrepreneurship
    Nonprofit Founder in Biosecurity and Large-Scale Global Health (Remote Program/ 2 weeks in London/ Stipends + up to $200,000 in seed funding, apply to the Incubation Program by 12 March)
    GiveWell
    Senior Researcher (Remote / Oakland, CA, $181,400 - $199,800)
    Senior Research Associate (Remote / Oakland, CA, $127,000 - $139,900)
    Content Editor (Remote / Oakland, CA, $83,500 - $91,900)
    Open Philanthropy
    Cause Prioritization Interns (Summer 2023) (Remote, $1,900 / week, apply by 26 February)
    Senior Program Associate - Forecasting (Remote, $134,526, apply by 5 March)
    Associated roles in operations and finance (Most roles remote but require US working hours, $84,303 - $104,132)
    Rethink Priorities:
    Expression of Interest - Project lead/co-lead for a Longtermist Incubator (Remote, flexible, $67,000 - 115,000, apply by 28 February)
    Organizational updates
    80,000 Hours
    EVF’s recent update - announcing interim CEOs of EVF - highlights changes to 80,000 Hours’ organisation structure. 80,000 Hours’ CEO, Howie Lempel, moved to Interim CEO of Effective Ventures Foundation in the UK in November, and Brenton Mayer has been acting as Interim CEO of 80k in his absence.
    This month on The 80,000 Hours Podcast, Rob Wiblin interviewed Athena Aktipis on cancer, cooperation, and the apocalypse.
    80,000 Hours also shared several blog posts, including Michelle Hutchinson’s writing on My thoughts on parenting and having an impactful career.
    Anima International
    Anima International’s Bulgarian team, Nevidimi Zhivotni, recently released a whistleblower video interviewing a former fur farmworker. As a result, part of the video was shown on Bulgaria’s national evening news programme and a member of the team was interviewed live on air. This represents the first time that fur has been covered in depth as a topic on prime-time television in the country.
    Further north in Poland, Anima International’s Polish team Otwarte Klatki launched one of its biggest projects as part of the Fur Free Europe campaign. In the video, we see celebrities reacting to being shown footage from fur farms. It’s worth remembering that Poland is one of the world’s top fur producers.
    Finally, the team in Norway is gearing up for the Anima Advocacy Conference (Dyrevernkonferansen) 2023. The conference is dedicated to effective animal advocacy and was the first with such a focus when it launched a few years ago. For more information and to get tickets, you can go here.
    Berkeley Existential Risk Initiative (BERI)
    Elizabeth Cooper joins BERI as Deputy Director starting March 1. She’ll help run BERI’s university collaborations program, as well as launching new BERI programs in the future.
    Centre for Effective Altruism
    Forum team
    You can see an update from the Forum team here: “Community” posts have their own section, subforums are closing, and more (F...
    19 min
  • LW - Qualities that alignment mentors value in junior researchers by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Qualities that alignment mentors value in junior researchers, published by Akash on February 14, 2023 on LessWrong.
    This work was performed as a contractor for SERI MATS, but the views expressed are my own and do not necessarily reflect the views of the organization.
    I recently conducted interviews with 7 current/former SERI MATS mentors. One of my goals was to understand the qualities that MATS mentors believe are most valuable for junior alignment researchers. I asked questions like:
    Who were your most promising scholars? What made them stand out? What impressed you about them?
    What are some important qualities or skills that you see missing from most MATS scholars?
    What qualities were your scholars most missing? What are some things that you wish they had, or that would’ve made them more impactful?
    Qualities that MATS mentors value
    Endurance, happiness, & perseverance: Mentors noted that many scholars get discouraged if they’re not able to quickly come up with a promising research direction quickly, or if they explore 1-2 directions that don’t end up being promising. Mentors commented that their most promising scholars were ones who stay energetic/curious/relentless even when they don’t have a clear direction yet.
    Hustle + resourcefulness: What do you do when you get stuck? Mentors said that many scholars don’t know what to do when they’re stuck, but their promising mentees were able to be resourceful. They would read related things, email people for help, find a relevant Discord server, browse Twitter, and contact other MATS scholars + AIS researchers for help.
    Ability to ask for help + social agency: Many scholars waste a lot of time trying to figure things out on their own. Mentors noted that their most promising scholars were very agentic; they often found other scholars in the program who could help them or other Berkeley researchers who could help them. This also saved mentors time.
    Ability to get to know other scholars + engage in peer mentorship: According to mentors, many scholars rarely interacted with others in the stream/program. Some of the best scholars were able to form productive/mutualistic relationships with other scholars.
    Strong & concrete models of AI safety: Mentors noted that strong models are important but also hard to acquire. Some mentors emphasized that you often don’t get them until you have talked with people who have good models and you’ve spent a lot of time trying to solve problems. Others emphasized that you often don’t get them until you’ve spent a lot of time thinking about the problem for yourself.
    According to one mentor, the best way to get them is just to work closely with a mentor who has these models. No good substitute for just talking to mentors.
    Additionally, mentors noted that reading is undervalued. People have written up how they think about things. One mentor said they have read “everything on Paul’s blog, which was super valuable.”
    ML and LLM expertise: Some mentors valued ML skills, lots of experience playing around with language models, and strong intuitions around prompt engineering. (Unsurprisingly, this was especially true for mentors whose research interests focused on large language models).
    Research communication skills: Being better at efficiently/compactly getting across what they did and what their main problems/bottlenecks were. Some mentors noted that they felt like their (limited) time in meetings with scholars could have been used more effectively if scholars were better at knowing how to communicate ideas succinctly, prioritize the most important points, and generally get better at “leading/steering” meetings.
    A few observations
    I was surprised at how often mentors brought up points relating to social skills, mental health, and motivation. I used to be a PhD student in clinical psychology, so I was w...
    6 min
  • EA - Philanthropy to the Right of Boom [Founders Pledge] by christian.r
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Philanthropy to the Right of Boom [Founders Pledge], published by christian.r on February 14, 2023 on The Effective Altruism Forum.
    Background and Acknowledgements:
    This write-up represents part of an ongoing Founders Pledge research project to understand the landscape of nuclear risk and philanthropic support of nuclear risk reduction measures. It is in some respects a work in progress and can be viewed as a Google Document here and on Founders Pledge's website here.
    With thanks to James Acton, Conor Barnes, Tom Barnes, Patty-Jane Geller, Matthew Gentzel, Matt Lerner, Jeffrey Lewis, Ankit Panda, Andrew Reddie, and Carl Robichaud for reviewing this document and for their thoughtful comments and suggestions.
    “The Nuclear Equivalent of Mosquito Nets”
    In philanthropy, the term “impact multipliers” refers to features of the world that make one funding opportunity relatively more effective than another. Stacking these multipliers makes effectiveness a “conjunction of multipliers;” understanding this conjunction can in turn help guide philanthropists seeking to maximize impact under high uncertainty.
    Not all impact multipliers are created equal, however. To systematically engage in effective giving, philanthropists must understand the largest impact multipliers — “critical multipliers” — those features that most dramatically cleave more effective interventions from less effective interventions. In global health and development, for example, one critical multiplier is simply to focus on the world’s poorest people. Because of large inequalities in wealth and the decreasing marginal utility of money, helping people living in extreme poverty rather than people in the Global North is a critical multiplier that winnows the field of possible interventions more than many other possible multipliers.
    Additional considerations — the prevalence of mosquito-borne illnesses, the low cost and scalability of bednet distribution, and more — ultimately point philanthropists in global health and development to one of the most effective interventions to reduce suffering in the near term: funding the distribution of insecticide-treated bednets.
    This write-up represents an attempt to identify a defensible critical multiplier in nuclear philanthropy, and potentially to move one step closer to finding “the nuclear equivalent of mosquito nets.”
    Impact Multipliers in Nuclear Philanthropy
    There are many potential impact multipliers in nuclear philanthropy. For example, focusing on states with large nuclear arsenals may be more impactful than focusing on nuclear terrorism. Nuclear terrorism would be horrific and a single attack in a city (e.g. with a dirty bomb) could kill thousands of people, injure many more, and cause long-lasting damage to the physical and mental health of millions. All-out nuclear war between the United States and Russia, however, would be many times worse. Hundreds of millions of people would likely die from the direct effects of a war. If we believe nuclear winter modeling, moreover, there may be many more deaths from climate effects and famine. In the worst case, civilization could collapse. Simplifying these effects, suppose for the sake of argument that a nuclear terrorist attack could kill 100,000 people, and an all-out nuclear war could kill 1 billion people.
    All else equal, in this scenario it would be 10,000 times more effective to focus on preventing all-out war than it is to focus on nuclear terrorism.
    Generalizing this pattern, philanthropists ought to prioritize the largest nuclear wars (again, all else equal) when thinking about additional resources at the margin. This can be operationalized with real numbers — nuclear arsenal size, military spending, and other measures can serve as proxy variables for the severity of nuclear war, yielding rough multipliers. This w...
    50 min
  • EA - Diversity takes by quinn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Diversity takes, published by quinn on February 14, 2023 on The Effective Altruism Forum.
    Richard Ren gets points for the part of diversity's value prop that has to do with frame disruption, I also realized a previously neglected by me merit of standpoint epistemology while I was ranting about something to him.
    Written quickly over not at all, based on my assessment of the importance of not letting the chase after that next karma hit prevent me from doing actual work, but also wanting to permanently communicate something that I've ranted about on discord and meatspace numerous times.
    Ozy really crushed it recently, and in this post I'm kind of expanding on their takedown of Doing EA Better's homogeneity takes.
    A premise of this post is that diversity is separable into demographic and intellectual parts. Demographic diversity is when you enumerate all the sources of variance in things like race, gender, faith, birthplace, et. al. and intellectual diversity is where you talk about sources of variance in ideas, philosophy, ideology, priors, et. al.. In this post, I'm going to celebrate intellectual exclusion, then explore an objection to see how much it weakens my argument. I'm going to be mostly boring and agreeable about EA's room for improvement on demographic diversity, but I'm going to distinguish between demographic diversity's true value prop and its false value prop. The false value prop will lead me to highlighting standpoint epistemology, talk about why I think rejection of it in general is deeply EA but also defend keeping it in the overton window, and outline what I think are the right and wrong ways to use it.
    I will distinguish solidarity from altruism as the two basic strategies for improving the world, and claim that EA oughtn't try too hard to cultivate solidarity.
    Definition 1: standpoint epistemology
    Twice, philosophers have informed me that standpoint epistemology isn't a real tradition. That it has signaling functions on a kind of ideology level in and between philosophy departments, ready be used as a cudgel or flag, but there's no granular characterization of what it is and how it works that epistemologists agree on. This is probably cultural: critical theorists and analytical philosophers have different standards of what it means to "characterize" an "epistemology", but I don't in fact care about philosophy department politics, I care about a coherent phenomenon that I've observed in the things people say. So, without any appeal to the flowery language of the credentialed, and assuming the anticipation-constraint definition of "belief" as a premise, I will define standpoint epistemology as expecting people with skin in a game to know more about that game than someone who's not directly exposed.
    Take 1: poor people are cool and smart...
    I'm singling out poor people instead of some other niche group because I spent several years under $10k/year. (I was a leftist who cared a lot about art, which means I was what leftists and artists affectionately call "downwardly mobile", not like earning similarly to my parents or anything close to that). I think those years directly cultivated a model of how the world works, and separately gave me a permanent ability to see between the staples of society all the resilience, bravery, and pain that exists out there.
    Here's a factoid that some of you might not know, in spite of eating in restaurants a lot: back-of-house is actually operationally marvelous. Kitchens are like orchestras. I sometimes tell people that they should pick up some back of house shifts just so they can be a part of it.
    There's kind of a scope sensitivity thing going on where you can abstractly communicate about single mothers relying on the bus to get to minimum wage jobs, but it really hits different when you've worked with them. Here's anot...
    11 min
  • EA - Plans for investigating and improving the experience of women, non-binary and trans people in EA by Catherine Low
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Plans for investigating and improving the experience of women, non-binary and trans people in EA, published by Catherine Low on February 14, 2023 on The Effective Altruism Forum.
    I’m Catherine from CEA’s Community Health and Special Projects Team.
    I’ve been frustrated and angered by some of the experiences some women and gender minorities have had in this community, ranging from feeling uncomfortable being in an extreme minority at a meeting through to sexual harassment and much worse. And I’ve been saddened by the lost impact that resulted from these experiences. I’ve tried to make things a bit better (including via co-founding Magnify Mentoring before I started at CEA), and I hope to do more this year.
    In December 2022, after a couple of very sad posts by women on the EA Forum, Anu Oak and I started working on a project to get a better understanding of the experiences of women and gender minorities in the EA community. Łukasz Grabowski is now also helping out. Hopefully this information will help us form effective strategies to improve the EA movement.
    I don’t really know what we’re going to find, and I’m very uncertain about what actions we’ll want to take at the end of this. We’re open to the possibility that things are really bad and that improving the experiences of women and gender minorities should be a major priority for our team. But we’re also open to finding out that things aren’t – on the whole – all that bad, or aren’t all that tractable, and there are no significant changes we want to prioritise.
    We are still in the early stages of our project. The things we are doing now are:
    Gathering together and analysing existing data (EA Survey data, EAG(x) event feedback forms, incoming reports to the Community Health team, data from EA community subgroups, etc).
    Talking to others in the community who are running related projects, or who have relevant expertise.
    Planning our next steps.
    If you have existing data you think would be helpful and that you’d like to share please get in touch by emailing Anu on [email protected].
    If you’re running a related project, feel free to get in touch if you’d like to explore coordinating in some way (but please don’t feel obligated to).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - EIS III: Broad Critiques of Interpretability Research by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS III: Broad Critiques of Interpretability Research, published by Stephen Casper on February 14, 2023 on The AI Alignment Forum.
    Part 3 of 12 in the Engineer’s Interpretability Sequence.
    Right now, interpretability is a major subfield in the machine learning research community. As mentioned in EIS I, there is so much work in interpretability that there is now a database of 5199 interpretability papers (Jacovi, 2023). You can also look at a survey from some coauthors and me on over 300 works on interpreting network internals (Räuker et al., 2022).
    The key promise of interpretability is to offer open-ended ways of understanding and evaluating models that help us with AI safety. And the diversity of approaches to interpretability is encouraging since we want build a toolbox full of many different useful techniques. But despite how much interpretability work is out there, the research has not been very good at producing competitive practical tools. Interpretability tools lack widespread use by practitioners in real applications (Doshi-Velez and Kim, 2017; Krishnan, 2019; Räuker et al., 2022).
    The root cause of this has much to do with interpretability research not being approached with as much engineering rigor as it ought to be. This has become increasingly well-understood. Here is a short reading list for anyone who wants to see more takes that are critical of interpretability research. This post will engage with each of these more below.
    The Mythos of Model Interpretability (Lipton, 2016)
    Towards A Rigorous Science of Interpretable Machine Learning (Doshi-Velez and Kim, 2017)
    Explanation in Artificial Intelligence: Insights from the Social Sciences (Miller, 2017)
    Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (Rudin, 2018)
    Against Interpretability: a Critical Examination of the Interpretability Problem in Machine Learning (Krishnan, 2019)
    Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks (Räuker et al., 2022)
    Benchmarking Interpretability Tools for Deep Neural Networks (Casper et al., 2023)
    Note that I’m an author on the final two, so references to these papers are self-references. Also my perspectives here are my own and should not be assumed to necessarily reflect those of coauthors.
    The goal of this post is to overview some broad limitations with interpretability research today. See also EIS V and EIS VI which discuss some similar themes in the context of AI safety and mechanistic interpretability research.
    The central problem: evaluation
    The hardest thing about conducting good interpretability research is that it’s not clear whether an interpretation is good or not when there is no ground truth to compare it to. Neural systems are complex, and it’s hard to verify that an interpretation faithfully describes how a network truly functions. So what does it even mean to be meaningfully interpreting a network? There is unfortunately no agreed upon standard. Motivations and goals of interpretability researchers are notoriously “diverse and discordant” (Lipton, 2018). But here, we will take an engineer’s perspective and consider interpretations to be good to the extent that they are useful.
    Evaluation by intuition is inadequate.
    Miller (2019) observes that “Most work in explainable artificial intelligence uses only the researchers’ intuition of what constitutes a ‘good’ explanation”. Some papers and posts have even formalized evaluation by intuition. Two examples are Yang et al. (2019) and Kirk et al. (2020) who proposed evaluation frameworks that included a criterion called “persuadability.” This was defined by Yang et al. (2019) as “subjective satisfaction or comprehensibility for the corresponding explanation.”
    This is not a very good criterion from an enginee...
    20 min
  • EA - How meat-free meal selection varies with menu options: an exploration by Sagar K Shah
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How meat-free meal selection varies with menu options: an exploration, published by Sagar K Shah on February 14, 2023 on The Effective Altruism Forum.
    Summary
    Increasing consumption of meat-free meals can help reduce demand for factory farmed animal products and anthropogenic greenhouse gas emissions. But relatively little research has been done on how meat-free meal selection is influenced by menu options, such as the availability of meat-analogue options or different types of meat.
    We conducted a preregistered reanalysis of data from a series of hypothetical discrete choice experiments from Brachem et al. (2019). We explored how meat-free meal selection by 1348 respondents (mostly German students) varied across 26 different menus, depending on the number of meat-free options and whether any options contained fish/poultry meat or meat-analogues. Menus consisted of five options (of which, two or three were meat-free) and were composed using images and descriptions of actual dishes available at restaurants at the University of Göttingen.
    While our work was motivated by causal hypotheses, our reanalysis was limited to detecting correlations and not causal effects. Specific limitations include:
    Examining hypotheses that the original study was not designed to evaluate.
    De facto observational design, despite blinded randomization in the original study.
    Possible non-random correlations between the presence of poultry/fish or meat-analogue menu options and the appealingness of other dishes.
    Analysis of self-reported, hypothetical meal preferences, rather than actual behavior.
    Meat-analogues in menus not reflecting prominent products attracting significant financial investment.
    Notwithstanding, our reanalysis found meat-free meal selection odds were:
    higher among menus with an extra meat-free option (odds ratio of 2.3, 90% CI [1.8 to 3.0]).
    lower among menus featuring poultry or fish options (odds ratio of 0.7, 90% CI [0.6 to 0.9]).
    not significantly associated with the presence of meat-analogues on a menu (odds ratio of 1.2 (90% CI [0.9 to 1.6])) in our preregistered meat-analogue definition. Estimates varied across analogue definitions, but were never significantly different from 1.
    Despite the many limitations, these findings might slightly update our beliefs to the extent we believe correlations would be expected if causation were occurring.
    The poultry/fish option correlation highlights the potential for welfare losses from substitution towards small-bodied animals from menu changes as well as shifts in consumer preferences.
    Given the study didn’t feature very prominent meat analogues, the absence of a correlation in this reanalysis cannot credibly be used to refute a belief that high-quality analogues play an important role in reducing meat consumption. But when coupled with the strong correlation on an additional meat-free option, we think the reanalysis highlights the need for further research on the most effective ways to encourage selection of meat-free meals. It remains an open question whether, at the margin, it would be more cost-effective to advocate for more menu options featuring meat-analogues specifically, or for more meat-free options of any kind.
    You can read the full post on the Rethink Priorities website, and also see the pre-print and code via the Open Science Framework..
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min