The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

The Nonlinear Library episodes

  • EA - International AI Institutions: a literature review of models, examples, and proposals by MMMaas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: International AI Institutions: a literature review of models, examples, and proposals, published by MMMaas on September 27, 2023 on The Effective Altruism Forum.
    The Legal Priorities Project has published a new report (link, PDF, SSRN) surveying models, different examples, and proposals for international institutions for AI governance.
    This literature review examines a range of institutional models that have been proposed over the year for the international governance of AI. The review specifically focuses on proposals that would involve the creation of new international institutions for AI. As such, it focuses on seven models for international AI institutions with distinct functions. These models are:
    Scientific consensus building
    Political consensus-building and norm-setting
    Coordination of policy and regulation
    Enforcement of standards or restrictions
    Stabilization and emergency response
    International joint research
    Distribution of benefits and access
    Part I consists of the literature review. For each model, we provide (i) a description of each model's functions and types; (ii) the most common examples of each model; (iii) some under-explored examples that are not (often) mentioned in the AI governance literature but that show promise; (iv) a review of proposals for the application of that model to the international regulation of AI; and (v) critiques of the model both generally and in its potential application to AI.
    Part II briefly discusses some considerations for further research concerning the design of international institutions for AI, including the effectiveness of each model at accomplishing its aims; treaty-based regulatory frameworks; other institutional models not covered in this review; the compatibility of institutional functions; and institutional options to host a new international AI governance body.
    Overall, the review covers seven institutional models, as well as more than thirty-three commonly invoked examples of those models, twenty-two additional examples, and forty-seven proposals of new AI institutions based on those models. Table 1 summarizes these findings.Table 1: Overview of institutional models, examples, and proposed institutions surveyed
    ModelScientific consensus- buildingPolitical consensus- building and norm-settingCoord. of policy and regulationEnforcement of standards or restrictionsStabilization and emergency responseIntern. joint researchDistribution of benefits and access
    Common examples
    Under- explored examples
    Proposed AI institutions
    IPCC
    IPBES
    SAP
    CEP
    WMO
    IPAI
    Commission on Frontier AI
    Intergovernmental Panel on Information Technology
    COPs (e.g. UNFCCC COP)
    OECD
    G20
    G7
    ISO
    IEC
    ITU
    Various soft law instruments
    Lysøen Declaration
    Codex Alimentarius Commission
    BRICS
    IAIO
    Emerging Technology Coalition
    IAAI
    Data Governance Structure
    Data Stewardship Organization
    International Academy for AI Law and Regulation
    WTO
    ICAO
    IMO
    IAEA
    FATF
    UNEP
    ILO
    UNESCO
    EMEP
    World Bank
    IMF
    WSIS
    Advanced AI Governance Organisation
    IAIO
    EU AI Agency
    GAIA
    Generative AI global governance body
    Coordinator and Catalyser of International AI Law
    IAEA (Department of Safeguards)
    Nuclear Suppliers Group
    Wassenaar Arrangement
    Missile Technology Control Regime
    Open Skies Consultative Commission
    Atomic Development Authority
    OPCW
    BWC Implementation Unit
    IMO
    CITES Secretariat
    UN AI control agency
    Global watchdog agency
    International Enforcement Agency
    Emerging Technologies Treaty
    IAIA (multiple)
    UN Framework Convention on AI (UNFCAI) & Protocol on AI, supported by Intergovernmental Panel on AI, AI GLobal Authority, and supervisory body
    Advanced AI Governance Organization
    AIEA for Superintelligence
    NPT+
    Multilateral AI governance initiative
    International AI Safety Agency
    Advanced AI chips registry
    Code of conduct for state behavior
    AI CBMs
    Open Skies for AI
    Bilateral...
    6 min
  • LW - GPT-4 for personal productivity: online distraction blocker by Sergii
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPT-4 for personal productivity: online distraction blocker, published by Sergii on September 27, 2023 on LessWrong.
    There are many apps for blocking distracting websites: freedom.to, leechblock, selfcontrol, coldturkey, just to name a few. They are useful for maintaining focus, avoiding procrastination, and curbing addictive web surfing.
    They work well for blocking a list of a few distracting websites. For me, this is not enough, because I'm spending a large portion of my time on a large number of websites, which I check out for a minute or two and then never visit again. It's just impossible to maintain a blocklist for this long tail. Also, the web has grown so much that there are just too many easily found alternatives for any blocked distraction.
    Well, GPT-4 to the rescue! With an LLM it's possible to block websites based on the content, checking each page - if it's distracting or useful/productive.
    To test the idea I have implemented a prototype of a distraction filtering browser extension. This way, GPT-4 is turning into a personal productivity assistant!
    The extension sends the content of each loaded page to OpenAI API, and asks GPT if the page should be blocked. The prompt can be edited in the config window; the following prompt is used by default:
    Sensitive content, whitelist & blacklist.
    While the extension is active, it sends a sample of each visited page's content to OpenAI API. This might be a problem for pages with sensitive content.
    You can add any domains which you do not want to expose to OpenAI to the whitelist or the blacklist. Pages that are matched are allowed or blocked without sending content to OpenAI.
    OpenAI is claiming to handle user data securely, and to not use data submitted via API for model training. Still, if you have any concerns about the privacy and security of the pages that you visit, and if you do not want to risk leaking your browsing history, avoid using this extension.
    Installation and testing
    To try it out, download the extension (github.com/coolvision/awf/releases/download/0.1/awf-0.1.zip), install it (instructions), and enter your API key in the extension's config page.
    Then, navigate to any page, it might get blocked:
    Does it work?
    I have been using it for a few days, and it does work quite well, with correct decisions in most cases.
    One problem is that GPT-4 is expensive, and my usage has been up to ~$1/day. It would probably cost $10-30/month, which is not too much, but still a thing to improve.
    Another issue is that OpenAI API is quite slow, it takes several seconds (up to 50-10s) to validate each page. I haven't decided yet if it's a feature or a problem - on one hand, it does make web browsing more mindful which is good, but then it does kill the flow/momentum when I want to quickly research something.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • EA - GWWC's new community strategy by Giving What We Can
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GWWC's new community strategy, published by Giving What We Can on September 27, 2023 on The Effective Altruism Forum.
    I'm extremely excited to announce the launch of our new community strategy that aims to connect our existing community better, and inspire more people to join us - I've been working on this for the past few months and am thrilled to finally share it!
    Over the almost two years I've been working with Giving What We Can, I've seen how important our community is - whether it's a supportive network to ask questions, a friendly face in a new city, or a shared sense that none of us are alone in wanting to do what we can to make a better world, I've personally taken much-needed inspiration, motivation, and comfort more times than I can count, simply by hearing from you!
    I want to make it easier for you to do the same.
    We are launching some new spaces for you to meet, connect, and chat with each other. Everyone is welcome: whether you just subscribe to our newsletter, have taken the pledge, have donated through our platform, or are simply interested in meeting like-minded people who care about making a difference in the world:
    In brief, Giving What We Can's new community strategy includes:
    The launch of GWWC Local Groups in key cities like London, NYC, Berlin, and Sydney; we plan to launch in other cities as well
    An online community space for you to connect, no matter where you are located
    A directory of other (non-GWWC) effective giving groups, to better connect the effective giving community more broadly
    I firmly believe that community and connection are an integral part of working towards a society where giving significantly and effectively is a cultural norm. We've heard from many of you that one of the best things about having found Giving What We Can is learning that there are other people with similar values. We hope this makes it a little easier to connect!
    TL;DR: How do I get involved?
    Join one of the GWWC Local groups below or apply to start one in your city!
    London
    New York
    Berlin
    Vancouver
    Sydney
    Melbourne
    You can read more about each of our new initiatives below:
    1: GWWC Local Groups
    Since its founding in 2009, Giving What We Can has been connecting people through their commitment to effective giving. Our mission isn't just to marginally increase the amount of money going to effective charities; we're aiming for meaningful cultural change - with a goal to make giving effectively and significantly a global norm, in service of a better world. The community we've built (and are continuing to build) is instrumental to reaching that goal.
    The way that GWWC has engaged with the community has changed a lot over the years. It all started as small local groups bound by a common thread of pledging 10% back in 2009 through "Giving What We Can Chapters".
    One of the biggest changes that occurred was GWWC Chapters transitioning to become EA Groups several years ago.
    Since then, we've worked with EA groups to promote effective giving, had volunteers host monthly virtual meetups, and run an ambassador program where our ambassadors ran events too!
    Some highlights:
    PISE (Positive Impact Society Erasmus) has hosted pledge celebration events.
    An ambassador John Yan has hosted Giving Tuesday after-parties in New York.
    Grace (Head of Marketing) ran a community event in Melbourne earlier this year.
    Since GWWC was revitalised in 2020, we have often received requests to restart local groups, given that some of its strongest growth historically came from our former Chapters. Many of our smaller initiatives (e.g. one-off events, working with EA groups, and running the ambassador program) have shown a desire for this too.
    We've also been excited to see successful communities around effective giving start to grow and adapt to local cultural contexts i.e. in London and in the ...
    14 min
  • EA - Inside the Mind of an Aspiring Charity Entrepreneur [Follow Along] #1 - From Layoff to Co-founding in a Breathtaking Two Months by Harry Luk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inside the Mind of an Aspiring Charity Entrepreneur [Follow Along] #1 - From Layoff to Co-founding in a Breathtaking Two Months, published by Harry Luk on September 27, 2023 on The Effective Altruism Forum.
    TL;DR
    I want to extend a MASSIVE, heartfelt thank you to the abundant resources put together by the EA community and the wonderful EAs who are so generous with their knowledge and time (even when talking to a complete newbie like me).
    Because of you, I went from being laid off in June and knowing nothing about EA in July, to being offered a nonprofit co-founder position in early September, to finishing some initial groundwork of setting up a charity (productivity system, Founder's agreement, funding proposal draft, etc) in late September.
    The initial groundwork implemented thus far were all influenced by the action steps found in How to Launch a High-Impact Nonprofit. Thus, this aims to be a post sequence dedicated to putting the book's concepts to work and showing some actual work examples.
    Hopefully with more of your support and feedback, the StakeOut.AI charity startup can keep a good steady pace and accomplish more crucial milestones in Oct and onwards.
    As this post is the intro of the sequence, it details more specifically my journey (learning and doing applications) - including important tips for other applicants (apply early, handling rejections) and some feedback for EA organizations taking applications (setup to process many applications, send responders a copy of their responses, informing those who didn't make the cut).
    I hope this brings value to the EA community as this is perspective from someone very green in EA, eager to take action quickly even though I came from a totally different world before.
    Acknowledgment
    I would like to give a special thank you to Dr. Peter S. Park for editing this post and for all our future collaborations! There is also a I'm so grateful for everyone who has been a part of the journey thus far section later in this post.
    Introduction
    Hi, my name is Harry, it's nice to meet all of you. This post, my first ever post on the EA forum, is inspired by Three lessons I've learnt from starting a small EA org and Why and how to start a pilot project in EA - so thank you Ben Williamson.
    The purpose of this post is to [1] document my journey publicly (hopefully to inspire others who are thinking they want to start a side project, or apply to CE, or go down the path of charity entrepreneurship), [2] maximize feedback as per Ben's post (both directly through the post (in the comments) and indirectly (people who reach out privately with additional suggestions/ advice)), and [3] share key lessons & milestones to hopefully contribute to the forum via actual work examples.
    The hope is that it is going to be a follow along post sequence, as my co-founder Dr. Peter S. Park and I get our nonprofit startup up and running.
    Though the journey with EA has only been 2 months, I already have so many people to thank who have answered questions, provided feedback and pointed me in the right direction. I am forever grateful for you :)
    The Catalyst: The Layoff that Sparked My Journey
    I first found out about Effective Altruism from my Google search about impactful tithing. As a Christian, I have been tithing 10% of my income for many years, but because I was laid off from shortage of work in my previous marketing role, I was interested in finding more impactful ways to give due to our now lowered family income.
    This is when I stumbled upon GivingMultiplier.org and was introduced to the idea of super-effective charities. I was intrigued by their approach where you can still give from the heart (your favorite charity), but at the same time incentivizes you to give more to super-effective charities by matching your donations (the higher % you give to super...
    28 min
  • EA - A Primer for Insect Sentience and Welfare (as of Sept 2023) by Meghan Barrett
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Primer for Insect Sentience and Welfare (as of Sept 2023), published by Meghan Barrett on September 27, 2023 on The Effective Altruism Forum.
    This post was written by me, Meghan Barrett, in my independent capacity as an academic research scientist and entomologist. None of the organizations with which I'm affiliated should be taken to endorse or support any particular conclusions or resources listed herein based on this post.
    Introduction
    If you attended my talk at EAG London in May 2023, you may remember this basic narrative:
    Insects might matter morally. There are a lot of them. We can use scientific evidence to make their lives better.
    This is the quick case I provide for working on insect welfare. Since that talk, I've been encouraged by the amount of interest in the topic among members of the EA community, so many of whom want to learn about insects and their welfare in farmed, wild, and research contexts. The lives and capabilities of our planet's ~5.5 million species of insects are often surprising and quite poorly understood (even by entomologists!), which can lead us all to make empirically-unsupported assumptions about their sentience, capacity for welfare, and welfare concerns. Although there are many significant unknowns in the science of insect sentience and welfare, it is clear that if we want to help insects, we need to learn what we can about them.
    Lots of advocates are doing just that: they're putting in the work to understand insects' nervous systems, behavior, and physiology - as well as the scale and contexts of their use and management. I'm heartened to see people take insects seriously. So, as an insect neurobiologist and physiologist by training, I want to do my part to make it easier to learn about these fascinating, diverse, and highly neglected animals.
    This post is a quick, non-exhaustive, and lightly-annotated list of resources that can serve as a primer for folks interested in getting up to speed on insect pain, sentience, and welfare as of September 2023. For future readers, it also points toward some places where people can go for the most recent information on insect welfare and sentience.
    I hope this guide is useful for introducing you to the topic - and for demonstrating that there is a lot of rigorous empirical, or otherwise expert, conversation currently happening on the topics of insect sentience and welfare.
    Quick caveats
    The welfare-focused work on this list (as compared to the pain and sentience research) is skewed toward the work of Rethink Priorities, my collaborators' publications, and my own efforts. I'm obviously biased, but I think these folks have done much of the most rigorous work in the space to date.
    The list is biased towards biological information over, say, economic models or philosophical considerations. I'm a biologist. It's also probably biased towards neurobiology and physiology over, say, ethology (though I've tried hard to include behavioral resources, too).
    The list is intentionally non-exhaustive - it's a primer, not a research database (if you want that, check out this link here) - so you shouldn't expect these resources to provide a complete overview of everything you might need to contribute meaningfully to the conversation on insect sentience or welfare.
    This list won't be updated regularly. It's the list 'as of September 2023'. The insect welfare and sentience space is starting to move faster, so this may be seriously out of date within a year or two.
    Not all work on this list is peer-reviewed (at least in the traditional, academic sense).
    Inclusion doesn't equal endorsement. Instead, inclusion on this list is an expression of confidence that either (1) there's something of value to the work conducted therein or (2) it is an important part of the history and debate of the discipline.
    Insect Sentienc...
    15 min
  • EA - Vegans in the UK: are you getting enough iodine? by Stan Pinsent
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vegans in the UK: are you getting enough iodine?, published by Stan Pinsent on September 26, 2023 on The Effective Altruism Forum.
    This post was written very quickly by a non-expert. Do your own reading before changing your diet!
    You should make sure you get iodine:
    Unlike most countries, table salt in the UK is not iodised
    Most Brits get plenty of iodine from dairy
    Deficiency risk factors include being vegan, being female
    Iodine deficiency while (pre) pregnant or breastfeeding can cause IQ drop and even cretinism in the child
    For everyone else, symptoms are usually mild
    Buy iodised salt or even supplements. Careful, you can get too much.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • LW - Making AIs less likely to be spiteful by Nicolas Macé
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Making AIs less likely to be spiteful, published by Nicolas Macé on September 26, 2023 on LessWrong.
    Which forms of misalignment might result in particularly bad outcomes? And to what extent can we prevent them even if we fail at intent alignment? We define spite as a terminal preference for frustrating others' preferences, at least under some conditions. Reducing the chances that an AI system is spiteful is a candidate class of interventions for reducing risks of AGI conflict, as well as risks from malevolence. This post summarizes some of our thinking on the topic. We give an overview of why spite might lead to catastrophic conflict; how we might intervene to reduce it; ways in which the intervention could fail to be impactful, or have negative impact; and things we could learn that would update us on the value of this intervention.
    Key takeaways
    Spiteful preferences include a generalized preference for harming others, as well as other preferences like vengefulness and spite towards certain groups. The basic reason to focus on reducing spite is that such interventions may stably make AIs less likely to take risks of mutually costly conflict (or deliberately create suffering because they intrinsically value it), even if alignment fails. (more)
    Spite might be selected for in ML systems because (a) it serves as a strategically valuable commitment device, (b) it is a direct proxy for high-scoring behavior in environments where the optimal behavior involves harming other agents (e.g., environments with competition between agents), (c) it is (correctly or incorrectly) inferred from human preferences, or (d) it results from miscellaneous generalization failures. (more)
    Thus potentially low-cost interventions to reduce the chances of spite include modifications to the training data or loss function to reduce selection pressure towards spite (e.g., avoiding selecting agents based on relative performance in multi-agent tasks, or filtering human feedback that could select for spite). (more)
    Reducing spite carries some prima facie backfire risk, via potentially increasing the exploitability of AIs that share human values to some extent. We currently don't think that this consideration makes the sign of spite reduction negative, however. One reason is that, for interventions to backfire by making our AI more exploitable, they have to change other agents' beliefs about how spiteful our AI is, and there are various reasons to doubt this will happen.
    Interventions to reduce spite seem most likely to be counterfactual in worlds where alignment fails. However, it's currently very unclear to us how an AI's goals will relate to design features we can intervene on (e.g., training environments), conditional on alignment failure.
    If we are in a world in which inner objectives are highly path-dependent, we need it to be the case that (i) we can reliably influence motivations formed early in training and that (ii) these motivations reliably influence the agent's final goals. (more)
    If we are in a world in which inner objectives are not very highly path-dependent, we need it to be the case that deceptive alignment isn't favored by models' inductive biases, and that the agent's inner objective generalizes off-distribution as intended. (more)
    Our future work: We are not confident that it will be possible to reduce spite in misaligned AIs. However, (i) based on our reading of the alignment literature and conversations with alignment researchers, it doesn't seem strongly ruled out by the current understanding of alignment, and (ii) there may be spite interventions that are particularly cheap and easy to persuade AI developers to implement. Thus we think it's worthwhile to continue to devote some of our portfolio to spite for now.
    Our conceptual work on this topic will likely include:
    Developing...
    29 min
  • EA - Net global welfare may be negative and declining by kyle fish
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Net global welfare may be negative and declining, published by kyle fish on September 26, 2023 on The Effective Altruism Forum.
    Overview
    The total moral value of the world includes humans as well as all other beings of moral significance. As such, a picture of the overall trajectory of net global welfare that accounts for both human and non-human populations is important context for thinking about the future on any timescale, and about the potential impacts of transformative technologies.
    There's compelling evidence that life has gotten better for humans recently, but the same can't be said for other animals, especially given the rise of industrial animal agriculture. How do these trends cash out, and what's the overall trajectory?
    I've used human and farmed animal population data, estimates of welfare ranges across different species, and estimates of the average wellbeing of different species to get a rough sense of recent trends in total global welfare. Importantly, this initial analysis is limited to humans and some of the most abundant farmed animals - it does not consider effects on insects or wild animals, the inclusion of which could plausibly change the top-line conclusions (see e.g. here). I focus on the years from 1961-2021, as this is the period for which the most reliable data exists, and the period most relevant to understanding the current trajectory.
    My tentative conclusion is that net global welfare may be both negative and declining. That is, the entire good of humanity may be outweighed by the cumulative suffering of farmed animals, with total animal suffering growing faster than human wellbeing is increasing, especially in recent decades. Below I lay out some of the many assumptions upon which this work depends, the core of my analysis, and some tentative reflections on how these findings shape my thinking about the future.
    Notes and Assumptions
    This analysis was performed as a quick/rough estimate, and should not be mistaken for a comprehensive treatment of the topic. I am highly uncertain about both the quantitative findings and my interpretations.
    I opted for using point estimates rather than confidence intervals for the sake of simplicity. My confidence intervals would be wide if they were included. That said, I've aimed to make my estimates such that they fall at or below the median of my best-guess distributions (i.e. for the parts of the analysis that relied on my own judgment calls, I think it's more likely than not that I've underestimated animal suffering relative to humans, rather than overestimated).
    I've limited the scope of this analysis to humans and farmed animals, in part due to data availability, in part for simplicity, and in part to focus on humanity's most direct impacts on other beings. Inclusion of wild animals in this analysis could plausibly change the signs of my conclusions.
    This work draws heavily on the Moral Weight Project from Rethink Priorities and relies on the same assumptions: utilitarianism, hedonism, valence symmetry, unitarianism, use of proxies for hedonic potential, and more. Although I think the Rethink Priorities welfare range estimates are currently the best tool available for interspecies welfare comparisons, I do not necessarily endorse these assumptions in full, nor do I think the Rethink Priorities welfare ranges are the "correct" weights - only the best available. I consider the following entries in the Moral Weight Project Sequence to be particularly useful background reading:
    An Introduction to the Moral Weight Project
    Rethink Priorities' Welfare Range Estimates
    Don't Balk at Animal-friendly Results
    Analysis
    Background and Definitions
    The central concept of my analysis is that the total welfare of a given species in a given year can be calculated as follows:
    Welfare Capacity = Population Welfare Ran...
    15 min
  • AF - Announcing the CNN Interpretability Competition by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the CNN Interpretability Competition, published by Stephen Casper on September 26, 2023 on The AI Alignment Forum.
    TL;DR
    I am excited to announce the CNN Interpretability Competition, which is part of the competition track of SATML 2024.
    Dates: Sept 22, 2023 - Mar 22, 2024
    Competition website:/
    Total prize pool: $8,000
    NeurIPS 2023 Paper: Red Teaming Deep Neural Networks with Feature Synthesis Tools
    Github:
    For additional reference: Practical Diagnostic Tools for Deep Neural Networks
    Correspondence to: [email protected]
    Intro and Motivation
    Interpretability research is popular, and interpretability tools play a role in almost every agenda for making AI safe. However, there are some gaps between the research and engineering applications. If one of our main goals for interpretability research is to help us align highly intelligent AI systems in high-stakes settings, we need more tools that help us better solve practical problems.
    One of the unique advantages of interpretability tools is that, unlike test sets, they can sometimes allow humans to characterize how networks may behave on novel examples. For example, Carter et al. (2019), Mu and Andreas (2020), Hernandez et al. (2021), Casper et al. (2022a), and Casper et al. (2023) have all used different interpretability tools to identify novel combinations of features that serve as adversarial attacks against deep neural networks.
    Interpretability tools are promising for exercising better oversight, but human understanding is hard to measure, and it has been difficult to make clear progress toward more practically useful tools. Here, we work to address this by introducing the CNN Interpretability Competition (accepted to SATML 2024).
    The key to the competition is to develop interpretations of the model that help human crowdworkers discover trojans: specific vulnerabilities implanted into a network in which a certain trigger feature causes the network to produce unexpected output. In addition, we also offer an open-ended challenge for participants to discover the triggers for secret trojans by any means necessary.
    The motivation for this trojan-discovery competition is that trojans are bugs caused by novel trigger features -- they usually can't be identified by analyzing model performance on some readily available dataset. This makes finding them a challenging debugging task that mirrors the practical challenge of finding unknown bugs in models. However, unlike naturally occurring bugs in neural networks, the trojan triggers are known to us, so it will be possible to know when an interpretation is causally correct or not. In the real world, not all types of bugs in neural networks are likely to be trojan-like. However, benchmarking interpretability tools using trojans can offer a basic sanity check.
    The Benchmark
    This competition follows new work from Casper et al. (2023) (will be at NeurIPS 2023), in which we introduced a benchmark for interpretability tools based on helping human crowdworkers discover trojans that had interpretable triggers. We used 12 trojans of three different types: ones that were triggered by patches, styles, and naturally occurring features.
    We then evaluated 9 methods meant to help detect trojan triggers: TABOR, (Guo et al., 2019), four variants of feature visualizations, (Olah et al., 2017; Mordvintsev et al., 2018), adversarial patches (Brown et al., 2017), two variants of robust feature-level adversaries (Casper et al., 2022a), and SNAFUE (Casper et al., 2022b). We tested each based on how much they helped crowdworkers identify trojan triggers in multiple-choice questions. Overall, this work found some successes. Adversarial patches, robust feature-level adversaries, and SNAFUE were relatively successful at helping humans discover trojan triggers.
    However, even the best-perfo...
    7 min
  • EA - AMA: Andy Weber (U.S. Assistant Secretary of Defense from 2009-2014) by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Andy Weber (U.S. Assistant Secretary of Defense from 2009-2014), published by Lizka on September 26, 2023 on The Effective Altruism Forum.
    Andy Weber was the U.S. Assistant Secretary of Defense for Nuclear, Chemical & Biological Defense Programs from 2009 to 2014. He's now a senior fellow at the Council on Strategic Risks. You might also know him from his appearance on the 80,000 Hours Podcast. Ask him anything!
    He'll try to answer some questions on Friday, September 29 (afternoon, Eastern Time), and might get to some earlier.
    I (Lizka) am particularly excited that Andy can share his experience in nuclear (and other kinds of) threat reduction given that it is Petrov Day today.
    Instructions and practical notes:
    Please post your questions as comments on this post.
    Posting questions earlier is better than later.
    If you have multiple questions, it might be better to post them separately.
    Feel free to upvote questions that others have posted, as it might help prioritize questions later.
    Other context and topics that might be especially interesting to talk about:
    Risks of "tactical" nuclear weapons like the new sea-launched cruise missile (Reuters)
    Andy's experience with Project Sapphire and the Nunn-Lugar program
    Andy's thoughts on biosecurity and preventing bioweapons use
    For those who want to explore more: The Dead Hand by David Hoffman might be interesting; Project Sapphire and some of the work against biological threats are captured in it.
    He might not get to some questions, or be unable to answer some.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min

About The Nonlinear Library

From the publisher's feed

The Nonlinear Library allows you to easily listen to top EA and rationalist content on your podcast player. We use text-to-speech software to create an automatically updating repository of audio…