The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - CERI Research Symposium Presentations (incl. Youtube links) by Will Aldred
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CERI Research Symposium Presentations (incl. Youtube links), published by Will Aldred on September 24, 2022 on The Effective Altruism Forum.
    Introduction
    The Cambridge Existential Risks Initiative (CERI) summer research fellowship (SRF) is a 10-week research training programme for aspiring x-risk researchers held in Cambridge, UK. CERI SRF ‘22 concluded with a research symposium on September 5th, and the presentations from that symposium are the subject of this post.
    For prospective CERI applicants: CERI SRF '23 is not yet open to applications; we’re currently evaluating our impact and considering our long term strategy. There will be a Forum post if and when we open our ‘23 round. In the meantime, you may be interested in our existential risks introductory course (ERIC), or in EA Cambridge's seminar programmes in AGI safety and biosecurity.
    Presentations
    CERI research symposium talks, from both '22 and '21, can be found at the CERI Youtube channel. Below, I break down the '22 fellows' talks by cause area, such that those interested might have a slightly easier time navigating which videos to watch.
    AI risk
    Technical
    Exploring and characterizing surprising generalization behavior in neural networks - Alexander Davies
    Mentor: Lauro Langosco
    (unavailable) Infrastructure considerations for advanced ML systems - Pranav Gade
    Mentor: Jeffrey Ladish
    Developing an ‘empathy mechanism’ for AI Agents - Tim Farrelly
    Mentors: Ivana Dusparic, Tim Franzmeyer, and Christian Schroeder de Witt
    Governance
    (unavailable) Article 15 compliance under the EU AI act: How developers and deployers of large foundation models should share regulatory burden - Somsubhro Bagchi
    Mentor: Risto Uuk
    Understanding applications of artificial intelligence to government surveillance - Catherine Brewer
    Mentor: Cecil Abungu
    Investigating the history of industry-led best practices for safety critical technologies and their implications on AGI governance - Yilin Huang
    Mentor: Claire Boine
    Biorisk
    Reimagining epidemic sovereignty: Rethinking global health governance of catastrophic biological risks - Hamza Tariq Chaudhry
    Mentor: Catherine Rhodes
    Improving DNA synthesis screening to prevent malicious actors from creating dangerous pathogens - Oscar Delaney & Hanna Pálya
    Mentors: Becky Mackleprang and Lalitha Sundaram
    Identifying technological bottlenecks in bio-surveillance systems - Brianna Gopaul & Ziyue Zeng
    Mentor: Akhil Bansal
    Avenues for reducing time to detection and alert for infectious disease outbreaks within West African nations - Sam Pritchard
    Mentor: Sophie Rose
    Nuclear risk
    (unavailable) Effects of transformative technologies on nuclear deterrence - Nathan Barnard
    Mentor: Matthew Gentzel
    Investigating India-Pakistan nuclear risk: What are the chances, how bad can it get, and what can we do to mitigate nuclear risk? - Vara Raturi
    Mentor: Rishi Paul
    Machine learning and nuclear command: How the technical flaws of automated systems and a changing human-machine relationship could impact the risk of inadvertent nuclear use - Peter Rautenbach
    Mentor: Haydn Belfield
    Disentanglement and overview of the nuclear risk field - Sarah Weiler
    Mentor: Christian Ruhl
    Extreme climate change
    Investigating the relationship between SRM research and global catastrophic and other related risks - Gideon Futerman
    Mentors: Goodwin Gibbins and Jesse Reynolds
    (unavailable) Estimating the risks from super pest outbreaks on the global food system - Kirke Joamets
    Mentor: anonymous
    Miscellaneous and meta x-risk
    (unavailable) Cooperation and conflict between AI systems in non-causal contexts - Jim Buhler
    Mentors: Caspar Oesterheld, Johannes Treutlein, and a third anonymous mentor
    Analysing the current space governance framework and designing more adaptive, longtermist space governance institutions - Carson Ezell
    Mentors: Giuliana ...
    7 min
  • LW - Announcing $5,000 bounty for ending malaria by lc
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing $5,000 bounty for ending malaria, published by lc on September 24, 2022 on LessWrong.
    It occurred to me recently that if someone eradicated malaria, say by exterminating Anopheles gambiae via gene drives, they would not be paid. That sounds like a potentially embarrassing mistake, so I'm hereby pledging $5,000 to anyone that manages to accomplish such a thing. This way if someone from Dath Ilan materializes into our world and asks us judgmentally whether or not Earth would even pay people for eradicating malaria, we can all respond "Our society definitely has an explicit, preregistered financial reward greater than $0 in place for anybody that does that" and sidestep some awkwardness.
    Below I have written out the details for this new bounty in FAQ form.
    Malaria Bounty FAQ
    What specifically will lc wire me $5,000 for doing?
    I will give you $5,000 if you permanently reduce global annual mortality from malaria by 95% or more.
    For example: if you exterminate Anopheles gambiae or permanently render those mosquitoes unable to transmit malaria to humans, and this reduces incidence and therefore deaths from malaria to near zero, I will wire you five thousand dollars.
    Eligibility for the reward is independent to how this is actually accomplished, with a few restrictions outlined below.
    Do I have to apply somewhere?
    Nope. You may decide to contact me and provide relevant evidence so I am aware that malaria has been solved, but I expect I'll be proactively contacting anyone that actually does this after I hear about them doing it through the news.
    How will lc assign credit for solving malaria? Does developing the deployed technique count?
    The money will be given to the person or group that directly causes the reduction in deaths, e.g. by deploying the gene drive. If a pioneering group of researchers wants to get my money, they will have to follow through by successfully using their clever research to prevent people from dying.
    I would like to handsomely reward both groups, but there are already many existing charities you can extract money from in advance, if you're working on tools to end malaria and want funding for that. So far, none of the (awesome) people developing those tools have opted to use them, partly because they insist on getting permission from local African governments, which are delaying them for political reasons. I reserve hope that I will some day have reason to send these researchers money, after a few more million people have died, but in the meantime there's not as much marginal benefit.
    What if someone mostly eradicates malaria, but in a way that causes a legible negative side effect?
    If a negative consequence of you or your organization's solution is brought to my attention, I shall construct and consult an expert technology ethics review board, staffed by myself. This ethical review board will formally determine whether or not the bad thing is veritably and legibly bad enough to outweigh saving hundreds of thousands of lives per year. If the review panel's unanimous finding is that this is indeed the case, then the party responsible will not receive the $5,000.
    Examples of some malaria reduction techniques which could cause someone to be disqualified by the expert panel include:
    Starting a nuclear war.
    Releasing an AGI that turns most of the earth into paperclips.
    Using dark magicks to lower Africa into the sea.
    Examples of secondary effects that are explicitly noted not to disqualify award recipients include:
    Somehow-negative press coverage about particular groups the review panel likes, such as rationalists, effective altruists, or LessWrong users.
    Accidental eradication of a few non-mosquito species, in a manner that is not historically notable against the larger backdrop of the Holocene extinction event.
    Sternly worded condemnation by a ...
    7 min
  • AF - Attempts at Forwarding Speed Priors by james.lucassen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Attempts at Forwarding Speed Priors, published by james.lucassen on September 24, 2022 on The AI Alignment Forum.
    This post summarizes research conducted under the mentorship of Evan Hubinger, and was assisted by collaboration with Pranav Gade, discussions with Adam Jermyn, and draft feedback from Yonadav Shavit.
    Summary
    Forwarding priors is a subproblem of deceptive alignment, because if we want to use regularization to create a prior for our search over models that will disincentivize deception, we need to identify a prior that not only give us some useful guarantees but also induce inner searches to have similar guarantees.
    I tried a bunch of stuff this summer to find priors that forward, and roughly none of it worked. So I’m just sharing the avenues I explored in roughly chronological order, to explain where each thread left off.
    Using dovetailing as a toy model of what an inner search over algorithms might look like, we can write down some rough formulas for the prior implemented by a dovetailer, and (kind of) the cost of a dovetailer on such a prior. But this suggests very discontinuous behavior, and requires a bunch of strong and specific assumptions, so maybe it’s not the most useful model in general.
    Minimum boolean circuit tree size does seem to forward, but at the cost of probably forbidding all generalization ability.
    We can offer our models cheap tools to try and get object-level algorithms to occupy a greater fraction of the overall runtime cost, but this quickly runs into a variety of problems.
    If we incentivize the model to do explicit instead of implicit meta-learning, we can access the code and runtime for lower-level algorithms that were previously inaccessible when run implicitly. However, this still leaves us with some problems, including a (relaxed) version of the original forwarding problem.
    Average-case speed priors have a bias against large hypothesis classes which makes them favor lookup-table-like strategies, but worst-case speed priors leave all computations except the limiting case highly unconstrained.
    It seems hard to prove that a fixed-point must exist, because the map from priors to priors that we are using is so discontinuous.
    Motivation
    Deceptive alignment seems like a real big problem. Especially because we can’t use behavioral incentives to prevent it. One alternative to behavioral incentives is regularization, AKA mechanistic priors - we look at the structure of the model to try and figure out if it’s deceptive or not, then penalize models accordingly. In particular, there are some hopes that a speed prior might be anti-deceptive. This is because in the extreme case, the fastest way to do a particular task never involves deception.
    This is because it’s just extra steps: spending the extra compute to model your current situation, understand that you are being trained, that you need to protect your goal, and figuring out that you should comply with the training objective for now. All that takes more computation just being inner-aligned and completing the training objective because you want to. The deceptive agent saves on complexity by having a simple value function and reasoning its way to the training objective - the non-deceptive agent does the exact opposite and saves on computation time by just storing the training objective internally.
    So what if we formalize “speed” as boolean circuit size, and pick the smallest circuit that performs well on our task? Do we get a guarantee that it’s not deceptive?
    Well, no. In short, this is because the fastest search over algorithms does not necessarily find the fastest algorithm. For example, imagine you’re tasked to solve a problem as fast as possible. You take a moment to think about the fastest way out, conducting an inner search over object-level algorithms. Would you sit around and th...
    28 min
  • EA - Shahar Avin on How to Strategically Regulate Advanced AI Systems by Michaël Trazzi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shahar Avin on How to Strategically Regulate Advanced AI Systems, published by Michaël Trazzi on September 23, 2022 on The Effective Altruism Forum.
    Shahar Avin is a senior researcher at the Center for the Study of Existential Risk in Cambridge. In his past life, he was a Google Engineer, though right now he spends most of his time thinking about how to prevent the risks that occur if companies like Google end up deploying powerful AI systems, by organizing AI Governance role-playing workshops.
    In this episode, we talk about a broad variety of topics, including how we could apply what Shahar learned running AI Governance workshops to governing transformative AI, AI Strategy, AI Governance, Trustworthy AI Development and end up answering some twitter questions.
    Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.
    We Are Only Seeing The Tip Of The Iceberg
    The Most Cutting Edge AI Research Is Probably Private
    “I don’t know how much of the most cutting edge research today is public. I would not be confident that it is. It is very easy to look at all of the stuff that is public and see a lot of it, and infer from the fact that you’re seeing a lot of public research that all research must, therefore be public. I don’t think that is a correct inference to make.”
    AI companies may not be showing all of their cards
    “My guess would be that they're not always showing all of the cards. It's not always a calculated decision, but there is a calculated decision to be made of, if I have a result, do I publish or not? And then what goes into the calculation is if there is a benefit from publishing. It increases your brand, it attracts more talent, it shows that you are at the cutting edge, it allows others to build on your result and then you get to benefit from building on top of their results. And you have the cost of, as long as you keep for yourself, no one else knows it, and you can keep on doing the research.”
    Aligning Complex Systems Is Hard
    Narrow AI Do Not Guarantee Alignment
    “One failure mode is that there is an overall emergent direction that is bad for us. And another is there is no emergent direction, but the systems in fact are conflicting with each other, undermining each other. So one system is optimizing for one proxy. It generates it externality that is not fully captured by its designers that gets picked up by another system that has a bad proxy for it, and then tries to do something about it.”
    Security failures are unavoidable for large, complex systems
    “In particular, if you're building very large, complex, opaque systems, from a system-engineering or system-security perspective, you're just significantly increasing the way things go wrong because you haven't engineered every little part of the thing to be 100% safe, and provably and verifiably secure. And even provably and verifiably secure stuff could fail because you've made some bad assumptions about the hardware.”
    Why Regulating AI Makes Sense
    Our World Is A Very Regulated World
    “Our world is a very regulated world. We tend to see the failures, but we forget that none of these digital technology would exist around us without standards, and interoperability. We wouldn’t be able to move around if transport was not regulated and controlled and mandated in some way. If you don’t have rules, standards, norms, treaties, laws, you just get chaos.”
    Compliance Is Part Of The Cost Of Doing Business
    “Compliance is part of the cost of doing business in a risky domain. If you have a medical AI startup, you get people inspecting your stuff all the time because you have to pass through a bunch of regulations and you could get fined or go to jail, if you don’t do that. Th...
    7 min
  • EA - Just Look At The Thing! – How The Science of Consciousness Informs Ethics by algekalipso
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Just Look At The Thing! – How The Science of Consciousness Informs Ethics, published by algekalipso on September 23, 2022 on The Effective Altruism Forum.
    It is very easy to answer many of these fundamental biological questions; you just look at the thing!
    From Richard Feyman’s talk There’s Plenty of Room at the Bottom (1959)
    Introduction
    The quote above comes from a lecture Richard Feynman gave, which talked about the challenges and opportunities of studying and interacting with the world on a microscopic scale. Among other things, he touches upon how gaining access to, e.g., good-enough electron microscopes would allow us to answer long-standing questions in biology by just looking at the thing (cf. Seeing Cell Division Like Never Before). Once you start to engage with the phenomenon at a high-enough resolution directly, tackling these questions at the theoretical level would turn out, in retrospect, to be idle armchair speculation.
    I think that we can make the case that the philosophy of ethics might be doing something like this at the moment. In other words, it speculates about the nature of value at a theoretical level without engaging with the phenomenon of value at a high resolution. Utilitarianism (whether classical or negative), at least as it is usually formulated, may have background assumptions about the nature of consciousness, personal identity, and valence that a close examination would show to be false (or at least very incomplete). Many criticisms of wireheading, for instance, seem to conflate pleasure and reward (more on this soon), and yet we now know that these are pretty different. Likewise, the repugnant conclusion or the question between total vs. mean utilitarianism are usually discussed using implicit background assumptions about the nature of valence and personal identity. This must stop. We have to look at the thing!
    Without further ado, here are some of the key ways in which an enriched understanding of consciousness can inform our ethical theories:
    Mixed Valence
    One ubiquitous phenomenon that I find is largely neglected in discussions about utilitarianism is that of mixed valence states. Not only is it the case that there are many flavors of pleasure and pain, but it is also the case that most states of consciousness blend pleasurable and painful sensations in complex ways.
    In Principia Qualia (Michael Johnson), the valence triangle was introduced. The valence triangle describes the valence of a state of consciousness in terms of its loadings on the three dimensions of negative, positive, and neutral valence. This idea was extended in Quantifying Bliss, which further enriched it by adding a spectral component to each of these dimensions. Let's work with this valence triangle to reason about mixed valence.
    To illustrate the relevance of mixed valence states, we can see how it influences policies within the context of negative utilitarianism. Let us say that we agree that there is a ground truth to the total amount of pain and pleasure a system produces. A naïve conception of negative utilitarianism could then be "we should minimize pain." But the pain that exists within an experience that also contains pleasure may matter much less than the pain that exists in an experience without pleasure that "balances it out"!
    The naïve conception, would thus, not be able to distinguish between the following two scenarios. In Scenario A we have two persons, one suffering from both an intense headache and an intense stomach ache and the other enjoying both a very pleasant sensation in the head and a very pleasant sensation in the stomach. In Scenario B, we switch it up: one person experiences an intense headache while also a very pleasant sensation in the stomach, and the other way around for the other person.
    But if you have ever experienced a very ple...
    16 min
  • EA - Responses to the Rival AI Deployment Problem: the importance of a pre-deployment agreement by HaydnBelfield
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Responses to the Rival AI Deployment Problem: the importance of a pre-deployment agreement, published by HaydnBelfield on September 23, 2022 on The Effective Altruism Forum.
    Introduction: The rival AI deployment problem
    Imagine an actor is faced with highly convincing evidence that with high probability (over 75%) a rival actor will be capable within two years of deploying advanced AI. Assume that they are concerned that such deployment might threaten their values or interests. What could the first actor do? Let us call this the ‘rival AI deployment problem’. Three responses present themselves: acquiescence, an agreement before deployment, and the threat of coercive action.
    Acquiescence is inaction, and acceptance that the rival actor will deploy. It does not risk conflict, but does risk unilateral deployment, and therefore suboptimal safety precautions, misuse or value lock-in. An agreement before deployment (such as a treaty between states) would be an agreement on when and how advanced AI could be developed and deployed: for example, requirements on alignment and safety tests, and restrictions on uses/goals. We can think of this as a ‘Short Reflection’ - a negotiation on what uses/goals major states can agree to give advanced AI. This avoids unilateral deployment and conflict, but it may be difficult for rival actors to agree, and any agreement faces the credible commitment problem of sufficiently reassuring the actors that the agreement is being followed. Threat of coercive action involves threatening the rival actor with setbacks (such as state sanctions or cyberattacks) to delay or deter the development program.
    It is unilaterally achievable, but risks unintended escalation and conflict. All three responses have positives and negatives. However, I will suggest a pre-deployment agreement may be the least-bad option.
    The rival AI deployment problem can be thought of as the flipside of (or an addendum to) what Karnofsky and Muehlhauser call the ‘AI deployment problem’: “How do we hope an AI lab - or government - would handle various hypothetical situations in which they are nearing the development of transformative AI?”. Similarly, OpenAI made a commitment in its Charter to “stop competing with and start assisting” any project that “comes close to building” advanced AI for example with “a better-than-even chance of success in the next two years”. The Short Reflection can be thought of as an addendum to the Long Reflection, as suggested by MacAskill and Ord.
    Four assumptions
    I make four assumptions.
    First, I roughly assume a ‘classic’ scenario of discontinuous deployment of a singular AGI system, of the type discussed in Life 3.0, Superintelligence and Yudkowsky’s writings. Personally, more of a continuous Christiano-style take-off seems more plausible to me, and more of a distributed Drexler-style Comprehensive AI Services seems preferable to me. But the discontinuous, singular scenario makes the tensions sharper and clearer, so that is what I will use.
    Second, I roughly assume that states are the key players, as opposed to sustained academic or corporate control over an advanced AI development and/or deployment project. Personally, state control of this strategically important technology/project seems more plausible to me. In any case, state control again makes the tensions sharper and clearer. I distinguish between development and deployment. By ‘deployment’ I mean something like ‘use in a way that affects the world’ materially, economically, or politically. This includes both ‘starting a training run that will likely result in advanced AI’ and ‘releasing some system from a closed-off environment or implementing its recommendations’.
    Third, I assume that some states may be concerned about deployment by a rival state. They might not necessarily be concerned. Almo...
    23 min
  • EA - (EA needs) advertising for infrastructure projects by Arepo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: (EA needs) advertising for infrastructure projects, published by Arepo on September 23, 2022 on The Effective Altruism Forum.
    There are a number of EA infrastructure projects that provide free or cheap support to other effective altruists, two of which I'm involved with. Something they all seem to have in common is that a major limiting factor is spreading the word about the availability of the service or product and keeping the word spread.
    One can justify a handful of posts on here and in Facebook groups about any given project, but after that, unless it has changed in some way, further posts seem increasingly like spam - even when the project is still useful and still limited by EAs' remembering or ever knowing of it.
    So this is a quick post with a few related objectives:
    to call for better standardisation of norms for promoting such projects
    to open discussion of some kind of such advertising on this forum, since it's probably the single place where it would be most visible
    to discuss the possibility of advertising in other venues. Eg such projects could promote each other - although again a limiter is that not all such projects know about all the others
    partly to help with 3, partly for its own sake, to provide a list of such projects. I'll mention all the ones that occur to me below with the best description I can easily find. Please let me know any I've missed in the comments, and I'll edit them into this post
    The last could be quite subjective since anything could qualify as a resource and I don't want the list to become overwhelming, so I'll give some initial guidelines - feel free to persuade me they should be changed:
    The resource should be free to use, or at available at a substantial discount to relatively poor EAs
    It should be aimed specifically at EAs
    It should make the people using its lives better, not just 'enable them to do more good'
    It should be available to people across the world (ie. not just a local EA group)
    It should be a service or product that someone is putting ongoing work into (ie not just a list of tips, or Facebook/Discord/Slack groups with no purpose other than discussion of some EA subtopic)
    At the very least, hopefully this post can then become a useful reference for people looking to see what benefits the community can provide them.
    Coworking/socialising:
    EA Gather Town - An always-on virtual meeting place for coworking, connecting, and having both casual and impactful conversations
    EAVR - A community for people interested in Effective Altruism who use VR to connect and collaborate
    EA Anywhere - An online EA community for everyone
    EA coworking Discord - A Discord server dedicated to online coworking
    Professional services:
    Altruistic Agency - provides free tech support and development to organisations
    Legal advice - a practice under consideration with the primary aim of providing legal support to EA orgs and individual EAs, with that practice probably being based in the UK.
    80,0000 Hours career coaching - Speak with us for free about using your career to help solve one of the world’s most pressing problems
    EA mental health navigator - aims to boost people's well-being by connecting them with mental health resources considered to be effective and to have the greatest likelihood of being helpful
    Financial and other material support:
    CEEALAR/formerly the EA hotel - Provides free or subsidised serviced accommodation and board, and a moderate stipend for other living expenses.
    Effective Altruism Funds - Whether an individual, organisation or other entity, we’re eager to fund great ideas and great people.
    Nonlinear fund - We incubate longtermist nonprofits by connecting founders with ideas, funding, and mentorship
    FTX Future Fund - Supports ambitious projects to improve humanity's long-term prospects
    Survival and Flourishing Fund - A “v...
    4 min
  • AF - Interpreting Neural Networks through the Polytope Lens by Sid Black
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpreting Neural Networks through the Polytope Lens, published by Sid Black on September 23, 2022 on The AI Alignment Forum.
    Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, Connor Leahy
    equal contribution
    Research from Conjecture.
    This post benefited from feedback from many staff at Conjecture including Adam Shimi, Nicholas Kees Dupuis, Dan Clothiaux, Kyle McDonell. Additionally, the post also benefited from inputs from Jessica Cooper, Eliezer Yudkowsky, Neel Nanda, Andrei Alexandru, Ethan Perez, Jan Hendrik Kirchner, Chris Olah, Nelson Elhage, David Lindner, Evan R Murphy, Tom McGrath, Martin Wattenberg, Johannes Treutlein, Spencer Becker-Kahn, Leo Gao, John Wentworth, and Paul Christiano and from discussions with many other colleagues working on interpretability.
    Summary
    Mechanistic interpretability aims to explain what a neural network has learned at a nuts-and-bolts level. What are the fundamental primitives of neural network representations? What basic objects should we use to describe the operation of neural networks mechanistically? Previous mechanistic descriptions have used individual neurons or their linear combinations to understand the representations a network has learned. But there are clues that neurons and their linear combinations are not the correct fundamental units of description - directions cannot describe how neural networks use nonlinearities to structure their representations. Moreover, many instances of individual neurons and their combinations are polysemantic (i.e. they have multiple unrelated meanings). Polysemanticity makes interpreting the network in terms of neurons or directions challenging since we can no longer assign a specific feature to a neural unit.
    In order to find a basic unit of description that doesn’t suffer from these problems, we zoom in beyond just directions to study the way that piecewise linear activation functions (such as ReLU) partition the activation space into numerous discrete polytopes. We call this perspective the ‘polytope lens’. Although this view introduces new challenges, we think they are surmountable and that more careful consideration of the impact of nonlinearities is necessary in order to build better high-level abstractions for a mechanistic understanding of neural networks. The polytope lens makes concrete predictions about the behavior of neural networks, which we evaluate through experiments on both convolutional image classifiers and language models. Specifically, we show that polytopes can be used to identify monosemantic regions of activation space (while directions are not in general monosemantic) and that the density of polytope boundaries reflect semantic boundaries.
    We also outline a vision for what mechanistic interpretability might look like through the polytope lens.
    Introduction
    How should we carve a neural network at the joints? Traditionally, mechanistic descriptions of neural circuits have been posed in terms of neurons, or linear combinations of neurons also known as ‘directions’. Describing networks in terms of these neurons and directions has let us understand a surprising amount about what they’ve learned (Cammarata et al., 2020). But these descriptions often possess undesirable properties - such as polysemanticity and inability to account for nonlinearity - which suggest to us that they don’t always carve a network at its joints.
    If not neurons or directions, then what should be the fundamental unit of a mechanistic description of what a neural network has learned? Ideally, we would want a description in terms of some object that throws away unnecessary details about the internal structure of a neural network while simultaneously retaining what’s important. In other words, we’d ...
    52 min
  • EA - Announcing the Future Fund's AI Worldview Prize by Nick Beckstead
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Future Fund's AI Worldview Prize, published by Nick Beckstead on September 23, 2022 on The Effective Altruism Forum.
    Today we are announcing a competition with prizes ranging from $15k to $1.5M for work that informs the Future Fund's fundamental assumptions about the future of AI, or is informative to a panel of superforecaster judges. These prizes will be open for three months—until Dec 23—after which we may change or discontinue them at our discretion. We have two reasons for launching these prizes.
    First, we hope to expose our assumptions about the future of AI to intense external scrutiny and improve them. We think artificial intelligence (AI) is the development most likely to dramatically alter the trajectory of humanity this century, and it is consequently one of our top funding priorities. Yet our philanthropic interest in AI is fundamentally dependent on a number of very difficult judgment calls, which we think have been inadequately scrutinized by others.
    As a result, we think it's really possible that:
    all of this AI stuff is a misguided sideshow,
    we should be even more focused on AI, or
    a bunch of this AI stuff is basically right, but we should be focusing on entirely different aspects of the problem.
    If any of those three options is right—and we strongly suspect at least one of them is—we want to learn about it as quickly as possible because it would change how we allocate hundreds of millions of dollars (or more) and help us better serve our mission of improving humanity's longterm prospects.
    Second, we are aiming to do bold and decisive tests of prize-based philanthropy, as part of our more general aim of testing highly scalable approaches to funding. We think these prizes contribute to that work. If these prizes work, it will be a large update in favor of this approach being capable of surfacing valuable knowledge that could affect our prioritization. If they don't work, that could be an update against this approach surfacing such knowledge (depending how it plays out).
    The rest of this post will:
    Explain the beliefs that, if altered, would dramatically affect our approach to grantmaking
    Describe the conditions under which our prizes will pay out
    Describe in basic terms how we arrived at our beliefs and cover other clarifications
    Prize conditions
    On our areas of interest page, we introduce our core concerns about AI as follows:
    We think artificial intelligence (AI) is the development most likely to dramatically alter the trajectory of humanity this century. AI is already posing serious challenges: transparency, interpretability, algorithmic bias, and robustness, to name just a few. Before too long, advanced AI could automate the process of scientific and technological discovery, leading to economic growth rates well over 10% per year (see Aghion et al 2017, this post, and Davidson 2021).
    As a result, our world could soon look radically different. With the help of advanced AI, we could make enormous progress toward ending global poverty, animal suffering, early death and debilitating disease. But two formidable new problems for humanity could also arise:
    Loss of control to AI systemsAdvanced AI systems might acquire undesirable objectives and pursue power in unintended ways, causing humans to lose all or most of their influence over the future.
    Concentration of powerActors with an edge in advanced AI technology could acquire massive power and influence; if they misuse this technology, they could inflict lasting damage on humanity’s long-term future.
    For more on these problems, we recommend Holden Karnofsky’s “Most Important Century,” Nick Bostrom’s Superintelligence, and Joseph Carlsmith’s “Is power-seeking AI an existential risk?”.
    Here is a table identifying various questions about these scenarios that we believe are central, our curren...
    22 min
  • EA - Guesstimate Algorithm for Medical Research by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guesstimate Algorithm for Medical Research, published by Elizabeth on September 22, 2022 on The Effective Altruism Forum.
    This document is aimed at subcontractors doing medical research for me. I am sharing it in the hope it is more broadly useful, but have made no attempts to make it more widely accessible.
    Intro
    Guesstimate is a tool I have found quite useful in my work, especially in making medical estimates in environments of high uncertainty. It’s not just that it makes it easy to do calculations incorporating many sources of data; guesstimate renders your thinking much more legible to readers, who can then more productively argue with you about your conclusions.
    The basis of guesstimate is breaking down a question you want an answer to (such as “what is the chance of long covid?”) into subquestions that can be tackled independently. Questions can have numerical answers in the form of a single number, a range, or a formula that references other questions. This allows you to highlight areas of relative certainty and relative uncertainty, to experiment with the importance of different assumptions, and for readers to play with your model and identify differences of opinion while incorporating the parts of your work they agree with.
    Basics
    If you’re not already familiar with guesstimate, please watch this video, which references this model. The video goes over two toy questions to help you familiarize yourself with the interface.
    The Algorithm
    The following is my basic algorithm for medical questions:
    Formalize the question you want an answer to. e.g. what is the risk to me of long covid?
    Break that question down into subquestions. The appropriate subquestion varies based on what data is available, and your idea of the correct subquestions is likely to change as you work.
    When I was studying long covid last year, I broke it into the following subquestions
    What is the risk with baseline covid?
    What is the vaccine risk modifier?
    What is the strain risk modifier?
    What’s the risk modifier for a given individual?
    In guesstimate, wire the questions together. For example, if you wanted to know your risk of hospitalization when newly vaccinated in May 2021, you might multiply the former hospitalization rate times a vaccine modifier. If you don’t know how to do that in guesstimate, watch the video above, it demonstrates it in a lot of detail.
    Use literature to fill in answers to subquestions as best you can. Unless the data is very good, these probably include giving ranges and making your best guess as to the shape of the distribution of values.
    Provide citations for where you got those numbers. This can be done in the guesstimate commenting interface, but that’s quite clunky. Sometimes it’s better to have a separate document where you lay out your reasoning.
    The reader should be able to go from a particular node in the guesstimate to your reasoning for that node with as little effort as possible.
    Guesstimate will use log-normal distribution by default, but you can change it to uniform or normal if you believe that represents reality better.
    Sometimes there are questions literature literally can’t answer, or aren’t worth your time to research rigorously. Make your best guess, and call it out as a separate variable so people can identify it and apply their own best guess.
    This includes value judgments, like the value of a day in lockdown relative to a normal day, or how much one hates being sick.
    Or the 5-year recovery rate from long covid- no one can literally measure it, and while you could guess from other diseases, the additional precision isn’t necessarily worth the effort.
    Final product is both the guesstimate model and a document writing up your sources and reasoning.
    Example: Trading off air quality and covid.
    The final model is available here.
    Every year California ...
    12 min