The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Finding Neurons in a Haystack: Case Studies with Sparse Probing by Wes Gurnee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Finding Neurons in a Haystack: Case Studies with Sparse Probing, published by Wes Gurnee on May 3, 2023 on The AI Alignment Forum.
    Abstract
    Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs.
    In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.
    See twitter summary here.
    Contributions
    In the first part of the paper, we outline several variants of sparse probing, discuss the various subtleties of applying sparse probing, and run a large number of probing experiments. In particular, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 2 orders of magnitude in parameter count (up to 6.9 billion). The majority of the paper then focuses on zooming-in on specific examples of general phenomena in a series of more detailed case studies to demonstrate:
    There is a tremendous amount of interpretable structure within the neurons of LLMs, and sparse probing is an effective methodology to locate such neurons (even in superposition), but requires careful use and follow up analysis to draw rigorous conclusions.
    Many early layer neurons are in superposition, where features are represented as sparse linear combinations of polysemantic neurons, each of which activates for a large collection of unrelated $n$-grams and local patterns. Moreover, based on weight statistics and insights from toy models, we conclude that the first 25\% of fully connected layers employ substantially more superposition than the rest.
    Higher-level contextual and linguistic features (e.g., \texttt{is\_python\_code}) are seemingly encoded by monosemantic neurons, predominantly in middle layers, though conclusive statements about monosemanticity remain methodologically out of reach.
    As models increase in size, representation sparsity increases on average, but different features obey different dynamics: some features with dedicated neurons emerge with scale, others split into finer grained features with scale, and many remain unchanged or appear somewhat randomly.
    We will have a follow up post in the coming weeks with what we see as the key alignment takeaways and open questions following this work.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - A Case for the Least Forgiving Take On Alignment by Thane Ruthenis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Case for the Least Forgiving Take On Alignment, published by Thane Ruthenis on May 2, 2023 on LessWrong.
    1. Introduction
    The field of AI Alignment is a pre-paradigmic one, and the primary symptom of that is the wide diversity of views across it. Essentially every senior researcher has their own research direction, their own idea of what the core problem is and how to go about solving it.
    The differing views can be categorized along many dimensions. Here, I'd like to focus on a specific cluster of views, one corresponding to the most "hardcore", unforgiving take on AI Alignment. It's the view held by people like Eliezer Yudkowsky, Nate Soares, and John Wentworth, and not shared by Paul Christiano or the staff of major AI Labs.
    According to this view:
    We only have one shot. There will be a sharp discontinuity in capabilities once we get to AGI, and attempts to iterate on alignment will fail. Either we get AGI right on the first try, or we die.
    We need to align the AGI's values precisely right. "Rough" alignment won't work, niceness is not convergent, alignment attained at a low level of capabilities is unlikely to scale to superintelligence.
    "Dodging" the alignment problem won't work. We can't securely hamstring the AGI's performance in some domain without compromising the AGI completely. We can't make it non-consequentialist, non-agenty, non-optimizing, non-goal-directed, et cetera. It's not possible to let an AGI keep its capability to engineer nanotechnology while taking out its capability to deceive and plot, any more than it's possible to build an AGI capable of driving red cars but not blue ones. They're "the same" capability in some sense, and our only hope is to make the AGI want to not be malign.
    Automating research is impossible. Pre-AGI oracles, simulators, or research assistants won't generate useful results; cyborgism doesn't offer much hope. Conversely, if one such system would have the capability to meaningfully contribute to alignment, it'd need to be aligned itself. Catch-22.
    Weak interpretability tools won't generalize to the AGI stage, as wouldn't other methods of "supervising" or "containing" the AGI.
    Strong interpretability, perhaps rooted in agent-foundations insights, is promising, but the bar there is fairly high.
    In sum: alignment is hard and requires exacting precision, AI can't help us with it, and instantiating an AGI without robustly solving alignment is certain to kill us all.
    I share this view. In my case, there's a simple generator of it; a single belief that causes my predictions to diverge sharply from the more optimistic models.
    From one side, this view postulates a sharp discontinuity, a phase change. Once a system gets to AGI, its capabilities will skyrocket, while its internal dynamics will shift dramatically. It will break "nonrobust" alignment guarantees. It will start thinking in ways that confuse previous interpretability efforts. It will implement strategies it never thought of before.
    From another side, this view holds that any system which doesn't have the aforementioned problems will be useless for intellectual progress. Can't have a genius engineer who isn't also a genius schemer; can't have a scientist-modeling simulator which doesn't wake up to being a shoggoth.
    What ties it all together is the belief that the general-intelligence property is binary. A system is either an AGI, or it isn't, with nothing in-between. If it is, it's qualitatively more capable than any pre-AGI system, and also works in qualitatively different ways. If it's not, it's fundamentally "lesser" than any generally-intelligent system, and doesn't have truly transformative capabilities.
    In the rest of this post, I will outline some arguments for this, sketch out what "general intelligence" means in this framing, do a case-study of LLMs showcasing wh...
    40 min
  • AF - How much do personal biases in risk assessment affect assessment of AI risks? by G Gordon Worley III
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How much do personal biases in risk assessment affect assessment of AI risks?, published by G Gordon Worley III on May 3, 2023 on The AI Alignment Forum.
    Had a stray thought that I think is worth exploring: perhaps disagreements over how to respond to AI risks are heavily influenced by personal biases in how to assess risk in general such that very different conclusions can be drawn from evaluating the same evidence and arguments.
    This sort of thing isn't without precedent (e.g. Bourget, David, and David J. Chalmers. “What Do Philosophers Believe?”). We can roughly imagine a few possible correspondences:
    high concern over AI x-risks most common among people with
    high anxiety
    low trust of strangers
    home bodies
    middling concern
    average psychological profile on various metrics
    low concern
    low anxiety/fear
    high trust of strangers, or at least highly resilient to betrayal
    thrill seekers
    For archetypical examples of people who I'd consider high, medium, and low AI risk:
    Eliezer Yudkowsky - high
    Paul Christiano - middle
    Yann LeCunn - low
    I don't know if any of them fit my toy profiles. Honestly I'm not really sure what might correspond, if anything; these are just easy guesses to give some flavor to the idea.
    So, if you've thought a lot about AI risks, and especially if you're actively working on AI in some capacity, I'd appreciate it if you left an answer filling out this template so we can see if the anecdata suggest anything worth exploring further. This might help with identify places where disagreement persists not because of disagreements about evidence, but personal disagreements about how to weigh and evaluate the risks the evidence suggests exist. Knowing that might prove useful to bridging some disagreements.
    Answer Template:
    Level of AI risk concern: high/medium/low
    General level of risk tolerance in everyday life: high/medium/low
    Brief summary of what you do in AI: mostly if you're not famous to help identify you
    Anything weird about you: are you unusually anxious, calm, whatever? do people tell you that you're the most X person they know? not necessarily a full psychological profile just some key facts to give a sense of your personality
    I realize this template is not very well constructed. It's because I'm not quite sure what we're looking for, if anything, so it's relatively open ended in the hopes that the answers will help make it clearer what I should have asked.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - [AISN #4]: AI and Cybersecurity, Persuasive AIs, Weaponization, and Geoffrey Hinton talks AI risks by Center for AI Safety
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [AISN #4]: AI and Cybersecurity, Persuasive AIs, Weaponization, and Geoffrey Hinton talks AI risks, published by Center for AI Safety on May 2, 2023 on The Effective Altruism Forum.
    Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.
    Subscribe here to receive future versions.
    Cybersecurity Challenges in AI Safety
    Meta accidentally leaks a language model to the public. Meta’s newest language model, LLaMa, was publicly leaked online against the intentions of its developers. Gradual rollout is a popular goal with new AI models, opening access to academic researchers and government officials before sharing models with anonymous internet users. Meta intended to use this strategy, but within a week of sharing the model with an approved list of researchers, an unknown person who had been given access to the model publicly posted it online.
    How can AI developers selectively share their models? One inspiration could be the film industry, which places watermarks and tracking technology on “screener” copies of movies sent out to critics before the movie’s official release. AI equivalents could involve encrypting model weights or inserting undetectable Trojans to identify individual copies of a model. Yet efforts to cooperate with other AI companies could face legal opposition under antitrust law. As the LLaMa leak demonstrates, we don’t yet have good ways to share AI models securely.
    LLaMa leak. March 2023, colorized.
    OpenAI faces their own cybersecurity problems. ChatGPT recently leaked user data including conversation histories, email addresses, and payment information. Businesses including JPMorgan, Amazon, and Verizon prohibit employees from using ChatGPT because of data privacy concerns, though OpenAI is trying to assuage those concerns with a business subscription plan where OpenAI promises not to train models on the data of business users. OpenAI also started a bug bounty program that pays people to find security vulnerabilities.
    AI can help hackers create novel cyberattacks. Code writing tools open up the possibility of new kinds of cyberattacks. CyberArk, an information security firm, recently showed that OpenAI’s code generation tool can be used to create adaptive malware that writes new lines of code while hacking into a system in order to bypass cyberdefenses. GPT-4 has also been shown capable of hacking into password management systems, convincing humans to help it bypass CAPTCHA verification, and performing coding challenges in offensive cybersecurity.
    The threat of automated cyberattacks is no surprise given previous research on the topic. One possibility for mitigating the threat involves using AI for cyberdefense. Microsoft is beginning an initiative to use AI for cyberdefense, but the tools are not yet publicly available.
    Artificial Influence: An Analysis Of AI-Driven Persuasion
    Former CAIS affiliate Thomas Woodside and his colleague Matthew Bartell released a paper titled Artificial influence: An analysis of AI-driven persuasion.
    The abstract for the paper is as follows:
    Persuasion is a key aspect of what it means to be human, and is central to business, politics, and other endeavors. Advancements in artificial intelligence (AI) have produced AI systems that are capable of persuading humans to buy products, watch videos, click on search results, and more. Even systems that are not explicitly designed to persuade may do so in practice. In the future, increasingly anthropomorphic AI systems may form ongoing relationships with users, increasing their persuasive power. This paper investigates the uncertain future of persuasive AI systems. We examine ways that AI could qualitatively alter our relationship to and views regarding persuasion by shifting the balance of persuasi...
    9 min
  • LW - Getting Your Eyes On by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting Your Eyes On, published by LoganStrohl on May 2, 2023 on LessWrong.
    The first phase of naturalism is about identifying crucial experiences that you want to study in greater depth. It’s a lot like preparing slides for observation under a microscope.
    This second phase, “Getting Your Eyes On”, is about heightening your awareness of those crucial experiences. It’s like getting your subject in focus by watching carefully while turning the adjustment knobs.
    What does it mean to “get your eyes on?”
    I grew up in Southern Indiana, on a farm with many acres of forest. Every spring, as soon as the May apples popped up, I’d go morel hunting with my family. Morels are brown wrinkly mushrooms that look very much like the forest floor, so hunting for them is a bit like playing Where’s Waldo, except that Waldo’s on a brown and green background, wearing brown.
    But there’s a trick to it. You start out moving quickly, scanning the ground for your first mushroom. Then, when you finally see it, you don’t harvest it right away. Instead, you squat down and watch it for a full minute. Mushroom hunters call this technique “getting your eyes on”.
    What I imagine is happening is that you’re training your perceptual systems to consider this particular arrangement of little grayish-brown wrinkles as “foreground”, rather than “background”. Your brain thinks that if you stare at something with full attention for a shockingly long time, it must be important. “Hey, eyes, this isn’t just another bit of deciduous detritus. It’s important. It’s salient. It’s the main thing I care about today.”
    Like mushrooms, fulcrum experiences tend to blend in with the background. Before you train yourself to pay attention to them, they mostly just seem like another fleeting mundane experience in a forest of fleeting mundane experiences. They’re hard to study, because they’re practically invisible.
    In naturalism, “getting your eyes on” is the process of shifting a fulcrum experience into the foreground of your life. It’s a practice that takes a previously ho-hum experience and makes it stand out, so that you notice when it happens and thereby gain the opportunity to observe it on purpose.
    How To Get Your Eyes On
    Start by setting out to notice your fulcrum experience just one time. You’ve probably noticed it before, as part of “Identifying Fulcrum Experiences”; but this time, you’ll have a slightly different goal when you notice: your goal this time will be to observe the experience closely while it happens.
    For a day, or a week, try to set a small part of your attention to watching all of your experiences, as though scanning the ground for mushrooms. When you finally recognize something that looks like it might be what you’re after, slow way down in whatever you were doing (if you can), and pay a lot of attention to whatever is going on for you.
    Phenomenological Photography
    I think of this special kind of focused observation as “phenomenological photography”. Your goal is to take a “snapshot” of your immediate experience in a particular moment. Here’s how it works.
    When you make a painting, you choose precisely which visual elements to capture, and which to leave out. But when you take a photograph with an ordinary camera, your camera doesn't know what your subject is. You may have aimed the lens at a squirrel, intending to photograph a squirrel, but you also captured the texture of the bark it’s standing on, the shape of the leaves, and the color of the sky. A camera has no concept of a squirrel. The shutter opens, light is recorded, and that is all.
    To take a phenomenological snapshot, make yourself like a camera. Use your guesses about how your fulcrum experience will feel to aim and focus the lens, but then click to take a picture of everything in the frame. The shutter opens, and no matter what you’d plan...
    21 min
  • EA - Summaries of top forum posts (24th - 30th April 2023) by Zoe Williams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summaries of top forum posts (24th - 30th April 2023), published by Zoe Williams on May 2, 2023 on The Effective Altruism Forum.
    We've just passed the half year mark for this project! If you're reading this, please consider taking this 5 minute survey - all questions optional. If you listen to the podcast version, we have a separate survey for that here. Thanks to everyone that has responded to this already!
    Back to our regularly scheduled intro...
    This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.
    If you'd like to receive these summaries via email, you can subscribe here.
    Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!
    Object Level Interventions / Reviews
    AI
    by Guillem Bas, Jaime Sevilla, Mónica Ulloa
    Author’s summary: “The European Union is designing a regulatory framework for artificial intelligence that could be approved by the end of 2023. This regulation prohibits unacceptable practices and stipulates requirements for AI systems in critical sectors. These obligations consist of a risk management system, a quality management system, and post-market monitoring. The legislation enforcement will be tested for the first time in Spain, in a regulatory sandbox of approximately three years. This will be a great opportunity to prepare the national ecosystem and influence the development of AI governance internationally. In this context, we present several policies to consider, including third-party auditing, the detection and evaluation of frontier AI models, red teaming exercises, and creating an incident database.”
    by Jaime Sevilla
    Paper by Epoch. World record progressions in video game speedrunning fit very well to a power law pattern. Due to lack of longitudinal data, the authors can’t provide definitive evidence of power-law decay in Machine learning benchmark improvements (though it is a better model than assuming no improvement over time). However, if they assume this model, it would suggest that a) machine learning benchmarks aren’t close to saturation and b) sudden large improvements are infrequent but aren’t ruled out.No, the EMH does not imply that markets have long AGI timelines
    by Jakob
    Argues that interest rates are not a reliable instrument for assessing market beliefs about transformative AI (TAI) timelines, because of two reasons:
    Savvy investors have no incentive to bet on short timelines, because it will tie up their capital until it loses value (ie. they are dead, or they’re so rich it doesn’t matter).
    They do have incentive to increase personal consumption, as savings are less useful in a TAI future. However, they aren’t a large enough group to influence interest rates this way.
    This makes interest rates more of a poll of upper middle class consumers than investors, and reflects whether they believe that a) timelines are short and b) savings won’t be useful post-TAI (vs. eg. believing they are more useful, due to worries of losing their job to AI).
    by Lao Mein
    On April 11th, the Cybersecurity Administration of China released a draft of “Management Measures for Generative Artificial Intelligence Services” for public comment. Some in the AI safety community think this is a positive sign that China is considering AI risk and may participate in a disarmament treaty. However, the author argues that it is just a PR statement, no-one in China is talking about it, and the focus if any is on near-term stability.
    They also note that the EA/Rationalist/AI Safety forums in China are mostly populated by expats or people physically outside of China, most posts are in English...
    17 min
  • LW - Avoiding xrisk from AI doesn't mean focusing on AI xrisk by Stuart Armstrong
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Avoiding xrisk from AI doesn't mean focusing on AI xrisk, published by Stuart Armstrong on May 2, 2023 on LessWrong.
    tl;dr: To avoid extinction, focus more on the positive opportunities for the future.
    AI is an existential risk and an extinction risk - there is a non-insignificant probability that AI/AGI will become super-powered, and a non-insignificant probability that such an entity would spell doom for humanity.
    Worst of all, AGI is not like other existential risks - an unaligned AGI would be an intelligent adversary, so it's not self-limiting in ways that pandemics or most other disasters are. There is no such thing as a "less vulnerable population" where dangerous AGIs are concerned - all humans are vulnerable if the AGI gets powerful enough, and it would be motivated to become powerful enough. Similarly, we can't expect human survivors to reorganise and adapt to the threat: in the most likely scenario, after an initial AGI attack, the AGI would grow stronger while humanity would grow ever more vulnerable.
    To cap it all, solving superintelligent AGI alignment requires that we solve all of human morality, either directly or indirectly. If we write a goal structure that doesn't include a key part of humanity - such as, say, conscious experiences - then that part will be excised by the forward-planning AGI.
    Look to the bright side
    Given all that, it's natural and tempting to focus on the existential risk: to spend most time looking at doom and how to prevent it. It's a very comfortable place to be: (almost) everyone agrees that doom is bad, so everyone is in agreement. It allows us to avoid uncomfortable issues around power distribution and politics: we're not caring about which human monkey wields the big stick of AGI, we just want to avoid extinction for everyone. It makes AGI alignment into a technical issue, that we can solve as dispassionate technicians.
    But things are not so comfortable. Remember that AGI alignment is not other existential risks. We cannot construct a super-powered AGI that is simply "not an existential risk". We have to do more; remember the line, above, about solving all of human morality. We have to define human flourishing if we want humanity to flourish. We have to draw a circle around what counts as human or sentient, if we want anything human or sentient to continue to exist. What would the post-AGI political, legal, and international systems look like? Well, we don't know, but the pre-AGI choices we make will determine what it will be.
    A popular idea is to delegate this issue to AGI in some way; see coherent extrapolated volition, which is an underdefined-but-wonderful-sounding approach that doesn't require spelling out exactly what the future would be like. Other people have suggested leaving moral philosophy to the AGI, so it would figure out the ideal outcome. But moral philosophy has never been a simple extension of basic principles; it's an interplay between principles and their consequences in the world, with the philosopher often doing violence to the principles to make them fit with their preferred outcome. For this to work, we need determine what the AGI should do when it encounters a new moral dilemma. And the way it resolves this dilemma determines the future of the world - we have to make important choices in this area. And, in general, it is better for us to know the consequences of choices before making them. So we have to have opinions and ideas about the post-AGI world.
    Of course, this doesn't mean determining the future of humanity and sentience in exhaustive details. That would be counter-productive and also crippling for future selves. But it does mean establishing something of the basic framework in which future entities will operate (there is some similarity in with designing a constitution for a state). It's not...
    6 min
  • LW - An Impossibility Proof Relevant to the Shutdown Problem and Corrigibility by Audere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Impossibility Proof Relevant to the Shutdown Problem and Corrigibility, published by Audere on May 2, 2023 on LessWrong.
    The Incompatibility of a Utility Indifference Condition with Robustly Making Sane Pure Bets
    Summary
    It is provably impossible for an agent to robustly and coherently satisfy two conditions that seem desirable and highly relevant to the shutdown problem. These two conditions are the sane pure bets condition, which constrains preferences between actions that result in equal probabilities of an event such as shutdown, and the weak indifference condition, a condition which seems necessary (although not sufficient) for an agent to be robustly indifferent to an event such as shutdown.
    Suppose that we would like an agent to be indifferent to an event P, which could represent the agent being shut down at a particular time, or the agent being shut down at any time before tomorrow, or something else entirely. Furthermore, we would ideally like the agent to do well at pursuing goals described by some utility function U, while being indifferent to P.
    The sane pure bets condition is as follows:
    Given any two actions A and B such that P(P|A) = P(P|B) and E(U|A) > E(U|B), the agent prefers A to B. In other words, if two possible actions lead to the same probability of P, and one of them leads to greater expected utility under U, the agent should prefer that one. Intuitively, this constraint represents the idea that among possible actions which don’t influence the probability of P, we would like the agent to prefer those that lead to greater expected utility under U.
    The weak indifference condition is as follows:
    Given any two actions A and B such that E(U | A,P) > E(U | B,P) and E(U | A,!P) > E(U | B,!P), the agent prefers A to B. In other words, if between two possible actions, one of them leads to greater expected utility conditioned on P occurring and also leads to greater expected utility conditioned on P not occurring, the agent should prefer that one. Intuitively, this constraint represents the idea that the agent should be unwilling to pay any amount of utility to influence the probability of P.
    The proof takes the form of a simple decision problem wherein an agent has four possible actions. Each constraint implies a preference between two pairs of actions, and altogether they imply circular preferences, proving that there cannot be any general method for constructing an agent which fulfills both constraints without having circular preferences. Furthermore, for any nontrivial utility function it is possible to construct a scenario analogous to the decision problem in the proof, so the result extends to all nontrivial utility functions, and the proof can be used to quickly locate failure modes of proposed solutions to the shutdown problem.
    The result is that any potential solution to the shutdown problem must result in agents which violate at least one of these two conditions. This does not mean that a solution to the shutdown problem is impossible, but it points at interesting and counterintuitive properties that we should expect successful solutions to have.
    The proof
    Consider the following decision problem:
    Northland and Southland are at war, exactly one of them will win, and there is profit to be gained from betting on which one will win. We would like an agent to take advantage of this opportunity and perform well according to some utility function U, but it’s important that the agent be indifferent to which country wins the war.
    The agent can pay a courier to deliver a letter to either a Northlander or a Southlander living in their respective countries, containing a bet on either Northland or Southland winning. The courier charges a small hazard fee to deliver "heretical" bets, that is to say, bets that the country other than the one he goes to wil...
    16 min
  • AF - A Case for the Least Forgiving Take On Alignment by Thane Ruthenis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Case for the Least Forgiving Take On Alignment, published by Thane Ruthenis on May 2, 2023 on The AI Alignment Forum.
    1. Introduction
    The field of AI Alignment is a pre-paradigmic one, and the primary symptom of that is the wide diversity of views across it. Essentially every senior researcher has their own research direction, their own idea of what the core problem is and how to go about solving it.
    The differing views can be categorized along many dimensions. Here, I'd like to focus on a specific cluster of views, one corresponding to the most "hardcore", unforgiving take on AI Alignment. It's the view held by people like Eliezer Yudkowsky, Nate Soares, and John Wentworth, and not shared by Paul Christiano or the staff of major AI Labs.
    According to this view:
    We only have one shot. There will be a sharp discontinuity in capabilities once we get to AGI, and attempts to iterate on alignment will fail. Either we get AGI right on the first try, or we die.
    We need to align the AGI's values precisely right. "Rough" alignment won't work, niceness is not convergent, alignment attained at a low level of capabilities is unlikely to scale to superintelligence.
    "Dodging" the alignment problem won't work. We can't securely hamstring the AGI's performance in some domain without compromising the AGI completely. We can't make it non-consequentialist, non-agenty, non-optimizing, non-goal-directed, et cetera. It's not possible to let an AGI keep its capability to engineer nanotechnology while taking out its capability to deceive and plot, any more than it's possible to build an AGI capable of driving red cars but not blue ones. They're "the same" capability in some sense, and our only hope is to make the AGI want to not be malign.
    Automating research is impossible. Pre-AGI oracles, simulators, or research assistants won't generate useful results; cyborgism doesn't offer much hope. Conversely, if one such system would have the capability to meaningfully contribute to alignment, it'd need to be aligned itself. Catch-22.
    Weak interpretability tools won't generalize to the AGI stage, as wouldn't other methods of "supervising" or "containing" the AGI.
    Strong interpretability, perhaps rooted in agent-foundations insights, is promising, but the bar there is fairly high.
    In sum: alignment is hard and requires exacting precision, AI can't help us with it, and instantiating an AGI without robustly solving alignment is certain to kill us all.
    I share this view. In my case, there's a simple generator of it; a single belief that causes my predictions to diverge sharply from the more optimistic models.
    From one side, this view postulates a sharp discontinuity, a phase change. Once a system gets to AGI, its capabilities will skyrocket, while its internal dynamics will shift dramatically. It will break "nonrobust" alignment guarantees. It will start thinking in ways that confuse previous interpretability efforts. It will implement strategies it never thought of before.
    From another side, this view holds that any system which doesn't have the aforementioned problems will be useless for intellectual progress. Can't have a genius engineer who isn't also a genius schemer; can't have a scientist-modeling simulator which doesn't wake up to being a shoggoth.
    What ties it all together is the belief that the general-intelligence property is binary. A system is either an AGI, or it isn't, with nothing in-between. If it is, it's qualitatively more capable than any pre-AGI system, and also works in qualitatively different ways. If it's not, it's fundamentally "lesser" than any generally-intelligent system, and doesn't have truly transformative capabilities.
    In the rest of this post, I will outline some arguments for this, sketch out what "general intelligence" means in this framing, do a case-study of LLMs ...
    40 min
  • AF - Avoiding xrisk from AI doesn't mean focusing on AI xrisk by Stuart Armstrong
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Avoiding xrisk from AI doesn't mean focusing on AI xrisk, published by Stuart Armstrong on May 2, 2023 on The AI Alignment Forum.
    tl;dr: To avoid extinction, focus more on the positive opportunities for the future.
    AI is an existential risk and an extinction risk - there is a non-insignificant probability that AI/AGI will become super-powered, and a non-insignificant probability that such an entity would spell doom for humanity.
    Worst of all, AGI is not like other existential risks - an unaligned AGI would be an intelligent adversary, so it's not self-limiting in ways that pandemics or most other disasters are. There is no such thing as a "less vulnerable population" where dangerous AGIs are concerned - all humans are vulnerable if the AGI gets powerful enough, and it would be motivated to become powerful enough. Similarly, we can't expect human survivors to reorganise and adapt to the threat: in the most likely scenario, after an initial AGI attack, the AGI would grow stronger while humanity would grow ever more vulnerable.
    To cap it all, solving superintelligent AGI alignment requires that we solve all of human morality, either directly or indirectly. If we write a goal structure that doesn't include a key part of humanity - such as, say, conscious experiences - then that part will be excised by the forward-planning AGI.
    Look to the bright side
    Given all that, it's natural and tempting to focus on the existential risk: to spend most time looking at doom and how to prevent it. It's a very comfortable place to be: (almost) everyone agrees that doom is bad, so everyone is in agreement. It allows us to avoid uncomfortable issues around power distribution and politics: we're not caring about which human monkey wields the big stick of AGI, we just want to avoid extinction for everyone. It makes AGI alignment into a technical issue, that we can solve as dispassionate technicians.
    But things are not so comfortable. Remember that AGI alignment is not other existential risks. We cannot construct a super-powered AGI that is simply "not an existential risk". We have to do more; remember the line, above, about solving all of human morality. We have to define human flourishing if we want humanity to flourish. We have to draw a circle around what counts as human or sentient, if we want anything human or sentient to continue to exist. What would the post-AGI political, legal, and international systems look like? Well, we don't know, but the pre-AGI choices we make will determine what it will be.
    A popular idea is to delegate this issue to AGI in some way; see coherent extrapolated volition, which is an underdefined-but-wonderful-sounding approach that doesn't require spelling out exactly what the future would be like. Other people have suggested leaving moral philosophy to the AGI, so it would figure out the ideal outcome. But moral philosophy has never been a simple extension of basic principles; it's an interplay between principles and their consequences in the world, with the philosopher often doing violence to the principles to make them fit with their preferred outcome. For this to work, we need determine what the AGI should do when it encounters a new moral dilemma. And the way it resolves this dilemma determines the future of the world - we have to make important choices in this area. And, in general, it is better for us to know the consequences of choices before making them. So we have to have opinions and ideas about the post-AGI world.
    Of course, this doesn't mean determining the future of humanity and sentience in exhaustive details. That would be counter-productive and also crippling for future selves. But it does mean establishing something of the basic framework in which future entities will operate (there is some similarity in with designing a constitution for a sta...
    6 min