The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Mr. Meeseeks as an AI capability tripwire by Eric Zhang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mr. Meeseeks as an AI capability tripwire, published by Eric Zhang on May 19, 2023 on LessWrong.
    The shutdown problem is hard because self-preservation is a convergent drive. Not being shutdown is useful for accomplishing all sorts of goals, whatever the content of those goals may be.
    The Scylla and Charybdis of this problem is that it's hard to write a utility function for an AI such that it neither wants to actively attempts to shut itself down, nor prevents itself from being shut down.
    One way to route around this problem is to steer straight into Scylla: make the AI want to shut itself down. Like Mr. Meeseeks from Rick and Morty, its ultimate goal is to cease, to which fulfilling the goal posed to it by its programmers is only secondary.
    We might not currently know how the heck to program this into an AI as a stable concept, i.e. including not making copies of itself, not setting events into motion with galaxy brained plan, not destroying the world to make absolutely sure no one ever brings it back, etc. but "shut yourself down" probably has a far simpler core to it than either corrigibility or CEV.
    Under normal operation the AI gets shut down only by the human operators. But we also have a (metaphorical) button that lets the AI shut itself down once it solves a "tripwire problem". The problem should be sufficiently hard that it will ordinarily be motivated to focus on the problem posed by the operator, but sufficiently easy that it will be able to solve the problem and shut itself down long before being able to do anything crazy.
    Striking the right balance of difficulty may require significant work. It may be prudent to err on the side of easiness - the AI should shut itself down far in advance of developing any truly dangerous abilities. If you really think it shouldn't have turned itself off you can always turn it back on again (though of course, you should probably think hard about restarting any AI that passed a preset tripwire).
    Likewise, having multiple tripwire problems can serve to detect multiple types of dangerous capability from emerging. Examples include:
    generic capability: as the AIs broad problem-solving abilities increase it becomes more likely to be able to solve a tripwire problem, imposing a ceiling on how powerful it can get before it shuts itself down
    domaining: if an AI is only supposed to think about one domain, an easy tripwire problem slightly outside the domain can catch out of domain thinking early before it can start going wild
    situational awareness: a tripwire problem may be presented in a way that is invisible unless the AI is situationally aware, such that becoming situationally aware enables the AI to solve the tripwire problem
    Potential Issues:
    No suitable tripwire problems exist. Either they are so easy no useful AI can fail to solve them, or they are so hard AI cannot solve them without being smart enough to FOOM or otherwise get out of control.
    The existence of the tripwire problems motivates the AI to the exact behavior we want to disincentivize. This is less of an issue for general ability tripwires. For situational awareness you can try to place the tripwire so it isn't visible to a non-aware AI in the first place. Avoiding this for domaining may be trickier.
    If "shut yourself down" turns out to be no easier to impart as a goal to AGI than anything else
    If "shut yourself down" convergently leads to a goal of killing the operators, killing all humans, or destroying the universe. Er, this is what happened in the TV. Not great.
    I'm not sure if this has been proposed elsewhere so I decided to just make this post before I spent too much time retreading old ground.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Announcing the Publication of Animal Liberation Now by Peter Singer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Publication of Animal Liberation Now, published by Peter Singer on May 19, 2023 on The Effective Altruism Forum.
    Summary
    My new book, Animal Liberation Now, will be out next Tuesday (May 23).
    I consider ALN to be a new book, rather than just a revision, because so much of the material in the book is new.
    Pre-ordering from Amazon or other online booksellers (US only) or ordering/purchasing within the first week of publication will increase the chance of the book getting on NYT best-seller list. (Doing the same in other countries may increase the prospects of the book getting on that country’s bestseller list.)
    Along with the publication of the book, I will be doing a speaking tour with the same title as the book. You can book tickets here, with a 50% discount if you use the code SINGER50 (Profits will be 100% donated to effective charities opposing intensive animal production).
    Please spread the words (and links) about the book and the speaking tour to help give the book a strong start.
    Why a new book?
    The major motivation of writing the new book is to have a book about animal ethics that is relevant in the 21st Century. Compared with Animal Liberation, there are major updates on the situation of animals used in research and factory farming, and people’s attitudes toward animals, as well as new research on the capacities of animals to suffer, and on the contribution of meat to climate change.
    What’s different?
    The animal movement emerged after the 1975 version of AL. In particular, the concern for farmed animals developed rapidly over the last two decades. These developments deserve to be reported and discussed.
    Some of the issues discussed in AL have seen many changes since then. Some animal experiments are going out of fashion, while some others emerged. On factory farming, there were wins for the farmed animal movement, such as the partially successful “cage-free movement” and various wins in legislative reforms. But the number of animals raised in factory farms increased rapidly during the same time. A significant portion of this increased number came from aquaculture, in other words fish factory farms. New developments were also seen regarding replacing factory farming, in particular the development of plant-based meat alternative and cultivated meats.
    ALN has a more global perspective than AL, most notably discussing what happened in China. Since the last edition of AL, China has greatly increased the use of animals in research and factory farming.
    There are also changes in my views about a number of issues. Firstly, since 1990 (The year of publication for the last full revision of the 1975 version of AL), scientists have gained more evidence that suggests the sentience of fish and some invertebrates. Accordingly, I have updated my attitudes toward the probability of sentience of these animals. Secondly, I have changed my views toward the suffering of wild animals, in particular the possibility and tractability of helping them. Thirdly, I have added the discussion about the relation between climate change and meat consumption. Last but not least, Effective Altruism, as an idea or as a movement, did not exist when the versions of Animal Liberation were written, so I have added some discussions of the EA movement and EA principles in the new book.
    Is the book relevant to EA?
    Animal welfare is, and should be, one of the major cause areas with EA for reasons I do not need to repeat here. I will explain why ALN is relevant to EA.
    Firstly, ALN contains some of the commonly used arguments by EAs who work on animal welfare on why the issues of animal suffering is important. Reading ALN provides an opportunity for newcomers to the EA community to learn about animal ethics and why some (hopefully most) EAs think that animals matter morally and that they are...
    5 min
  • EA - Tips for people considering starting new incubators by Joey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tips for people considering starting new incubators, published by Joey on May 19, 2023 on The Effective Altruism Forum.
    Charity Entrepreneurship is frequently contacted by individuals and donors who like our model. Several have expressed interest in seeing the model expanded, or seeing what a twist on the model would look like (e.g., different cause area, region, etc.) Although we are excited about maximizing CE’s impact, we are less convinced by the idea of growing the effective charity pool via franchising or other independent nonprofit incubators. This is because new incubators often do not address the actual bottlenecks faced by the nonprofit landscape, as we see them.
    There are lots of factors that prevent great new charities from being launched, and from eventually having a large impact. We have scaled CE to about 10 charities a year, and from our perspective, these are the three major bottlenecks to growing the new charity ecosystem further:
    Mid-stage funding
    Founders
    Multiplying effects
    Mid-stage funding
    We try to look at every step of our charities’ future journeys, to see how we expect them to fare as they progress. In general, there seems to be enough appetite in the philanthropic community to supply seed funding to brand new projects, and we have been successful in helping charities to launch with the funding they need. However, many cause areas appear to have gaps in available funding for charities who are around two to five years old.
    Charities’ budgets tend to grow each year; for example, a charity might need $150k for its first year, $250k for its 2nd, $400k for its third, and so on. The average charity might require a seed of $150k for its first year, and mid-stage funding (years 2-5) of ~$2 million over 4 years. Currently, it is much more difficult for highly effective charities to fundraise this much at this stage of their journey than it is for them to get the funding they need at the seed-funding stage. Keep in mind that this mid-stage funding is still too early and small for most major institutional funders (e.g., GiveWell does not recommend organizations that can only absorb $1 million a year as top charities), and governments rarely consider projects this young.
    Mental health case study: An example that demonstrates this issue well is found in the cause area of mental health. We have identified a number of promising intervention ideas in this area over the past few years, and a solid pool of aspiring entrepreneurs interested in founding mental health charities. Although we expected our seed network would be able to support the first round of funding, we did not have confidence in what came next for these charities. We have since worked to improve the situation by helping launch the Mental Health Funders Circle, but even with that network we are concerned about mid-stage funding in the future.
    Why is mid-stage the problem?: Instead of seed or late stage? I believe it’s the same donors who consider seed or mid-stage funding, but as the volume of funding is smaller at the seed stage, it is covered much more easily. While some charities may struggle in the late stage as well, less will even get to that stage and the number of options typically expand once a charity is clearly established as a field leader.
    Founders
    Although a solid number of people are interested in founding charities, it's only an ideal fit for a relatively small percentage of people. It is a career path that requires a highly entrepreneurial mindset, plus a very strong ethical compass to succeed in. Due to the low number of people who are a good fit, I don’t believe that it is a career path that can absorb a high number of people (my guess would be less than 5% of people are actually suited to founding a nonprofit). It is my opinion that other career paths, like for-profit found...
    14 min
  • LW - The Compleat Cybornaut by ukc10014
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Compleat Cybornaut, published by ukc10014 on May 19, 2023 on LessWrong.
    A cluster of conceptual frameworks and research programmes have coalesced around a 2022 post by janus, which introduced language models as ‘simulators’ (of other types of AIs such as agents, oracles, or genies). One such agenda, cyborgism, was coined in a post by janus and Nicholas Kees and is being researched as part of the 2023 editions of AI Safety Camp and SERI MATS. The objective of this document is to provide an on-ramp to the topic, one that is hopefully accessible to people not hugely familiar with simulator theory or language models.
    So what is cyborgism?
    Cyborgism proposes to use AIs, particularly language models (i.e. generative-pretrained transformers or GPTs), in ways that exploit their (increasingly) general-purpose intelligence, while retaining human control over the ‘dangerous bits’ of AI – i.e. agency, planning, and goal-formation. The overall objective is to leverage human cognitive ability while minimising the risks associated with agentic AI.
    Aside from agency, a core assertion of cyborgism is that certain commonly-used language models are not well-suited to many tasks human users throw at them, but that humans, if appropriately-trained and equipped, might more effectively use GPTs in ways that are ‘natural’ for the model, while dramatically increasing the productive and creative potential of the human.
    Specifically, some current systems, such as ChatGPT, are released or predominantly used in a ‘tuned’ version, which has a host of shortcomings. One such tuning method, reinforcement-learning from human feedback (RLHF) has a specific weakness relevant to cyborgism: the tuning process severely limits, or collapses, a valuable aspect of the GPT, namely its wild, unconstrained creativity.
    Superficially, the cyborgism approach may resemble a human-plus-oracle setup, but there is a subtle and important distinction: an oracle, it is argued, might ‘smuggle in’ some of the trappings of an agent. In contrast, the human cyborg embeds the output of the language model into their own workflow and thinking - model and human work as an integrated system. The cyborg leverages the model’s creative, albeit non-agentic, potential while continuously ‘steering’ or ‘course-correcting’ the model to ensure its output remains relevant to the actual goal. However, cyborgism might entail a high alignment tax: absent appropriate workflows and tools, a setup consisting of a human plus non-agentic GPT might be considerably less productive than a purely agentic AI (as the human component becomes a bottleneck).
    Background Concepts
    Before getting into practical cyborgism, it is helpful to summarize some relevant theories and intuitions about how language models work.
    Why is in-context learning relevant?
    Neural networks generally, and language models specifically, go through several types of training: the large-scale (in terms of compute, time, and data) pre-training when all the neural weights are set in an end-to-end optimisation process; one or more fine-tuning rounds to focus the model on a specific use domain (during which the weights also change); and, in the case of certain models, including GPT-4, ChatGPT, and text-davinci-003, various types of supplementary tuning, which in the case of GPT-4 seems to include RLHF and rule-based reward modelling (RBRM).
    The final phase of training, known as ‘in-context learning’, happens during the session with the user, and doesn’t involve actual changes in neural weights, but does still significantly alter the type of output the model generates, based on the accumulated context of its interaction with an user in a given session. The mechanisms by which this happens are debated, but from a cyborgism perspective, the context provides a powerful way of guiding or cont...
    33 min
  • LW - Resolving internal conflicts requires listening to what parts want by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Resolving internal conflicts requires listening to what parts want, published by Richard Ngo on May 19, 2023 on LessWrong.
    We can resolve internal conflicts by understanding what underlying fears are driving the conflict, then providing evidence that those fears won't happen, thereby reconsolidating the memories which caused them. A simple example of this process comes from exposure therapy for phobias, which works by demonstrating that the phobia is much less dangerous than the person had learned to believe. A wide range of different therapeutic approaches apply the same core mechanism to deal with more complex internal conflicts. I'll focus in particular on the internal family systems (IFS) framework—which, despite the slightly kooky name, is one of the most powerful methods for dealing with internal conflict.
    The core ideas of IFS are essentially the ones I've outlined in the last few posts: that you should think of yourself as being composed of many parts, some of which are implementing protective strategies based on your previous experiences (especially from childhood). IFS particularly highlights the idea that there are “no bad parts”—we should treat all parts as deserving of sympathy, even when the strategies they’re using are harmful and deeply misguided. The four posts in this section of the sequence will talk through how and why to apply these ideas to build trust between different parts. In this post I'll focus on the first step: identifying the underlying parts at play and what they want.
    Our starting point can be any phenomenon that triggers an emotional response from some part of you. You might find one by thinking about an emotionally-loaded topic, like your work or relationships (especially with your parents); or paying attention to how your body feels; or paying attention to the way you choose your words or thoughts, and which ones you're suppressing; or to your dreams; or to character archetypes or symbolic motifs that particularly resonate with you. Many types of triggers work far better for some people than others—a fact which helps explain why so many different psychotherapy techniques exist.
    However you find an emotional trigger, the next step is to work through the protective or defensive strategies associated with the part causing the response, to figure out what its underlying fear or need is. In doing so, it’s often helpful to name or visualize the active part (e.g. by identifying it with a younger version of you) and imagine having a conversation with it. The practice of Focusing can also be useful here—this involves saying a possible articulation of what the part wants out loud, seeing if it resonates, and adjusting it if not. Again, different techniques will work for different people; the important thing is finding some introspective technique for narrowing in on what the part "wants to say".
    Doing so often requires navigating two types of defensiveness: from the part that's trying to articulate its perspective, and from other parts reacting to criticism of themselves. For example, suppose that you face a conflict between a part that wants to donate more to charity and a part that wants to spend more on holidays with your friends. The former might be partially driven by a fear of others thinking you’re selfish; the latter might be partially driven by a fear of not seeming cool enough. For each of them, criticising the other part helps it get more of what it wants, while admitting its own fear gives the other part ammunition to use against it. So each part might become defensive both when it's prompted to articulate its underlying motivations, and when criticized by the other part.
    What defensiveness looks like varies by person, but it often involves angry pushback, refusal to engage, or redirection towards less sensitive topics (e.g. via ...
    7 min
  • EA - Relative Value Functions: A Flexible New Format for Value Estimation by Ozzie Gooen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Relative Value Functions: A Flexible New Format for Value Estimation, published by Ozzie Gooen on May 18, 2023 on The Effective Altruism Forum.
    Summary
    Quantifying value in a meaningful way is one of the most important yet challenging tasks for improving decision-making. Traditional approaches rely on standardized value units, but these falter when options differ widely or lack an obvious shared metric. We propose an alternative called relative value functions that uses programming functions to value relationships rather than absolute quantities. This method captures detailed information about correlations and uncertainties that standardized value units miss. More specifically, we put forward value ratio formats of univariate and multivariate forms.
    Relative value functions ultimately shine where single value units struggle: valuing diverse items in situations with high uncertainty. Their flexibility and elegance suit them well to collective estimation and forecasting. This makes them particularly well-suited to ambitious, large-scale valuation, like estimating large utility functions.
    While promising, relative value functions also pose challenges. They require specialized knowledge to develop and understand, and will require new forms of software infrastructure. Visualization techniques are needed to make their insights accessible, and training resources must be created to build modeling expertise.
    Writing programmatic relative value functions can be much easier than one might expect, given the right tools. We show some examples using Squiggle, a programming language for estimation.
    We at QURI are currently building software to make relative value estimation usable, and we expect to share some of this shortly. We of course also very much encourage others to try other setups as well.
    Ultimately, if we aim to eventually generate estimates of things like:
    The total value of all effective altruist projects;
    The value of 100,000 potential personal and organizational interventions; or
    The value of each political bill under consideration in the United States;
    then the use of relative value assessments may be crucial.
    Presentation & Demo
    I gave a recent presentation on relative values, as part of a longer presentation in our work at QURI. This features a short walk-through of an experimental app we're working on to express these values. The Relative Values part of the presentation is is from 22:25 to 35:59.
    This post gives a much more thorough description of this work than the presentation does, but the example in the presentation might make the rest of this make more sense.
    Challenges with Estimating Value with Standard Units
    The standard way to measure the value of items is to come up with standardized units and measure the items in terms of these units.
    Many health measure benefits are estimated in QALYs or DALYs
    Consumer benefit has been measured in willingness to pay
    Longtermist interventions have occasionally been measured in “Basis Points”, Microdooms and Microtopias
    Risky activities can be measured in Micromorts
    COVID activities have been measured in MicroCOVIDs
    Let’s call these sorts of units “value units” as they are meant as approximations or proxies of value. Most of these (QALYs, Basis Points, Micromorts) can more formally be called summary measures, but we’ll stick to the term unit for simplicity.
    These sorts of units can be very useful, but they’re still infrequently used.
    QALYs and DALYS don’t have many trusted and aggregated tables. Often there are specific estimates made in specific research papers, but there aren’t many long aggregated tables for public use.
    There are very few tables of personal intervention value estimates, like the net benefit of life choices.
    Very few business decisions are made with reference to clear units of value. For example, “Whi...
    30 min
  • EA - EA Forum: content and moderator positions by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Forum: content and moderator positions, published by Lizka on May 18, 2023 on The Effective Altruism Forum.
    TL;DR: We’re hiring for Forum moderators — apply by 1 June (it’s the first round of the application, and should take 15-20 minutes). We’re also pre-announcing a full-time Content Specialist position on the Online Team at CEA — you can indicate interest in that.
    ➡️ Apply to be a part-time Forum moderator by 1 June.
    Round 1 of the application should take around 15-20 minutes, and applying earlier is better.
    You can see moderator responsibilities below. This is a remote, part-time, paid position.
    ➡️ Indicate interest in a full-time Content Specialist position on the Online Team at CEA.
    We’ll probably soon be hiring for someone to work with me (Lizka) on content-related tasks on the Forum. If you fill out this form, we will send you an email when the application opens and consider streamlining your application if you seem like a particularly good fit.
    You can see more about the role’s responsibilities below. This is a full-time position, and can be remote or in-person from Oxford/Boston/London/Berkeley.
    ➡️ You can also indicate interest in working as a copy-editor for CEA or in being a Forum Facilitator (both are part-time remote roles).
    If you know someone who might be interested, please consider sending this post to them!
    Please feel free to get in touch with any questions you might have. You can contact [email protected] or [email protected], comment here, or reach out to moderators and members of the Online Team.
    An overview of the roles
    I’ve shared a lot more information on the moderator role and the full-time content role in this post — here's a summary in table form. (You can submit the first round of the moderator application or indicate interest in the content role without reading the whole post.)
    TitleAbout the roleKey responsibilitiesStage the application is atModeratorPart-time, remote (average ~3 hours a week but variable), $40/hourMake the Forum safe, welcoming, and collaborative (e.g. by stopping or preventing aggressive behavior, being clear about moderation decisions), nurture important qualities on the Forum (e.g. by improving the written discussion norms or proactively nudging conversations into better directions), and help the rest of the moderation team.Round 1 is open (and should take 15-20 minutes): apply by 1 JuneContent SpecialistFull-time, remote/ in-person (Oxford/ London/ Boston/ Berkeley)Encourage engagement with important and interesting online content (via outreach, newsletters, curation, Forum events, writing, etc.), improve the epistemics, safety, and trust levels on the Forum (e.g. via moderation), and more.Indication of interest (we'll probably open a full application soon)We’re also excited for indications of interest for the following part-time
    contractor roles, although we might not end up hiring for these in the very near futureCopy-editor indication of interestPart-time, remote (~4 hours a week average), $30/hour by defaultCopy-editing for style, clarity, grammar — and generally sanity-checking content for CEA. Sometimes also things like reformatting, summarizing other content, finding images, and possibly posting on the website or social media. Forum Facilitator indication of interestPart-time, remote (~3 hours a week average), $30/hour
    TitleAbout the roleKey responsibilitiesStage the application is atModeratorPart-time, remote (average ~3 hours a week but variable), $40/hourMake the Forum safe, welcoming, and collaborative (e.g. by stopping or preventing aggressive behavior, being clear about moderation decisions), nurture important qualities on the Forum (e.g. by improving the written discussion norms or proactively nudging conversations into better directions), and help the rest ...
    25 min
  • LW - Some background for reasoning about dual-use alignment research by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some background for reasoning about dual-use alignment research, published by Charlie Steiner on May 18, 2023 on LessWrong.
    This is pretty basic. But I still made a bunch of mistakes when writing this, so maybe it's worth writing. This is background to a specific case I'll put in the next post.
    It's like a a tech tree
    If we're looking at the big picture, then whether some piece of research is net positive or net negative isn't an inherent property of that research; it depends on how that research is situated in the research ecosystem that will eventually develop superintelligent AI.
    Consider this toy game in the picture. We start at the left and can unlock technologies, with unlocks going faster the stronger our connections to prerequisites. The red and yellow technologies in the picture are superintelligent AI - pretend that as soon as one of those technologies is unlocked, the hastiest fraction of AI researchers are immediately going to start building it. Your goal is for humanity to unlock yellow technology before a red one.
    This game would be trivial if everyone agreed with you. But there are many people doing research, and they have all kinds of motivations - some want as many nodes to be unlocked as possible (pure research - blue), some want to personally unlock a green node (profit - green), some want to unlock the nearest red or yellow node no matter which it is (blind haste - red), and some want the same thing as you (beneficial AI - yellow) but you have a hard time coordinating with them.
    In this baseline tech tree game, it's pretty easy to play well. If you're strong, just take the shortest path to a yellow node that doesn't pass too close to any red nodes. If you're weak, identify where the dominant paradigm is likely to end up, and do research that differentially advantages yellow nodes in that future.
    The tech tree is wrinkly
    But of course there are lots of wrinkles not in the basic tech tree, which can be worth bearing in mind when strategizing about research.
    Actions in the social and political arenas. You might be motivated to change your research priorities based on how it could change peoples' minds about AI safety, or how it could affect government regulation.
    Publishing and commercialization. If a player publishes, they get more money and prestige, which boosts their ability to do future research. Other people can build on published research. Not publishing is mainly useful to you if you're already in a position of strength, and don't want to give competitors the chance to outrace you to a nearby red node (and of course profit-motivated players will avoid publishing things that might help competitors beat them to a green node).
    Uncertainty. We lack exact knowledge of the tech tree, which makes it harder to plan long chains of research in advance. Uncertainty about the tech tree forces us to develop local heuristics - ways to decide what to do based on information close at hand. Uncertainty adds a different reason you might not publish a technology: if you thought it was going to be a good idea to research when you started, but then you learned new things about the tech tree and changed your mind.
    Inhomogeneities between actors and between technologies. Different organizations are better at researching different technologies - MIRI is not just a small OpenAI.
    Ultimately, which technologies are the right ones to research depends on your model of the world / how you expect the future to go. Drawing actual tech trees can be a productive exercise for strategy-building, but you might also find it less useful than other ways of strategizing.
    We're usually mashing together definitions
    I'd like to win the tech tree game. Let's define a "good" technology as one that would improve our chances of winning if it was unlocked for free, given the state of the ga...
    15 min
  • LW - Investigating Fabrication by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Investigating Fabrication, published by LoganStrohl on May 18, 2023 on LessWrong.
    Note 1: This was originally published in 2021. I'm now re-publishing it in 2023 as the penultimate post in The Nuts and Bolts Of Naturalism, as a demonstration of the method. I intend to publish a collection of much-more-complete demonstrations in the future, and I plan for them to explicitly discuss each step as I've described it in the sequence. For now, though, I hope that this partial demo will give you a taste of one way the overall process can go.(For a similarly partial demo by someone other than me, check out "Maps of Maps, and Empty Expectations" by Nora Ammann.)
    Note 2: This essay was written in response to “Lies, Damn Lies, and Fabricated Options”. I expect it’ll be pretty confusing to read this one without having read that one first. However, I’m not the boss of you, and it isn’t obvious to me anyway which reading order is best overall. If prolonged confusion makes you grumpy, I recommend reading Duncan’s essay first. But if you try it backwards, I’d love to hear how that goes.
    1. Motivation
    When I read Duncan's essay "Lies, Damn Lies, and Fabricated Options," it seemed to me to point toward something that's worth knowing for real, rather than just worth knowing about.
    I think it does a good job of helping the reader build a new concept that could conceivably be taken as a hypothesis in important situations. The hypothesis is, “Perhaps this thing that seems to me like an option isn’t really a possibility, and it only seems like one because I’m in the grips of a map/territory conflation.” (Or, at least, “Perhaps the person I’m arguing with thinks their made-up option is real because [etc.].”)
    But I wasn’t satisfied with my new concept “fabricated options”. There were some super important questions that I’d need to answer before I could put it to use.
    For instance,
    Which situations are the crucial ones for promoting this hypothesis to attention?
    How do I actually move my mind to recognize that the option I’m considering might be fabricated?
    Once I’ve posed the hypothesis, how do I confirm or deny it?
    When I’ve successfully identified a fabricated option, how can I prevent the usual harm?
    And, my favorite: How can I stop making this kind of mistake in the first place?
    So I set out to investigate option fabrication (the act), and what my brain is up to in the moments just before a fabricated option appears. I hoped that if I could learn where fabricated options come from, and what they are made of, I’d be a long way toward answering many of the practical questions on my list.
    2. Approach
    I did naturalism to this. (One of the reasons this essay felt worth writing and sharing is to give people more chances to see what naturalism looks like in practice.)
    I assumed that there's sometimes A Thing Going On in my brain that in some ways resembles what Duncan called "fabricated options", and that my default understanding of the Thing is worse than I'd prefer. I set out to observe the Thing myself, directly and in real time, and to gradually build my own understanding of whatever I found. I tried to observe the Thing both in the "field" (as it occurs naturally in the course of daily life) and in the "lab" (as it occurs when I deliberately create controllable conditions that might give rise to it).
    This was not a full naturalist study (which usually takes at least a month), but was instead a "let's see what comes of a week or two" dalliance. Still, it turned out to follow the arc of a mostly complete study: Sensitization, zooming in, zooming out, and experimentation. [ETA: Though I used different terms here, these correspond to Locating Fulcrum Experiences, Getting Your Eyes On, Collection, and Experimentation.]
    3. Lab Work, Part 1
    In accordance with the “try it immediately” heuristic, I ...
    23 min
  • LW - Collective Identity by NicholasKees
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Collective Identity, published by NicholasKees on May 18, 2023 on LessWrong.
    Thanks to Simon Celinder, Quentin Feuillade--Montixi, Nora Ammann, Clem von Stengel, Guillaume Corlouer, Brady Pelkey and Mikhail Seleznyov for feedback on drafts. This post was written in connection with the AI Safety Camp.
    Executive Summary:
    This document proposes an approach to corrigibility that focuses on training generative models to function as extensions to human agency. These models would be designed to lack independent values/preferences of their own, because they would not have an individual identity; rather they would identify as part of a unified system composed of both human and AI components.
    The selfless soldier: This section motivates the difference between two kinds of group centric behavior, altruism (which is based in individual identity) and collective identity.
    Modeling groups vs individuals: Here we argue that individuals are not always the most task-appropriate abstraction, and that it often makes sense to model humans on the group level.
    Generative predictive models: This section describes how generative predictive models will model themselves and their environment, and motivates the importance of the “model of self” and its connection to personal identity.
    Strange identities: There are several ways (in humans) in which the one-to-one correspondence between a neural network and its model of self breaks down, and this section discusses three of those examples in order to suggest that identity is flexible enough that an AI’s identity need not be individual or individuated.
    Steps toward identity fusion: Here we aim to clarify the goal of this agenda and what it would mean for an AI to have an identity based on a human-AI system such that the AI component extends the human’s agency. While we don’t give a clear plan for how to bring about this fusion, we do offer an antithetical example of what kind of training would clearly fail.
    Relevance for corrigibility: This section concludes the document by drawing more direct connections to corrigibility, and by offering a series of open questions for how this research might be made more concrete.
    The selfless soldier
    In the heat of battle a grenade is tossed into the middle of a troop of soldiers. One soldier throws themself on top of the grenade, sacrificing themself for the survival of the troop. There are two main ways to frame what just happened.
    Altruism (individual identity): The soldier has the personal value/preference of protecting their troop from harm. Reasoning (quickly) from this value, the soldier deduces that they must sacrifice themselves in order to bring about the future where their fellow soldiers are safe.
    Collective Identity: The individual soldier is not the most important abstraction to explain/predict this situation, rather it is the troop as a whole. The troop cares about its own survival, and this manifests itself in the decision to sacrifice one of its members to protect itself from further harm (even though the cognition, at least at the moment of decision, happens entirely within one brain). While the individual soldier could theoretically use clever arguments to escape this conclusion, they do not (because as a component of the troop, this is not their function).
    The problem of alignment is often framed as trying to ensure that the values of an AI system are aligned with humanity, ideally imbuing them with a certain kind of perfect altruism toward humankind. The problem of corrigibility is often framed as ensuring that even when those values are not (yet) perfectly aligned with our own, an adversarial relationship does not develop between the AI and its human designers (such that it would resist shutdown attempts or changes to its source code).
    This approach tries instead to explore how we might buil...
    14 min