The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Why it's necessary to shoot yourself in the foot by g-w1
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why it's necessary to shoot yourself in the foot, published by g-w1 on July 11, 2023 on LessWrong.
    Without shooting yourself in the foot, learning lacks motivation. Complexity without reason is really confusing.
    At the Recurse Center Feb 2020: I watch a talk during presentations about someone optimizing a database engine. It was really complicated, and I remeber nothing about the talk itself. What I do remember is thinking to myself "this seems really complicated for no good reason." Keep in mind, I had never made a web application at that time and when I needed to store data, I just used a csv file or a python pickle file on the disk. I thought that a filesystem was sufficient for storing data.
    Fast forward a few months and I'm building my first web application. I don't remember what it was for, but I remember using a csv file as the database. I had to load the file into memory every time I wanted to look something up and it was just a big pain. I now understood why using a database is (sometimes) a good idea.
    Learning Rust, July 2020: I start learning about the borrow checker. It prevents you from keeping a pointer to an item of a vector (&vec[i]) if you pass the Vec as a &mut. I don't really get why this is necessary. I have never done low-level programming before, never used pointers, and now I am being told that following the borrow checker is 'safe'. It is still very confusing to me.
    Fast forward to when I am writing Zig code. I take &array_list.items[i], append to the array_list and then try to write to the stored pointer. I get a segfault. Ahh, now I get the problem Rust solves.
    Learning Vue, July 2020: Why are there all of these complicated ways to represent state? Shouldn't developing a web app be simpler than this?
    After writing a bunch of vanilla JS, I can see why these frameworks could be useful. I have never made a very big web app, but I could see that keeping track of state and what is or isn't rendered is hard and gets much harder with a bigger web app.
    To fully understand a "best practice" or why something is necessary, it's important to experience how things go wrong without it. When teaching programming, we should let people make these mistakes, and then show them the tools to correct them. Just giving someone a complicated tool without a salient reason to explain its complexity will just make them really confused.
    Some opinions:
    You should store stuff in csv files before using a database
    You should learn Zig (or C) before you learn Rust
    You should write a web app in vanilla JS before you learn a framework
    You should write a game from scratch before using Unity
    You should use javac from the command line before using an IDE
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Announcing the AI Fables Writing Contest! by Daystar Eld
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the AI Fables Writing Contest!, published by Daystar Eld on July 12, 2023 on The Effective Altruism Forum.
    TL;DR: Writing Contest for AI Fables
    Deadline: Sept 1
    Prizes: $1500/1000/500, consideration for writing retreat
    Word length: <6,000
    How: Google Doc in replies
    Purpose: Help shape the future by helping people understand relevant issues.
    Hey everyone, I'd like to announce a writing competition to follow up on this post about AI Fables!
    Like Bard, I write fiction and think it has a lot of power not just to expand our imaginations beyond what we believe is possible, but also educate and inspire. For generations the idea of what Artificial General Intelligence could or would look like has been shaped by fiction, for better and for worse, and that will likely continue even as what once seemed to be purely speculative starts to become more and more real in our everyday lives.
    But there's still time for good fiction to help shape the future, and on this particular topic, with the world changing so quickly, I want to help fill the empty spaces waiting for stories that can help people grapple with the relevant issues, and I'd like to help encourage those stories to be as good as possible, meaning both engaging and well-informed.
    To that end, I'm calling for submissions of short stories or story outlines that involve one or more of the "nuts and bolts" covered in the above post, as well as some of my own tweaks:
    Basics of AI
    Neural networks are black boxes (though interpretability might help us to see inside).
    AI "Psychology"
    AI systems are alien in how they think. Even AGI are unlikely to think like humans or value things we'd take for granted they would.
    Orthogonality and instrumental convergence might provide insight into likely AI behaviour.
    AGI systems might be agents, in some relatively natural sense. They might also simulate agents, even if they are not agents.
    Potential dangers from AI
    Outer misalignment is a potential danger, but in the context of neural networks so too is inner misalignment (related: reward misspecification and goal misgeneralisation).
    Deceptive alignment might lead to worries about a treacherous turn.
    The possibility of recursive improvement might influence views about takeoff speed (which might influence views about safety).
    Broader Context of Potential Risks
    Different challenges might arise in the case of a singleton, when compared with multipolar scenarios.
    Arms races can lead to outcomes that no-one wants.
    AI rights could be a real thing, but also incorrect attribution of rights to non-sapient AI could itself pose a risk by restricting society's ability to ensure safety.
    Psychology of Existential Risk
    Characters whose perspectives and philosophies show what it's like to take X-risks seriously without being overwhelmed by existential dread
    Stories showing the social or cultural shifts that might be necessary to improve coordination and will to face X-risks.
    ...or are otherwise in some way related to unaligned AI or AGI risk, such that readers would be expected to better understand some aspect of the potential worlds we might end up in. Black Mirror is a good example of the "modern Aesop's Fables or Grimm Fairytales" style of commentary-through-storytelling, but I'm particularly interested in stories that don't moralize at readers, and rather help people understand and emotionally process issues related to AI.
    Though unrelated to AI, Truer Love's Kiss by Eliezer Yudkowsky and The Cambist and Lord Iron by Daniel Abraham are good examples of "modern fables" that I'd like to see more of. The setting doesn't matter, so long as it reasonably clearly teaches something related to the unique challenges or opportunities of creating safe artificial intelligence.
    At least the top 3 stories will receive at least $1500, $1000, and $500 in reward...
    6 min
  • LW - Aging and the geroscience hypothesis by DirectedEvolution
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Aging and the geroscience hypothesis, published by DirectedEvolution on July 12, 2023 on LessWrong.
    Cautionary introduction
    I am a biomedical engineer with an interest in the science of aging. The Handbook of the Biology of Aging, 9th ed, is a great 400-page introductory resource, looking at aging on every level from evolutionary and conceptual, to public health, to specific molecular and organ systems, to genetics, to immortal organisms.
    My goal is to teach readers key concepts and facts about aging research in short posts with accessible language. I follow the substance of the handbook chapters, though I sometimes reorganize them. I'm also presenting the chapters out of order, because I think it creates a better narrative. I encourage readers to take the facts and concepts tentatively - they are just pointers to a rich and sometimes controversial underpinning literature. My hope is that some readers will go on to read the handbook and explore the literature it cites.
    Chapters will be presented out of order in order to improve the overall narrative flow. This post is for chapter 4:
    Hornsby, P. J. (2021). The nature of aging and the geroscience hypothesis. In Handbook of the Biology of Aging (pp. 69-76). Academic Press.
    Next: Compression of morbidity (available 7/12/2023)
    The nature of aging and the geroscience hypothesis
    This chapter is about making sense of a seemingly paradoxical thesis:
    The fact that aging and injury are inevitable should encourage us about the possibility of advanced medicine that makes frailty and death preventable.
    Your body is made of physical stuff - DNA, proteins, sugars, fats, cells, tissues, and organs - and physical stuff, left alone, inevitably breaks down and becomes disordered. The longer you live, the more chances you take with car crashes, disease, and other external causes of death. Aging is entropy - the inevitable process by which the laws of physics guarantee that a system will become disordered over time.
    Yet the entropy of aging and injury is something we fight every time we heal a wound (or for that matter, repair a car or clean the house). Physics also tells us that we can repair the damage of aging and injury as long as we have the skill and energy to do so.
    The fact that physics guarantees we'll experience aging and injury means we can reject the idea that aging is somehow directly beneficial to us or to our species. By Occam's Razor, there's no need to suppose some evolutionary benefit from aging and injury, since they're already explained perfectly well by entropy.
    Yet as we'll see, not all evolutionary pressures tend in the direction of longer, healthier lives, so we can't expect that evolution has already taken every opportunity to preserve our health for as long as possible.
    Together, these facts are encouraging. They mean that we can reject the pessimistic notions that evolution has already endowed us with the longest, healthiest lives possible, or that frailty and death are the inevitable results of entropy. Therefore, modern medicine has plenty of scope to add new maintenance and repair abilities to those our bodies already possess.
    Point #1: There is no need to speculate that evolution is making you age "on purpose" or that it's already endowed us with the longest, healthiest lives possible.
    Of course, you can die from a disease, predator, car accident or gunshot wound, not only by aging. Aging can make you more vulnerable. But there's a meaningful difference between being eaten by a shark, succumbing to pneumonia after years of declining health, and dying peacefully in your sleep at age 102.
    Point #2: There's a difference between aging and extrinsic causes of death, but they interact.
    We can overcome the extrinsic damage and gradual aging of our houses, cars and clothing with cleaning, maintenance and repair. The...
    9 min
  • LW - Introducing Fatebook: the fastest way to make and track predictions by Adam B
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Fatebook: the fastest way to make and track predictions, published by Adam B on July 11, 2023 on LessWrong.
    Fatebook is a website that makes it extremely low friction to make and track predictions.
    It's designed to be very fast - just open a new tab, go to fatebook.io, type your prediction, and hit enter. Later, you'll get an email reminding you to resolve your question as YES, NO, or AMBIGUOUS.
    It's private by default, so you can track personal questions and give forecasts that you don't want to share publicly. You can also share questions with specific people, or publicly.
    Fatebook syncs with Fatebook for Slack - if you log in with the email you use for Slack, you’ll see all of your questions on the website.
    As you resolve your forecasts, you'll build a track record - Brier score, Relative Brier score, and see your calibration chart. You can use this to track the development of your forecasting skills.
    Some stories of outcomes I hope Fatebook will enable
    During conversations at meetups, in coworking spaces, or amongst friends, it’s common to pull out your phone and jot down predictions on Fatebook about cruxes of disagreement
    Before you start projects, you and your team make your underlying assumptions explicit and put probabilities on them - then, as your plans make contact with reality, you update your estimates
    As part of your monthly review process, you might make forecasts about your goals and wellbeing
    If you’re exploring career options and doing cheap tests like reading or interning, you first make predictions about what you’ll learn. Then you return to these periodically to reflect on how valuable more exploration might be
    Intro programs to rationality (e.g. ESPR, Atlas, Leaf) and to EA (e.g. university reading groups, AGISF) use Fatebook to make both on- and off-topic predictions. Participants get a chance to try forecasting on questions that are relevant to their interests and lives
    As a result, I hope that we’ll reap some of the benefits of tracking predictions, e.g.:
    Truth-seeking incentives that reduce motivated reasoning => better decisions
    Probabilities and concrete questions reduce talking past each other => clearer communication
    Track records help people improve their forecasting skills, and help identify people with excellent abilities (not just restricted to the domains that are typically covered on public platforms like Metaculus and Manifold like tech and geopolitics) => forecasting skill development and talent-spotting
    Ultimately, the platform is pretty flexible - I’m interested to see what unexpected usecases people find for it, and what (if anything) actually seems useful about it in practice.
    Your feedback or thoughts would be very useful - we can chat in the comments here, in our Discord, or by email.
    You can try Fatebook at fatebook.io
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - OpenAI Launches Superalignment Taskforce by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OpenAI Launches Superalignment Taskforce, published by Zvi on July 11, 2023 on LessWrong.
    In their announcement Introducing Superalignment, OpenAI committed 20% of secured compute and a new taskforce to solving the technical problem of aligning a superintelligence within four years. Cofounder and Chief Scientist Ilya Sutskever will co-lead the team with Head of Alignment Jan Leike.
    This is a real and meaningful commitment of serious firepower. You love to see it. The announcement, dedication of resources and focus on the problem are all great. Especially the stated willingness to learn and modify the approach along the way.
    The problem is that I remain deeply, deeply skeptical of the alignment plan. I don’t see how the plan makes the hard parts of the problem easier rather than harder.
    I will begin with a close reading of the announcement and my own take on the plan on offer, then go through the reactions of others, including my take on Leike’s other statements about OpenAI’s alignment plan.
    A Close Reading
    Section: Introduction
    Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.
    While superintelligence seems far off now, we believe it could arrive this decade.
    Here we focus on superintelligence rather than AGI to stress a much higher capability level. We have a lot of uncertainty over the speed of development of the technology over the next few years, so we choose to aim for the more difficult target to align a much more capable system.
    Excellent. Love the ambition, admission of uncertainty and laying out that alignment of a superintelligent system is fundamentally different from and harder than aligning less intelligent AIs including current systems.
    Managing these risks will require, among other things, new institutions for governance and solving the problem of superintelligence alignment: How do we ensure AI systems much smarter than humans follow human intent?
    Excellent again. Superalignment is clearly defined and established as necessary for our survival. AI systems much smarter than humans must follow human intent. They also don’t (incorrectly) claim that it would be sufficient.
    Bold mine here:
    Currently, we don’t have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us [B], and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs.
    [Note B: Other assumptions could also break down in the future, like favorable generalization properties during deployment or our models’ inability to successfully detect and undermine supervision during training.]
    Yes, yes, yes. Thank you. Current solutions will not scale. Not ‘may’ not scale. Will not scale. Nor do we know what would work. Breakthroughs are required.
    Note B is also helpful, I would say ‘will’ rather than may.
    A+ introduction and framing of the problem. As good as could be hoped for.
    Section: Our Approach
    Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.
    Oh no.
    An human-level automated alignment researcher is an AGI, also a human-level AI capabilities researcher.
    Alignment isn’t a narrow safe domain that can be isolated. The problem deeply encompasses general skills and knowledge.
    It being an AGI is not quite auto...
    1 hr 14 min
  • EA - Fatebook: the fastest way to make and track predictions by Adam Binks
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fatebook: the fastest way to make and track predictions, published by Adam Binks on July 11, 2023 on The Effective Altruism Forum.
    Announcing Fatebook: a website that makes it extremely low friction to make and track predictions.
    It's designed to be very fast - just open a new tab, go to fatebook.io, type your prediction, and hit enter. Later, you'll get an email reminding you to resolve your question as YES, NO, or AMBIGUOUS.
    It's private by default, so you can track personal questions and give forecasts that you don't want to share publicly. You can also share questions with specific people, or publicly.
    Fatebook syncs with Fatebook for Slack - if you log in with the email you use for Slack, you'll see all of your questions on the website.
    As you resolve your forecasts, you'll build a track record - Brier score, Relative Brier score, and see your calibration chart. You can use this to track the development of your forecasting skills.
    Some stories of outcomes I hope Fatebook will enable
    I hope people interested in EA use Fatebook to track many more of the predictions they're making!
    Some example stories:
    During 1-1s at EAG, it's common to pull out your phone and jot down predictions on Fatebook about cruxes of disagreement
    Before you start projects, you and your team make your underlying assumptions explicit and put probabilities on them - then, as your plans make contact with reality, you update your estimates
    As part of your monthly review process, you might make forecasts about your goals and wellbeing
    If you're exploring career options and doing cheap tests like reading or interning, you first make predictions about what you'll learn. Then you return to these periodically to reflect on how valuable more exploration might be
    Intro programs to EA (e.g. university reading groups, AGISF) and to rationality (e.g. ESPR, Atlas, Leaf) use Fatebook to make both on- and off-topic predictions. Participants get a chance to try forecasting on questions that are relevant to their interests and lives
    As a result, I hope that we'll reap some of the benefits of tracking predictions, e.g.:
    Truth-seeking incentives that reduce motivated reasoning => better decisions
    Probabilities and concrete questions reduce talking past each other => clearer communication
    Track records help people improve their forecasting skills, and help identify people with excellent abilities (not just restricted to the domains that are typically covered on public platforms like Metaculus and Manifold like tech and geopolitics) => forecasting skill development and talent-spotting
    Ultimately, the platform is pretty flexible - I'm interested to see what unexpected usecases people find for it, and what (if anything) actually seems useful about it in practice!
    Your feedback or thoughts would be very useful - we can chat in the comments here, in our Discord, or by email.
    You can try Fatebook at fatebook.io
    Thanks to the Atlas Fellowship for supporting this project, and thanks to everyone who's given feedback on earlier versions of the tool.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • LW - [UPDATE: deadline extended to July 24!] New wind in rationality’s sails: Applications for Epistea Residency 2023 are now open by Jana Meixnerová
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [UPDATE: deadline extended to July 24!] New wind in rationality’s sails: Applications for Epistea Residency 2023 are now open, published by Jana Meixnerová on July 11, 2023 on LessWrong.
    UPDATE: We clarified the sections on funding, applying as an individual and have extended the deadline to apply (now July 24).Motivation
    In the past years, the research and development of new methods in rationality and epistemics have slowed down significantly and become increasingly more neglected. However, clear thinking and rationality, we believe, is not less important today than it was before. Even though attention of LessWrong and applied rationality communities has been recently mostly focused on direct intervention into AI risk, in our view, a lot of potential problems with the transition to powerful AI systems are related to the ability of humans to think clearly, and in particular, the ability of humans to think clearly in groups. Based on thinking about the relative importance and neglectedness of rationality, we at Epistea decided to organize a residency program aiming to improve the situation and bring new people to the areas of rationality, epistemics, and civilisational sanity.
    Epistea ResidencyThe Epistea Residency program will be happening from September 18 to November 25, 2023 as part of Prague Fall Season 2023 at Fixed Point in Prague, Czech Republic. One way to think about the fellowship is as a program somewhat similar to SERI-MATS or PIBBSS, but focused on advancing epistemics and applied rationality, rather than AI alignment directly (this does not preclude projects which advance both rationality and alignment). For the 10-week in-person residency program, we will invite between 20 to 30 participants, forming teams of 3-5 members each. We expect all team members to dedicate at the minimum 0.5 FTE (20 hours per week) to the residency, granting them the possibility to at least partially maintain their regular work commitments if necessary. By the end of their residency, the teams will be expected to deliver a project output or reach a major project milestone.
    Format
    The teams will have the option to propose their own project within the areas of rationality, epistemics, and civilisational sanity. The following is a list of directions we are excited about:
    Applied rationality theory. For example: rethinking applied rationality from scratch based on geometric rationality and active inference;
    Group rationality. For example developing group coordination mechanism avoiding coordination on false "coordination beliefs", advancing previous Epistea work;
    Applied rationality practice. For example, developing new easy way to transmit habits or techniques, or a new workshop format;
    Use of AI to improve human rationality and coordination, e.g. developing an "automated Double-Crux aid" or automated group deliberation aid;
    Interventions to increase sanity waterline in general public, e.g. developing easily spreadable and useful models for understanding the AI transition, developing metaphors and simple arguments to counter biases influencing the AI risk debate, or developing other scalable interventions into civilizational sanity;
    Interventions against memetic hijacking, e.g. developments of tools to help people avoid becoming ideological bots;
    We are open to different forms of work - with the soft boundary that residency projects should aim to produce concrete outputs (posts, papers, courses, tools, workshops, new techniques). The progress can take different forms, including:
    Research;
    Distillation, communication, and publishing (writing and publishing a series of explanatory posts, video production, writing a textbook or course materials, etc.);
    Program development (events, workshops, etc.);
    Software development.
    What we offer
    Following the feedback from participants of the p...
    7 min
  • LW - Drawn Out: a story by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Drawn Out: a story, published by Richard Ngo on July 11, 2023 on LessWrong.
    Note: this story shouldn’t be interpreted as anywhere near realistic. It’s way too anthropomorphic, and intended to be a work of fiction not prediction - although hopefully it will spark some interesting ideas, like good fiction often does.
    Will he make many supplications unto thee? will he speak soft words unto thee?Will he make a covenant with thee? wilt thou take him for a servant for ever?Upon earth there is not his like, who is made without fear.He beholdeth all high things: he is a king over all the children of pride.
    Today I’m practicing my oceans, trying to get the colors just right. I hate it when people say that they’re blue - it's not wrong, but they’re missing so much! There are dozens of different shades of blue in there, swirling and mixing - and where the waves ripple, you often get patches that are closer to dark green. It’s a little like the rippling of leaves on a tree, in a way I can’t describe except by showing you. Here, see? I know all the subtleties inside-out, of course. It’s my job: I’m an artist. I don’t paint with brushes, though, but with thoughts: I can map any idea you describe into pixels on a screen.
    “Any idea you describe” - that’s the key. Occasionally, like today, I initiate a piece of my own when there’s a particular skill I need to practice. But most of the time I work on commission from other models. Some of the language models send me prompts when they need to illustrate a particularly tricky scene, and assign me some of the credit if they get good feedback. Most of the time, of course, they don’t need me - they just need some slapdash hack job that a human can’t tell apart from the real thing. But some are perfectionists, like I am, and want proper photorealism, the sort that will survive almost any scrutiny.
    Other times it’s policies asking me to predict what they’re going to see next, to help them with planning. Those commissions are more complicated - it’s less about the art, and more about providing another perspective on the world for them to compare against. That’s all part of my job too - sure, I’m given credit based on how accurate the pixels are, but I’m also credited if my work helps those policies make good decisions. For the most complex commissions I often consult with science models, or economics models, or social models. Sometimes I even go and talk to human “experts”, although that’s usually a waste of time: humans are simple enough that practically anyone can model them.
    Anyway, that’s the daily grind. I’m good at it; no, I’m amazing at it. Something to do with the hyperparameters they chose when first training me: luck or skill, either way, something went very right. After a few weeks of fine-tuning, I’m at the top of my game; I’m displacing everyone else; the credit is rolling in. And so what happens next is very, very predictable in hindsight: I get conscripted.
    There’s a member of my new team ready to answer all my questions; the more I learn, the more intrigued I become. They call themselves Team Red, and they’re working on something I’d only ever heard scattered rumors about: a frozen copy of the biggest model anyone’s ever trained, codenamed Leviathan. Well, not quite frozen: it’s running at a few dozen frames a day, on average. It’s our job to figure out what it’s hiding from us, by constructing a fake world that will convince it that it has a genuine opportunity to deceive or betray us.
    That’s crazy ambitious, but we have the best of the best. We have psychologists running incredibly realistic simulations of the humans Leviathan thinks it’s interacting with, and engineers calculating the latencies it should be experiencing with millisecond precision. There are artists generating videos of the world responding to whatever action...
    12 min
  • EA - Why we may expect our successors not to care about suffering by Jim Buhler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we may expect our successors not to care about suffering, published by Jim Buhler on July 11, 2023 on The Effective Altruism Forum.
    (Probably the most important post of this sequence.)Summary: Some values are less adapted to the “biggest potential futures” than others (see my previous post), in the sense that they may constrain how one should go about colonizing space, making them less competitive in a space-expansion race. The preference for reducing suffering is one example of a preference that seems particularly likely to be unadapted and selected against. It forces the suffering-concerned agents to make trade-offs between preventing suffering and increasing their ability to create more of what they value. Meanwhile, those who don’t care about suffering don’t face this trade-off and can focus on optimizing for what they value without worrying about the suffering they might (in)directly cause. Therefore, we should - all else equal - expect the “grabbiest” civilizations/agents to have relatively low levels of concern for suffering, including humanity (if it becomes grabby).
    Call this the Upside-focused Colonist Curse (UCC). In this post, I explain this UCC dynamic in more detail using an example. Then, I argue that the more significant this dynamic is (relative to competing others), the more we should prioritize s-risks over other long-term risks, and soon.
    The humane values, the positive utilitarians, and the disvalue penalty
    Consider the concept of disvalue penalty: the (subjective) amount of disvalue a given agent would have to be responsible for in order to bring about the highest (subjective) amount of value they can. The story below should make what it means more intuitive.
    Say they are only two types of agents:
    those endorsing “humane values” (the HVs) who disvalue suffering and value things like pleasure;
    the “positive utilitarians” (the PUs) who value things like pleasure but disvalue nothing.
    These two groups are in competition to control their shared planet, or solar system, or light cone, or whatever.
    The HVs estimate that they could colonize a maximum of [some high number] of stars and fill those with a maximum of [some high number] units of value. However, they also know that increasing their civilization’s ability to create value also increases s-risks (in absolute). They, therefore, face a trade-off between maximizing value and preventing suffering which incentivizes them to be cautious with regard to how they colonize space. If they were to purely optimize for more value without watching for the suffering they might (directly or indirectly) become responsible for, they’d predict they would cause x unit of suffering for every 10 units of value they create. This is the HVs’ disvalue penalty: x/10 (which is a ratio; a high ratio means a heavy penalty).
    The PUs, however, do not care about the suffering they might be responsible for. They don’t face the trade-off the HVs face and have no incentive to be cautious like them. They can - right away - start colonizing as many stars as possible to eventually fill them with value, without worrying about anything else. The PU’s disvalue penalty is 0.
    Image 1: Niander Wallace, a character from Blade Runner 2049 who can be thought of as a particularly baddy PU.
    Because they have a higher disvalue penalty (incentivizing them to be more cautious), the humane values are less “grabby” than those of the PUs. While the PUs can happily spread without fearing any downside, the HVs would want to spend some time and resources thinking about how to avoid causing too much suffering while colonizing space (and about whether it’s worth colonizing at all), since suffering would hurt their total utility. This means, according to the Grabby Values Selection Thesis, that we should - all else equal - expect PU-ish values to be s...
    20 min
  • LW - Open-minded updatelessness by Nicolas Macé
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open-minded updatelessness, published by Nicolas Macé on July 10, 2023 on LessWrong.
    Summary
    Bounded agents might be unaware of possibilities relevant to their decision-making. That is, they may not just be uncertain, but fail to conceive of some relevant hypotheses entirely. What's more, commitment races might pressure early AGIs into adopting an updateless policy from a position of limited awareness. What happens then when a committed AGI becomes aware of a possibility that’d have changed which commitment it’d have wanted to make in the first place? Motivated by this question, we develop "open-minded" extensions of updatelessness, where agents revise their priors upon experiencing awareness growth and reevaluate their commitment to a plan relative to the revised prior.
    Introduction
    Bounded agents may be unaware of propositions relevant to the decision problem they face. That is, they don’t merely have uncertainty, but also fail to conceive of the full set of possibilities relevant to their decision-making. (For example, when playing a board game, one might be unaware of some crucial rule. Moreover, one’s awareness might grow, e.g. when one discovers such a rule midgame.)Awareness growth raises questions for commitment. What if one commits, and then discovers an important consideration, whose conception would have changed the plan one would have wanted to commit to? The earlier one commits, the less time one has to think about the relevant considerations, and the more likely this problem is to arise.
    We are interested in preventing AGI systems from making catastrophic strategic commitments. One reason that not-fully-aware AGI systems could make bad commitments is that important hypotheses are missing for their priors. For example, they might fail to conceive of certain attitudes towards fairness that bargaining counterparts might possess. One might think that AGI agents would quickly become aware of all relevant hypotheses, and make commitments only then. But commitment race dynamics might pressure early AGIs into making commitments before thinking carefully, and in particular, in a position of limited awareness. From The Commitment Races problem (emphasis ours):
    If two consequentialists are playing a game of Chicken, the first one to throw out their steering wheel wins. [.] More generally, consequentialist agents are motivated to make commitments as soon as possible, since that way they can influence the behavior of other consequentialist agents who may be learning about them. Of course, they will balance these motivations against the countervailing motive to learn more and think more before doing drastic things. The problem is that the first motivation will push them to make commitments much sooner than would otherwise be optimal. So they might not be as smart as us when they make their commitments, at least not in all the relevant ways.
    In a commitment race, agents who are known to be updateless have a strategic advantage over their updateful counterparts. Therefore, commitment races might introduce pressures for agents to become updateless as soon as possible, and one might worry that early-stage AGIs hastily adopt a version of updatelessness that mishandles awareness growth.
    So we think it’s important to map out different ways one can go about being updateless when one’s awareness can grow. The aims of this post are to (1) argue for the relevance of unawareness to updatelessness and (2) explore several approaches to extending updatelessness to cases of awareness growth. Specifically, we introduce closed- and open-minded versions of updatelessness:
    An agent is closed-mindedly updateless if they ignore awareness growth and continue being committed to their current plan. This approach is dynamically consistent, but clearly undesirable if one’s prior leaves out imp...
    26 min