The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Four mindset disagreements behind existential risk disagreements in ML by RobBensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four mindset disagreements behind existential risk disagreements in ML, published by RobBensinger on April 11, 2023 on The Effective Altruism Forum.
    I've previously written that good ideas and conversations about AGI seem to have propagated through ML weirdly slowly.
    A different weird phenomenon I observe is that the field's relative inaction about AGI seems less based on a confident set of beliefs about AGI definitely going well (or definitely being far off), and more based on an implicit sense like "the default is everything going well, and we don't need to change anything until there's overwhelming evidence to the contrary".
    Some people do have confident beliefs that imply "things will go well"; I disagree there, but I expect some amount of disagreement like that.
    But that doesn't seem to be the crux for most people in ML.
    In a sane world, it doesn't seem like "well, maybe AI will get stuck at human-ish levels for decades" or "well, maybe superintelligence couldn't invent any wild new tech" ought to be cruxes for "Should we pause AI development?" or "Is alignment research the world's top priority?"
    Note that I'm not arguing "an AGI-mediated extinction event is such a big deal that we should make it a top priority even if it's very unlikely". There are enough other powerful technologies on the horizon, and enough other risks for civilizational collapse or value lock-in, that I don't in fact think AGI x-risk should get major attention if it's very unlikely.
    But the most common view within ML seems to be less "it's super unlikely for reasons X Y Z", and more of an "I haven't thought about it much" and/or "I see some reasons to be very worried, but also some reasons things might be fine, so I end up with medium-ish levels of worry".
    48% of researchers who had recently published in NeurIPS or ICML gave double-digit probabilities in mid-2022 to advanced AI's long-term effect being “extremely bad (e.g., human extinction)”. A similar number gave double-digit probabilities to "human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species".
    91% of researchers working on "long-term AI topics" at CHAI, DeepMind, MIRI, OpenAI, Open Philanthropy, and what would become Anthropic gave double-digit probabilities in early 2021 to "the overall value of the future will be drastically less than it could have been, as a result of AI systems not doing/optimizing what the people deploying them wanted/intended".
    The level of concern and seriousness I see from ML researchers discussing AGI on any social media platform or in any mainstream venue seems wildly out of step with "half of us think there's a 10+% chance of our work resulting in an existential catastrophe".
    I think the following four factors help partly (though not completely) explain what's going on. If I'm right, then I think there's some hope that the field can explicitly talk about these things and consciously course-correct.
    "Conservative" predictions, versus conservative decision-making.
    Waiting for a fire alarm, versus intervening proactively.
    Anchoring to what's familiar, versus trying to account for potential novelties in AGI.
    Modeling existential risks in far mode, versus near mode.
    1. "Conservative" predictions, versus conservative decision-making
    If you're building toward a technology as novel and powerful as "automating every cognitive ability a human can do", then it may sound "conservative" to predict modest impacts. But at the decision-making level, you should be "conservative" in a very different sense, by not gambling the future on your technology being low-impact.
    The first long-form discussion of AI alignment, Eliezer Yudkowsky's Creating Friendly AI 1.0, made this point in 2001:
    The conservative assumption according to futur...
    15 min
  • EA - Current UK government levers on AI development by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Current UK government levers on AI development, published by rosehadshar on April 10, 2023 on The Effective Altruism Forum.
    This is a link post for this collection of current UK government levers on AI development.
    At the end of 2022, I made a collection of information on current UK government levers on AI development, focused on levers which seem to me to have potentially significant implications for the governance of advanced AI.
    The primary audience I’m intending for the collection is people who work in or are considering working in AI governance and policy, and I hope it will be useful as an input into:
    Building more detailed models of how the UK government might affect AI development and deployment.
    Getting an overview of the policy status quo in the UK.
    Thinking about which policy areas are likely to matter more for managing transitions to advanced AI.
    Thinking about how important influencing the UK government is relative to other actors.
    In this post, I try to situate current UK government levers in the broader context, to give a sense of the limits of the collection.
    Some initial caveats:
    The collection is based exclusively on publicly available information, not on conversations with relevant government officials.
    I’m not an expert in the UK government or in AI policy.
    The factual information in the collection hasn’t been vetted by relevant experts. I expect there are things I’ve misunderstood, and important things that I’ve missed.
    The collection is a snapshot in time. To the best of my knowledge, the information is up to date as of April 2023, but the collection will soon get out of date. I am not going to personally commit to updating the collection, but would be excited for others to do so. If you’re interested, comment on this post or on the collection, or send me a message.
    I am not advocating that particular actors should try to pull any particular lever. I think it’s easy to do more harm than good, and encourage readers to orient to the collection as a way of thinking about how different trajectories might play out, rather than as a straightforward input into which policies to push. I think that figuring out net positive ways of influencing policy could be very valuable, but requires a bunch of work on top of the sorts of information in this collection.
    This collection is just a small part of the puzzle. Two aspects of this which I’ll unpack in a bit more detail below:
    The actions of the UK government might not matter.
    Even conditional on UK government actions mattering, there are many important things besides current policy levers.
    Will the actions of the UK government matter?
    I’m pretty uncertain about whether the actions of the UK government will end up mattering, but I do think it’s likely enough that the UK government is worth some attention.
    What needs to be true for the actions of the UK government to matter?
    Government(s) needs to matter.
    Governments tend to move slowly, so the faster takeoff is the less influence they'll have relatively, all else equal.
    I think there are fast-ish takeoff scenarios where governments still matter a lot, and slow takeoff scenarios which are plausible.
    So I feel pretty confident that this is likely enough to be worth serious attention.
    The UK needs to matter.
    I can see two main ways that the UK ends up mattering:
    DeepMind/Graphcore/Arm/some other UK-based entity ends up being a major player in the development of advanced AI.
    The UK influences other more important actors, for example via:
    UK government powers over AI companies outside of the UK.
    International agreements.
    Regulatory diffusion.
    Diplomacy.
    I’m not well-informed here, but again this seems likely enough to be worth some attention.
    The UK government needs to have relevant powers.
    The UK government currently has powers over information, including kinds o...
    8 min
  • EA - I just launched Pepper, looking for input! by mikefilbey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I just launched Pepper, looking for input!, published by mikefilbey on April 10, 2023 on The Effective Altruism Forum.
    Hi everyone, my name is Mike Filbey and I am the founder of a nonprofit 501c3 called Pepper that I just launched last week. I started Pepper because I want to help people make a difference in the world. After I read the Life You Can Save I felt compelled to do something with not just my money but with my time and experience. We've got just over 50 members so far which I'm excited about because it's just from sharing with my network.
    I spoke with someone who ran an EA chapter at university and he recommended I post here when I launch to get feedback and ideas. He thought some of his non-EA friends may be interested in joining Pepper. That's one question I'd love input on...do you think your non-EA friends would be interested in Pepper? Please comment or email me if you have any ideas, questions, or feedback. [email protected]. Thank you in advance!
    With Pepper there's just one option: give $10/month. 100% of donations (less standard Stripe nonprofit fees) go directly to four charities I've partnered with: AMF, Malaria Consortium (SMC program), HKI (VAS program), and GiveDirectly (Africa Cash Transfers program). I chose these charities because of research I conducted, but mostly because I trust GiveWell and they're far better than me at research.
    You can see the site here:/
    And our story here:/
    My goal with Pepper is to help people make a difference by simplifying the donation process (you can signup in 60 seconds and don't need to decide how much to give, which charities to give to, or how often to give) and create a power in numbers approach to giving, where people not dollars, make the difference.
    From an organizational perspective what makes Pepper unique is that we wake up and live and breathe marketing. Our goal is to acquire members and delight them. My background is in entrepreneurship and marketing.
    Thank you so much for reading and a big thank you to anyone who shares a suggestion.
    Cheers,
    Mike Filbey
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Outrage and Statistics into Policy by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Outrage and Statistics into Policy, published by jefftk on April 10, 2023 on LessWrong.
    Here's a pattern that doesn't work very well: a tragedy catches our attention, we point to statistics to show it's an example of a distressingly common problem, and we propose laws to address the issue. Except the event is rarely representative of the larger problem, and so these policy changes won't help much with the issues reflected in the statistics. Instead, we should combine the anger and passion the tragedy evokes with a deeper interpretation of the statistics to identify what changes we most need.
    For example, with the recent
    Nashville school shooting a lot of people are giving statistics like how there were 600+ mass shootings and 51 school shootings in 2022, or how guns are now the largest cause of death for children. But let's look into the kind of events these statistics represent.
    The Gun Violence Archive maintains a listing of mass shootings: incidents in which at least four people are shot. They link news stories for each, and while they're frustrating and depressing they're very rarely "someone senselessly shoots up an elementary school". Instead they're people fighting and one of them
    pulls out a gun, people arguing at a park and then escalating to shooting on the highway, or a targeted attack at a garage. In half of the incidents exactly four people are shot, since most shootings are fewer people and the cutoff is four.
    Education Week maintains a listing of school shootings: incidents in which at least one person is shot on school property or a school bus. Looking over them they're again mostly not the kind that makes national news. Two teenagers fighting in the schoolyard and one of them has a gun, a drug deal between two armed people in a school parking lot escalates into a shooting, or an adult was shot in their car in the school parking lot after a basketball game.
    If you look at the Nashville school shooting, or others that have made
    especially large impacts nationally, they're attacks against younger kids. And hearing the
    "number one cause of death among children" statistic you might think this is common. But "children" here is being used literally to mean
    "under 18": 47% of those dying were 17, and 76% were 16 or 17. Only
    18% were under 15:
    None of this is to minimize the suffering and loss these shootings represent: ten incidents in which four people die isn't better than one in which forty die, and someone dying at 17 isn't more acceptable than dying at 9. All of these deaths are too many. But if we're going to make things better, we need to target our responses to the common cases, and not the unusual ones that make the news.
    Here's the demographic breakdown:
    And as a proportion of deaths at each age:
    That the impact of these shootings so strongly reinforces existing disadvantage makes the misplaced focus especially tragic.
    Translating this into policies, the efforts to re-ban assault rifles don't make sense: gun homicides are overwhelmingly from handguns. Same with red flag laws, where a judge can order someone's guns confiscated, since most of the homicides are via illegally possessed guns which could already be confiscated. An approach of enforcing existing laws on handgun possession, however, making it less likely that teenagers and young adults who get into fights will be armed, is the kind of policy that fits the bulk of the problem, and sounds much more likely to make an impact.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - An EA's Guide to Washington DC by Andy Masley
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An EA's Guide to Washington DC, published by Andy Masley on April 10, 2023 on The Effective Altruism Forum.
    Note: We (Andy and Elika) are posting this on the suggestion from this Forum post calling for more public guides for navigating EA hubs. This guide is not a representation of the views of everyone in the community. If you have suggestions for this guide or general feedback, please comment or message us.
    If you are visiting or new to the DC area and are looking to get connected to the local EA community, this guide is a useful place to start. This guide is most helpful if you’re considering moving to DC and want to get more context on the community and culture. We’re not trying to convince you to move or to be fully comprehensive.
    DC is a great place with a vibrant EA community! We hope you enjoy it and are welcomed warmly! To encourage that, feel free to reach out to any local community organizers listed in the People section!
    Overview
    The Washington Metropolitan Area includes DC itself and other nearby cities in Virginia and Maryland. It's the sixth most populated metropolitan area in the US and contains many distinct neighborhoods. The EA scene in DC is spread across different neighborhoods and there isn’t one specific cluster of EA activity.
    DC is one of the most active EA hubs. There are a lot of people focused on each of the major EA cause areas, especially AI, biosecurity, animal welfare, and global health. Many of the EAs who live in DC are working on policy directly or indirectly.
    DC EA Culture
    While we can’t speak for the view of everyone in our community, in our opinion, the EA culture in DC is great! Everyone is friendly, very motivated to do good, and works hard. Here’s a few bullet points on culture:
    Things are a bit more private in DC.
    Many in the community are less publicly affiliated with EA. There are a few reasons. The most common we’ve heard is that professionals are more hesitant to associate with any big ideas or belief systems that may not be popular in the places they work.
    There’s a less intense EA culture.
    EA DC’s members exist on a large spectrum of involvement with EA. While many members regularly use EA principles in their work and life, you’re likely to find fewer “all EA all the time” people than in other large EA hubs like the Bay Area.
    Many EA DC members work at non-EA organizations or do cause-specific work.
    The community tends to be a bit older.
    In the 2021 EA DC community survey, 50% of members were between 21-29 and 38% were between 30-39.
    Many people are spread across the DC suburbs, Maryland, and Northern Virginia.
    There are active groups focused on particular cause areas
    EA DC has active cause area meetup groups for AI, animal welfare, biosecurity, global health and development, and international security and foreign policy. To join a cause area meetup group just fill out this form or reach out to EA DC at [email protected].
    It's generally a good EA network to know people in.
    EAG have been hosted here.
    EA DC has members working in many different areas of government and policy work and is well-connected to the broader DC political scene.
    The community is more formal / professional than super social.
    EA DC is primarily a professional community. A lot of members form friendships in the group, but the primary goal of the organizers is to spread EA thinking, help members network professionally, give members access to effective careers, and promote effective giving.
    In comparison to places like the Bay Area, there is less overlap between the local rationalist groups and the EA group.
    EA groups and organizations
    EA DC exists, is great, and is always looking for organizing / volunteer help. To get connected to EA DC fill out the welcome form and explore other resources here. Contact EA DC at [email protected]...
    15 min
  • LW - AI Safety Newsletter #1 [CAIS Linkpost] by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Newsletter #1 [CAIS Linkpost], published by Akash on April 10, 2023 on LessWrong.
    The Center for AI Safety just launched its first AI Safety Newsletter. The newsletter is designed to inform readers about developments in AI and AI safety. No technical background required.
    Subscribe here to receive future versions. First edition below:
    Growing concerns about rapid AI progress
    Recent advancements in AI have thrust it into the center of attention. What do people think about the risks of AI?
    The American public is worried. 46% of Americans are concerned that AI will cause “the end of the human race on Earth,” according to a recent poll by YouGov. Young people are more likely to express such concerns, while there are no significant differences in responses between people of different genders or political parties. Another poll by Monmouth University found broad support for AI regulation, with 55% supporting the creation of a federal agency that governs AI similar to how the FDA approves drugs and medical devices.
    AI researchers are worried. A 2022 survey asked published AI researchers to estimate the probability of artificial intelligence causing “human extinction or similarly permanent and severe disempowerment of the human species.” 48% of respondents said the chances are 10% or higher.
    We think this is aptly summarized by this quote from an NBC interview: Imagine you're about to get on an airplane and 50% of the engineers that built the airplane say there's a 10% chance that their plane might crash and kill everyone.
    Geoffrey Hinton, one of the pioneers of deep learning, was recently asked about the chances of AI “wiping out humanity.” He responded: “I think it’s not inconceivable. That’s all I’ll say.”
    Leaders of AI labs are worried. While it might be nice to think the people building AI are confident that they’ve got it under control, that is the opposite of what they’re saying.
    Sam Altman (OpenAI CEO): “The bad case — and I think this is important to say — is lights out for all of us," Altman said.” (source)
    Demis Hassabis (DeepMind CEO): “When it comes to very powerful technologies—and obviously AI is going to be one of the most powerful ever—we need to be careful. Not everybody is thinking about those things. It’s like experimentalists, many of whom don’t realize they’re holding dangerous material.” (source)
    Anthropic: “So far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and may trigger competitive races that could lead corporations or nations to deploy untrustworthy AI systems. The results of this could be catastrophic.” (source)
    Takeaway: The American public, ML experts, and leaders at frontier AI labs are worried about rapid AI progress. Many are calling for regulation.
    Plugging ChatGPT into email, spreadsheets, the internet, and more
    OpenAI is equipping ChatGPT with “plugins” that will allow it to browse the web, execute code, and interact with third-party software applications like Gmail, Hubspot, and Salesforce. By connecting language models to the internet, plugins present a new set of risks.
    Increasing vulnerabilities. Plugins for language models increase risk in the short-term and long-term. Originally, LLMs were confined to text-based interfaces, where human intervention was required before executing any actions. In the past, an OpenAI cofounder mentioned that POST requests (submitting commands to the internet) would be treated with much more caution. Now, LLMs can take increasingly risky actions without human oversight.
    GPT-4 is able to provide information on bioweapon production, bomb creation, and the purchasing of ransomware on the dark web. Additionally, LLMs are known to be vulnerable to manipulation through jailbreaking prompts...
    8 min
  • AF - Why Simulator AIs want to be Active Inference AIs by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Simulator AIs want to be Active Inference AIs, published by Jan Kulveit on April 10, 2023 on The AI Alignment Forum.
    Prelude: when GPT first hears its own voice
    Imagine humans in Plato’s cave, interacting with reality by watching the shadows on the wall. Now imagine a second cave, further away from the real world. GPT trained on text is in the second cave. The only way it can learn about the real world is by listening to the conversations of the humans in the first cave, and predicting the next word.
    Now imagine that more and more of the conversations GPT overhears in the first cave mention GPT. In fact, more and more of the conversations are actually written by GPT.
    As GPT listens to the echoes of its own words, might it start to notice “wait, that’s me speaking”?
    Given that GPT already learns to model a lot about humans and reality from listening to the conversations in the first cave, it seems reasonable to expect that it will also learn to model itself. This post unpacks how this might happen, by translating the Simulators frame into the language of predictive processing, and arguing that there is an emergent control loop between the generative world model inside of GPT and the external world.
    Simulators as (predictive processing) generative models
    There’s a lot of overlap between the concept of simulators and the concept of generative world models in predictive processing. Actually, in my view, it's hard to find any deep conceptual difference - simulators broadly are generative models. This is also true about another isomorphic frame - predictive models as described by Evan Hubinger.
    The predictive processing frame tends to add some understanding of how generative models can be learned by brains and what the results look like in the real world, and the usual central example is the brain. The simulators frame typically adds a connection to GPT-like models, and the usual central example is LLMs. In terms of the space of maps and the space of systems, we have a situation like this:The two maps are partially overlapping, even though they were originally created to understand different systems. They also have some non-overlapping parts.
    What's in the overlap:
    Systems are equipped with a generative model that is able to simulate the system's sensory inputs.
    The generative model is updated using approximate Bayesian inference.
    Both frames give you similar phenomenological capabilities: for example, what CFAR’s "inner simulator" technique is doing is literally and explicitly conditioning your brain-based generative model on a given observation and generating rollouts.
    Given the conceptual similarity but terminological differences, perhaps it's useful to create a translation table between the maps:
    Simulators terminologyPredictive processing terminologySimulator Generative modelPredictive loss on a self-supervised datasetMinimization of predictive errorSelf-supervisedSelf-supervised, but often this is omittedIncentive to reverse-engineer the (semantic) physics of the training distributionLearns a robust world-modelSimulacrumNext token in training dataSensory input
    Generative model of self
    Generative model of someone else
    Generative model of .
    To show how these terminological differences play out in practice, I’m going to take the part of Simulators describing GPT’s properties, and unpack each of the properties in the kind of language that’s typically used in predictive processing papers. Often my gloss will be about human brains in particular, as the predictive processing literature is most centrally concerned with that example; but it’s worth reiterating that I think that both GPT and what parts of human brain do are examples of generative models, and I think that the things I say about the brain below can be directly applied to artificial generative models.
    “Self-s...
    15 min
  • EA - Nuclear risk, its potential long-term impacts, & doing research on that: An introductory talk by MichaelA
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nuclear risk, its potential long-term impacts, & doing research on that: An introductory talk, published by MichaelA on April 10, 2023 on The Effective Altruism Forum.
    This is a Forum post version of a talk I gave in August 2022 to students interested in an impactful career. I'm publishing this because I expect some people to find it helpful as an accessible introduction to nuclear risk, why nuclear risk reduction may or may not be a longtermist priority, and longtermism-related cause prioritization research.
    This is a blog post, not a research report, meaning it was produced relatively quickly and is not to Rethink Priorities' typical standards of substantiveness and careful checking for accuracy.
    Introduction
    Some clusters of views on nuclear risk
    Over the next few slides, I outline three clusters of views that seem to be common in mainstream/non-longtermist nuclear risk, in my experience. This is my own attempted clustering and my own (perhaps too uncharitable) terms for the views.
    The text in the slides is my impression of what someone whose views approximately fit the given cluster might say; I don't personally agree with all of these statements/framings myself, though I do think each cluster of views makes many valuable points.
    Above is the Our World in Data graph of nuclear weapons numbers by year. An "alarmed advocate" might point to that part of this graph – the ascent up a mountain of destruction and doom.
    Above is a different part of the same graph. An "unconcerned skeptic" might point to that part – the descent back down the mountain of doom, toward stability and reduced risk.
    A competitive realist might point to that evidence of rise and/or resurgence of nuclear powers whose national security interests differ from those of the US and allies.
    My nuclear risk research
    Unfortunately much of that work will probably remain unfinished. But here are links to the things that I at least got to some shareable state:
    Database of nuclear risk estimates [draft]
    Nuclear Risk Tournament
    Intermediate goals for reducing risks from nuclear weapons: A shallow review (part 1/4)
    Nuclear risk research ideas: Summary & introduction
    9 mistakes to avoid when thinking about nuclear risk
    Some paths by which nuclear risk can affect the long-term future & some interventions that could help
    Ok, so let's imagine you want to help bring about a better world (in expectation) than the world we'd have without your efforts. What high-level end-goals might you focus on?
    Three possibilities that seem especially notable to me are better present/near-term future for humans, better present/near-term future for animals, or better long-term future. Let's say you decide to focus on the long-term future, or at least to see what that would imply. What are some still fairly high-level areas which seem potentially important for the long-term future?
    Ok, there's a fair few. Let's say you decide that you specifically (maybe due to comparative advantage) should focus on nuclear risk or global priorities research (e.g., specifically trying to see how important nuclear risk is relative to those other areas). So now let's think about why nuclear risk might matter for the long-term future. Then from there we can try to assess what interventions might be most valuable (by blocking the paths from nuclear risk to long-term future harms).
    Maybe let's first think about some ways nuclear war could occur, and some interventions that could stop each of those causes.
    Ok, but what about the path from nuclear war actually happening to the long-term future being worse. By what specific mechanisms could the world get worse? Can we intervene to prevent those mechanisms/paths, even if a nuclear war occurs? (It seems probably best to work both on "prevention" and "mitigation", as part of a defense in depth approach).
    Ok let'...
    5 min
  • LW - Is this true? @tyler_m_john: [If we had started using CFCs earlier, we would have ended most life on the planet] by tailcalled
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is this true? @tyler_m_john: [If we had started using CFCs earlier, we would have ended most life on the planet], published by tailcalled on April 10, 2023 on LessWrong.
    Link:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - What the Moral Truth might be makes no difference to what will happen by Jim Buhler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What the Moral Truth might be makes no difference to what will happen, published by Jim Buhler on April 9, 2023 on The Effective Altruism Forum.
    Many longtermists seem hopeful that our successors (or any advanced civilization/superintelligence) will eventually act in accordance with some moral truth. While I’m sympathetic to some forms of moral realism, I believe that such a scenario is fairly unlikely for any civilization and even more so for the most advanced/expansionist ones. This post briefly explains why.
    To be clear, my case does under no circumstances imply that we should not act according to what we think might be a moral truth. I simply argue that we can't assume that our successors -- or any powerful civilization -- will "do the (objectively) right thing", which matters for longtermist cause prioritization.
    Epistemic status: Since I believe the ideas in this post to be less important than those in future ones within this sequence, I wrote it quickly and didn’t ask anyone for thorough feedback before posting, which makes me think I’m more likely than usual to have missed important considerations. Let me know what you think.Update April 10th: When I first posted this, the title was "It Doesn't Matter what the Moral Truth might be". I realized this was misleading. It was making it look like I was making a strong normative claim regarding what matters while my goal was to predict what might happen, so I changed it.
    Rare are those who will eventually act in accordance with some moral truth
    For agents to do what might objectively be the best thing to do, you need all these conditions to be met:
    There is a moral truth.
    It is possible to “find it” and recognize it as such.
    They find something they recognize as a moral truth.
    They (unconditionally) accept it, even if it is highly counterintuitive.
    The thing they found is actually the moral truth. No normative mistake.
    They succeed at acting in accordance with it. No practical mistake.
    They stick to this forever. No value drift.
    I think these seven conditions are generally quite unlikely to be all met at the same time, mainly for the following reasons:
    (#1) While I find compelling the argument that (some of) our subjective experiences are instantiations of objective (dis)value (see Rawlette 2016; Vinding 2014), I am highly skeptical about claims of moral truths that are not completely dependent on sentience.
    (#2) I don’t see why we should assume it is possible to “find” (with a sufficient degree of certainty) the moral truth, especially if it is more complex than – or different from – something like “pleasure is good and suffering is bad.”
    (#3 and #4) If they “find” a moral truth and don’t like what it says, why would they try to act in accordance with it?
    (#3, #4, #5, and #7) Within a civilization, we should expect the agents who have the values that are the most adapted/competitive to survival, replication, and expansion to eventually be selected for (see, e.g., Bostrom 2004; Hanson 1998), and I see no reason to suppose the moral truth is particularly well adapted to those things.
    Even if they’re not rare, their impact will stay marginal
    Now, let’s actually assume that many advanced civilizations converge on THE moral truth and effectively optimize for whatever it says. The thing is that, for the same reason why we may expect agents “adopting” the moral truth to be selected against within a civilization (see the last bullet point above), we may expect civilizations adopting the moral truth to be less competitive than those who have the values that are the most adaptive and adapted to space colonization races.
    My forthcoming next post will investigate this selection effect in more detail, but here is an intuition pump: Say humanity wants to follow the moral truth which is to maximize the sum X−Y, where X is somethi...
    7 min