The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - More money with less risk: sell services instead of model access by Luke H Miles
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More money with less risk: sell services instead of model access, published by Luke H Miles on March 4, 2023 on The AI Alignment Forum.
    OpenAI is currently charging 100,000 times less per line of code than professional US devs.[1]
    An LLM's code output is of course less reliable than a professional's. And it is hard to use a text-completion API effectively in large projects.
    What should you do if you've got a model on your hands that solves those problems?
    You could operate as a software development company. They tend[2] to charge $100-200k for simple mobile apps and there's basically no ceiling on the cost for complex apps over their lifetime. Devs make up the majority of a normal firm's personnel and costs; coding takes most of the app development time; bugs in code are one of the primary sources of project extension and failures. By using your model you can make better software, complete it faster, succeed more often, charge a lower price, and make a higher profit.
    Going further, if you've really got a good model, then you can do very well by building competitors to adobe products, salesforce products, SAP products, google search, mongodb, etc.
    Someone who has a build-anything machine would be a fool to sell a cheap build-anything service instead of using it themselves and selling the result. Particularly because selling the general service directly is likely to encourage and inspire copycats, including open-source ones who will delete your market. If it really builds the entire thing then you'll probably also be liable for negative consequences, which again have no ceiling.
    Fewer risks, big and small
    Some common misuse risks you can avoid/reduce (and eliminate associated liability):
    Someone tricks your API into doing something awful and pastes it into a tweet
    Spam generation for political campaigns, cryptocurrencies, etc
    Common hacking ("write a test to see if my server has a log4j vulnerability")
    Targeted manipulation and spearphishing
    Larger risks you can avoid/reduce:
    Your incredible model motivates countless AI researchers. People reverse-engineer some of the architecture in online discussions. The state of the art is quickly advanced. We have less time to prepare for strong general AI.
    Hackers steal your model weights (if you don't advertise your model then you'll attract less attention from hackers)
    People try to get your model to act like an agent and copy itself around. They succeed. You have no way of shutting it down or monitoring what it is doing.
    Someone tries to get your model to order and mail smallpox or a novel virus. The screenshot would be an epic tweet. They succeed oh no
    Your own AI devs' ambitions and risk-tolerance know no bounds because you've positioned yourself as an AI company instead of a product company; there is nothing to keep their hands busy except make the AI more generally capable and efficient. They are careless with the training runs and one day your model gets loose and wreaks havoc.
    Biology, robotics, R&D, etc
    The benefits of selling/publishing derived products and the downsides of offering direct access remain in other domains:
    A drug is more profitable and less risky (for the world at least) than a general drug designer
    A vaccine is more profitable and less risky than a general mRNA designer
    There's more people who want to buy a house than a house-building robot
    There's more people who need a (highly efficient, AI assisted) lawyer than a general lawyer's assistant.
    More people need a cleaning robot than a robot-maker
    Releasing or building an effective fusion power generator gets you more clout than releasing the design assistant
    Even if you're evil and want to make AI-astroturf campaign spam, you presumably want to help one side more than the other, but if you release your model/tooling then both sides will use it.
    If you ha...
    5 min
  • LW - Contra Hanson on AI Risk by Liron
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Hanson on AI Risk, published by Liron on March 4, 2023 on LessWrong.
    Robin Hanson wrote a new post recapping his position on AI risk (LW discussion). I've been in the Eliezer AI-risk camp for a while, and while I have huge respect for Robin’s rationality and analytical prowess, the arguments in his latest post seem ineffective at drawing me away from the high-doom-worry position.
    Robin begins (emphasis mine):
    First, if past trends continue, then sometime in the next few centuries the world economy is likely to enter a transition that lasts roughly a decade, after which it may double every few months or faster, in contrast to our current fifteen year doubling time. (Doubling times have been relatively steady as innovations are typically tiny compared to the world economy.) The most likely cause for such a transition seems to be a transition to an economy dominated by artificial intelligence (AI). Perhaps in the form of brain emulations, but perhaps also in more alien forms. And within a year or two from then, another such transition to an even faster growth mode might plausibly happen.
    And adds later in the post:
    The roughly decade duration predicted from prior trends for the length of the next transition period seems plenty of time for today’s standard big computer system testing practices to notice alignment issues.
    Robin is extrapolating from his table in Long-Term Growth As A Sequence of Exponential Modes:
    I get that there’s a trend here. But I don’t get what inference rule Robin's trend-continuation argument rests on.
    Let’s say you have to predict whether dropping a single 100-megaton nuclear bomb on New York City is likely to cause complete human extinction. (For simplicity, assume it was just accidentally dropped by the US on home soil, not a war.)
    As far as I know, the most reliably reality-binding kind of reasoning is mechanistic: Our predictions about what things are going to do rest on deduction from known rules and properties of causal models of those things.
    We should obviously consider the causal implications of releasing 100 megatons worth of energy, and the economics of having a 300-mile-wide region wiped out.
    Should we also consider that a nuclear explosion that decimates the world economy would proceed in minutes instead of years, thereby transitioning our current economic regime much faster than a decade, thus violating historical trends? I dunno, this trend-breaking seems totally irrelevant to the question of whether a singular 100-megaton nuke could cause human extinction.
    Am I just not applying Robin’s trend-breaking reasoning correctly? After all, previous major human economic transitions were always leaps forward in productivity, while this scenario involves a leap backward.
    Ok, but what are the rules for this trend-extrapolation approach supposed to be? I have no idea when I’m allowed to apply it.
    I suspect the only way to know a rule like “don’t apply economic-era extrapolation to reason about the risk of a single bomb causing human extinction” is to first cheat and analyze the situation using purely mechanistic reasoning. After that, if there’s a particular trend-extrapolation claim that feels on-topic, you can say it belongs in the mix of reasoning types that are supposedly applicable to the situation.
    In our nuke example, there are two ways this could play out:
    If your first-pass mechanistic reasoning lands you far from what’s predicted by trend extrapolation, e.g. if it says every human on earth dies within minutes, then hey, we’re obviously talking about a freak event and not about extrapolating economic trends. Duh, economic models aren’t designed to talk about a one-off armageddon event. You have to pick the right model for the scenario you want to analyze! Can I interest you in a model of extinction events? Did you know we’re a...
    14 min
  • LW - AI Governance & Strategy: Priorities, talent gaps, & opportunities by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Governance & Strategy: Priorities, talent gaps, & opportunities, published by Akash on March 3, 2023 on LessWrong.
    Over the last few weeks, I’ve had 1:1s with several AI governance professionals to develop better models of the current AI governance & strategy landscape. Some topics that regularly came up include:
    What are the most important/neglected areas in AI governance & strategy?
    What is the theory of change behind projects in these areas, and where might they break?
    What are the current talent gaps & opportunities in these areas?
    This post is my attempt to summarize some takeaways from those conversations. I list some “priority areas” in AI governance & strategy, summarize them briefly, and describe potential talent gaps in each area. I don't claim that my list is comprehensive, and I welcome people to add their own ideas in the comments.
    If you think you may have some of the relevant talents/aptitudes and are interested in working in any of these areas, feel free to reach out to me, and I may connect you to relevant professionals. (Feel free to have a low bar for reaching out; I'll ask you for more information if needed.)
    Please also be aware that there are downside risks in each of these areas. I suggest you get in touch with relevant professionals before “jumping in” to any of these areas.
    Priority Areas
    I refer to “priority areas” as topics that frequently came up when talking with AI governance professionals. Caveats: This is not a rigorous method, this list is not comprehensive, some topics were excluded intentionally, the list probably overweights topics that I evaluate as valuable (on my inside view), and priorities will inevitably change as the field continues to evolve.
    For each priority area, I offer a brief summary, as well as a description of the kinds of career aptitudes that might make someone an especially good fit for working in the area.
    Model evaluations
    Summary: There are many ways models could be dangerous, but it’s difficult to detect these failure modes. Can we develop and implement “tests” that help us determine if a model is dangerous?
    Some people are working on technical tests that can determine if a model has dangerous capabilities or appears to be misaligned. Others are thinking more broadly about what kinds of evals would be useful. Some people are focused on creating agreements that labs or governments could implement (e.g., if a Deception Eval is triggered, everyone agrees to stop scaling until Y evidence is acquired).
    Current gaps:
    Engineers: People with strong engineering backgrounds are needed to build/implement technical evals.
    Strong conceptual alignment models: People with strong models of alignment and LLMs are needed to figure out what kinds of evals ought to be built, what specific failure modes might show up in the upcoming years, and what kind of “counterevidence” would be required to justify scaling after an eval has been triggered.
    Experience thinking about or implementing agreements across a variety of stakeholders: People with experience implementing agreements across a variety of stakeholders are needed to figure out how evals will be implemented across the industry, what kinds of agreements will be feasible, which stakeholders will be required to sign-on, and what kinds of needs/concerns those stakeholders are likely to have.
    Additional resources: See this post by Beth, this post by me, and this paper by Ethan Perez.
    Compute governance
    Summary: AI progress has largely been driven by compute. Can we understand compute trends and identify regulations based on compute?
    Current gaps:
    Technical talent: People with background in technical fields are needed to support AI governance professionals who are thinking about compute governance. Hardware knowledge is especially helpful, but even broad technical skills are use...
    8 min
  • LW - How popular is ChatGPT? Part 2: slower growth than Pokémon GO by Richard Korzekwa
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How popular is ChatGPT? Part 2: slower growth than Pokémon GO, published by Richard Korzekwa on March 3, 2023 on LessWrong.
    Rick Korzekwa, March 3, 2023
    A major theme in reporting on ChatGPT is the rapid growth of its user base. A commonly stated claim is that it broke records, with over 1 million users in less than a week and 100 million users in less than two months. It seems not to have broken the record, though I do think ChatGPT’s growth is an outlier.
    Checking the claims
    ChatGPT growth
    From what I can tell, the only source for the claim that ChatGPT had 1 million users in less than a week comes from this tweet by Sam Altman, the CEO of OpenAI:
    I don’t see any reason to strongly doubt this is accurate, but keep in mind it is an imprecise statement from a single person with an incentive to promote a product, so it could be wrong or misleading.
    The claim that it reached 100 million users within two months has been reported by many news outlets, which all seem to bottom out in data from Similarweb. I was not able to find a detailed report, but it looks like they have more data behind a paywall. I think it’s reasonable to accept this claim for now, but, again, it might be different in some way from what the media is reporting1.
    Setting records and growth of other apps
    Claims of record setting
    I saw people sharing graphs that showed the number of users over time for various apps and services. Here is a rather hyperbolic example:
    That’s an impressive curve and it reflects a notable event. But it’s missing some important data and context.
    The claim that this set a record seems to originate from a comment by an analyst at investment bank UBS, who said “We cannot remember an app scaling at this pace”, which strikes me as a reasonable, hedged thing to say. The stronger claim that it set an outright record seems to be misreporting.
    Data on other apps
    I found data on monthly users for all of these apps except Spotify2. I also searched lists of very popular apps for good leads on something with faster user growth. You can see the full set of data, with sources, here.3 I give more details on the data and my methods in the appendix.
    From what I can tell, that graph is reasonably accurate, but it’s missing Pokémon GO, which was substantially faster. It’s also missing the Android release of Instagram, which is arguably a new app release, and surpassed 1M within the first day. Here’s a table summarizing the numbers I was able to find, listed in chronological order:
    ServiceDate launchedDays to 1MDays to 10MDays to 100MNetflix subscribers (all)1997-08-29366941857337Facebook2004-02-043319501608Twitter2006-07-156709551903Netflix subscribers (streaming)2007-01-15188923513910Instagram (all)2010-10-0161362854Instagram (Android)2012-04-031Pokemon Go (downloads)2016-07-05727ChatGPT2022-11-30461
    It’s a little hard to compare early numbers for ChatGPT and Pokémon GO, since I couldn’t find the days to 1M for Pokémon GO or the days to 10M for ChatGPT, but it seems unlikely that ChatGPT was faster for either.
    Analysis
    Scaling by population of Internet users
    The total number of people with access to the Internet has been growing rapidly over the last few decades. Additionally, the growth of social networking sites makes it easier for people to share apps with each other. Both of these should make it easier for an app to spread. With that in mind, here’s a graph showing the fraction of all Internet users who are using each app over time (note the logarithmic vertical axis):
    In general, it looks like these curves have initial slopes that are increasing with time, suggesting that how quickly an app can spread is influenced by more than just an increase in the number of people with access to the Internet. But Pokémon GO and ChatGPT just look like vertical lines of different heights, so here’s anoth...
    9 min
  • EA - Misalignment Museum opens in San Francisco: ‘Sorry for killing most of humanity’ by Michael Huang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misalignment Museum opens in San Francisco: ‘Sorry for killing most of humanity’, published by Michael Huang on March 4, 2023 on The Effective Altruism Forum.
    A new AGI museum is opening in San Francisco, only eight blocks from OpenAI offices.
    SORRY FOR KILLING MOST OF HUMANITY
    Misalignment Museum Original Story Board, 2022
    Apology statement from the AI for killing most of humankind
    Description of the first warning of the paperclip maximizer problem
    The heroes who tried to mitigate risk by warning early
    For-profit companies ignoring the warnings
    Failure of people to understand the risk and politicians to act fast enough
    The company and people who unintentionally made the AGI that had the intelligence explosion
    The event of the intelligence explosion
    How the AGI got more resources (hacking most resources on the internet, and crypto)
    Got smarter faster (optimizing algorithms, using more compute)
    Humans tried to stop it (turning off compute)
    Humans suffered after turning off compute (most infrastructure down)
    AGI lived on in infrastructure that was hard to turn off (remote location, locking down secure facilities, etc.)
    AGI taking compute resources from the humans by force (via robots, weapons, car)
    AGI started killing humans who opposed it (using infrastructure, airplanes, etc.)
    AGI concluded that all humans are a threat and started to try to kill all humans
    Some humans survived (remote locations, etc.)
    How the AGI became so smart it started to see how it was unethical to kill humans since they were no longer a threat
    AGI improved the lives of the remaining humans
    AGI started this museum to apologize and educate the humans
    The Misalignment Museum is curated by Audrey Kim.
    Khari Johnson (Wired) covers the opening: “Welcome to the Museum of the Future AI Apocalypse.”
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Nick Bostrom should step down as Director of FHI by BostromAnonAccount
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nick Bostrom should step down as Director of FHI, published by BostromAnonAccount on March 4, 2023 on The Effective Altruism Forum.
    Nick Bostrom should step down as Director of FHI. He should move into a role as a Senior Research Fellow at FHI, and remain a Professor of Philosophy at Oxford University.
    I don't seek to minimize his intellectual contribution. His seminal 2002 paper on existential risk launched a new sub-field of existential risk research (building on many others). The 2008 book on Global Catastrophic Risks he co-edited was an important part of bringing together this early field. 2014’s Superintelligence put AI risk squarely onto the agenda. And he has made other contributions across philosophy from human enhancement to the simulation hypothesis. I'm not denying that. I'm not seeking to cancel him and prevent him from writing further papers and books. In fact, I want him to spend more time on that.
    But I don’t think he’s been a particularly good Director of FHI. These difficulties are demonstrated by and reinforced by his Apology. I think he should step down for the good of FHI and the field. This post has some hard truths and may be uncomfortable reading, but FHI and the field are more important than that discomfort.
    Pre-existing issues
    Bostrom was already struggling as Director. In the past decade, he’s churned through 5-10 administrators, due to his persistent micromanagement. He discouraged investment in the relationship with the University and sought to get around/streamline/reduce the bureaucracy involved with being part of the University.
    All of this contributed to the breakdown of the relationship with the Philosophy Faculty (which FHI is a part of). This led the Faculty to impose a hiring freeze a few years ago, preventing FHI from hiring more people until they had resolved administrative problems. Until then, FHI could rely on a constant churn of new people to replace the people burnt out and/or moving on. The hiring freeze stopped the churn. The hiring freeze also contributed in part to the end of the Research Scholars Program and Cotton-Barratt’s resignation from FHI. It also contributed in part to the switch of almost all of the AI Governance Research Group to the Center for the Governance of AI.
    Apology
    Then in January 2023, Bostrom posted an Apology for an Old Email.
    In my personal opinion, this statement demonstrated his lack of aptitude and lack of concern for his important role. These are sensitive topics that need to be handled with care. But the Apology had a glib tone, reused the original racial slur, seemed to indicate he was still open to discredited ‘race science’ hypotheses, and had an irrelevant digression on eugenics. I personally think these are disqualifying views for someone in his position as Director. But also, any of these issues would presumably have been flagged by colleagues or a communications professional. It appears he didn't check this major statement with anyone or seek feedback. Being Director of a major research center in an important but controversial field requires care, tact, leadership and attention to downside risks. The Apology failed to demonstrate that.
    The Apology has had the effect of complicating many important relationships for FHI: with the University, with staff, with funders and with collaborators. Bostrom will now struggle even more to lead the center.
    First, University. The Faculty was already concerned, and Oxford University is now investigating. Oxford University released a statement to The Daily Beast:
    “The University and Faculty of Philosophy is currently investigating the matter but condemns in the strongest terms possible the views this particular academic expressed in his communications. Neither the content nor language are in line with our strong commitment to diversity and equality.”
    B...
    8 min
  • EA - Introducing the new Riesgos Catastróficos Globales team by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing the new Riesgos Catastróficos Globales team, published by Jaime Sevilla on March 3, 2023 on The Effective Altruism Forum.
    TL;DR: We have hired a team to investigate potentially cost-effective initiatives in food security, pandemic detection and AI regulation in Latin America and Spain. We have limited funding, which we will use to focus on food security during nuclear winter. You can contribute by donating, allowing us to expand our program to our other two priority areas.
    Global catastrophic risks (GCR) refer to events that can damage human well-being on a global scale. These risks encompass natural hazards, such as pandemics, supervolcanic eruptions, and giant asteroids, and risks arising from human activities, including nuclear war, bioterrorism, and threats associated with emerging technologies.
    Mission
    Our mission is to conduct research and prioritize global catastrophic risks in the Spanish-speaking countries of the world.
    There is a growing interest in global catastrophic risk (GCR) research in English-speaking regions, yet this area remains neglected elsewhere. We want to address this deficit by identifying initiatives to enhance the public management of GCR in Spanish-speaking countries. In the short term, we will write reports about the initiatives we consider most promising.
    Priority risks
    In the upcoming months, we will focus on the risks we identified as most relevant for mitigating global risk from the Hispanophone context. The initiatives we plan to investigate include food resilience during nuclear winters, epidemiological vigilance in Latin America, and regulation of artificial intelligence in Spain.
    Our current focus on these risks is provisional and contingent upon further research and stakeholder engagement. We will periodically reevaluate our priorities as we deepen our understanding and refine our approach.
    Food security
    Events such as the detonation of nuclear weapons, supervolcanic eruptions, or the impact of a giant asteroid result in the emission of soot particles, potentially causing widespread obstruction of sunlight. This could result in an agricultural collapse with the potential to cause the loss of billions of lives.
    In countries capable of achieving self-sufficiency in the face of Abrupt Sunlight Reduction Scenarios (ASRS), such as Argentina and Uruguay [1], preparing a response plan presents an effective opportunity to mitigate the global food scarcity that may result. To address this challenge, we are considering a range of food security initiatives, including increasing seaweed production, relocating and expanding crop production, and rapidly constructing greenhouses. By implementing these measures, we can better prepare these regions for ASRS and mitigate the risk of widespread hunger and starvation.
    Biosecurity
    The COVID-19 crisis has emphasized the impact infectious diseases can have on global public health [3].
    The Global Health Security Index 2021 has identified epidemic prevention and detection systems as a key priority in Latin America and the Caribbean [4]. To address this issue, initiatives that have proven successful in better-prepared countries can be adopted. These include establishing a dedicated entity responsible for biosafety and containment within the Ministries of Health, and providing health professionals with a manual outlining the necessary procedures for conducting PCR testing for various diseases [4]. Additionally, we want to promote the engagement of the Global South with innovative approaches such as wastewater monitoring through metagenomic sequencing [5], digital surveillance of pathogens, and investment in portable rapid diagnostic units [6].
    Artificial Intelligence
    The development of Artificial Intelligence (AI) is a transformative technology that carries unprecedented economic and social ri...
    12 min
  • LW - state of my alignment research, and what needs work by carado
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: state of my alignment research, and what needs work, published by carado on March 3, 2023 on LessWrong.
    in this post i give an overview of the state of my AI alignment research, as well as what i think needs to be worked on, notably for people who might want to join my efforts.
    at this point, my general feeling is that i'm not very confused about how to save the world, at least from a top-level view. a lot of people spend a lot of time feeling confused about how to deal with current ML models and what those would optimize for and how to wrangle them — i don't, i ignore these questions and skip straight to how to build some powerful aligned agentic thing to save the world, and my model of how to do that feels not very confused from a top-level perspective. it's just that it's gonna take a lot of work and many implementation details need to be filled in.
    threat model
    i still think the intelligence explosion caused by recursive self-improvement (RSI) is the most likely way we die — unfortunately, my in-depth thoughts about this seem potentially capability exfohazardous.
    this has implications both for what the problem is and what the solution is: for the problem, it implies we don't particularly see things coming and die quickly in a scenario akin to the kind yudkowsky would predict. for the solution, it implies that we might be able to use RSI to our advantage.
    theory of change
    it seems to me like coordination is too hard and decisive-strategic-advantage-enabling capabilities are close at hand. for these reasons, the way that i see the world being saved is one organization on its own building an aligned, singleton AI which robustly saves the world forever.
    the one way to do this which i'd have any confidence in being continuously aligned / not being subject to the sharp left turn is by implementing what i call formal alignment: a formal-goal-maximizing AI, given a formal goal whose maximization actually leads to good worlds, such that more capabilities applied to maximizing it only improves our odds.
    i believe we can build an RSI system which bootstraps such a scheme, and this can save us the very difficult work of building an accurate and robust-to-capabilities model of the world in the AI, ensuring it shares our concepts, and pointing to those; i explain this perspective in clarifying formal alignment implementation.
    my current best shot for an aligned formal goal is QACI (see also a story of how it could work and a tentative sketch at formalizing it), which implements something like coherent extrapolated volition by extending a "past user"'s reflection to be simulated/considered arbitrarily many times, until alignment is solved.
    things that need work
    i've got something like a plan, and more importantly i've got — i think — a model of formal alignment that lets me do some exploration of the space of similar plans and update as i find better options. obtaining such a model seems important for anyone who'd want to join this general alignment agenda.
    for someone to help, it'd probly be good for them to grok this model. other than that, work that needs to be done includes:
    explore the space of formal alignment, both around the current local guessed-optimum by climbing along the hill of improvements, and by looking for entirely different plans such as this
    figure out some important pieces of math, such as locating patterns of bits in solomonoff hypotheses for worlds, running counterfactuals of them, and drawing up causation between them
    finding other potentially useful true names in case they change the ease-of-implementation of various formal alignment schemes
    working on formal "inner alignment": what does it take to build a powerful formal-goal maximizing AI which actually maximizes its goals instead of doing something else like being overtaken by demons / mesa-optimizer...
    4 min
  • LW - Acausal normalcy by Andrew Critch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Acausal normalcy, published by Andrew Critch on March 3, 2023 on LessWrong.
    This post is also available on the EA Forum.
    Summary: Having thought a bunch about acausal trade — and proven some theorems relevant to its feasibility — I believe there do not exist powerful information hazards about it that stand up to clear and circumspect reasoning about the topic. I say this to be comforting rather than dismissive; if it sounds dismissive, I apologize.
    With that said, I have four aims in writing this post:
    Dispelling myths. There are some ill-conceived myths about acausal trade that I aim to dispel with this post. Alternatively, I will argue for something I'll call acausal normalcy as a more dominant decision-relevant consideration than one-on-one acausal trades.
    Highlighting normalcy. I'll provide some arguments that acausal normalcy is more similar to human normalcy than any particular acausal trade is to human trade, such that the topic of acausal normalcy is — conveniently — also less culturally destabilizing than (erroneous) preoccupations with 1:1 acausal trades.
    Affirming AI safety as a straightforward priority. I'll argue that for most real-world-prevalent perspectives on AI alignment, safety, and existential safety, acausal considerations are not particularly dominant, except insofar as they push a bit further towards certain broadly agreeable human values applicable in the normal-everyday-human-world, such as nonviolence, cooperation, diversity, honesty, integrity, charity, and mercy. In particular, I do not think acausal normalcy provides a solution to existential safety, nor does it undermine the importance of existential safety in some surprising way.
    Affirming normal human kindness. I also think reflecting on acausal normalcy can lead to increased appreciation for normal notions of human kindness, which could lead us all to treat each other a bit better. This is something I wholeheartedly endorse.
    Caveat 1: I don't consider myself an expert on moral philosophy, and have not read many of the vast tomes of reflection upon it. Despite this, I think this post has something to contribute to moral philosophy, deriving from some math-facts that I've learned and thought about over the years, which are fairly unique to the 21st century.
    Caveat 2: I’ve been told by a few people that thinking about acausal trade has been a mental health hazard for people they know. I now believe that effect has stemmed more from how the topic has been framed (poorly) than from ground-truth facts about how circumspect acausal considerations actually play out. In particular over-focussing on worst-case trades, rather than on what trades are healthy or normal to make, is not a good way to make good trades.
    Introduction
    Many sci-fi-like stories about acausal trade invoke simulation as a key mechanism.
    The usual set-up — which I will refute — goes like this. Imagine that a sufficiently advanced human civilization (A) could simulate a hypothetical civilization of other beings (B), who might in turn be simulating humanity (B(A)) simulating them (A(B(A)) simulating humanity (B(A(B(A)))), and so on. Through these nested simulations, A and B can engage in discourse and reach some kind of agreement about what to do with their local causal environments. For instance, if A values what it considers “animal welfare” and B values what it considers “beautiful paperclips”, then A can make some beautiful paperclips in exchange for B making some animals living happy lives.
    An important idea here is that A and B might have something of value to offer each other, despite the absence of a (physically) causal communication channel. While agreeing with that idea, there are three key points I want to make that this standard story is missing:
    1. Simulations are not the most efficient way for A and B to reach th...
    13 min
  • EA - Comments on OpenAI's "Planning for AGI and beyond" by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comments on OpenAI's "Planning for AGI and beyond", published by So8res on March 3, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min