The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Making EA more inclusive, representative, and impactful in Africa by Ashura Batungwanayo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Making EA more inclusive, representative, and impactful in Africa, published by Ashura Batungwanayo on August 19, 2023 on The Effective Altruism Forum.
    Authors: Ashura Batungwanayo (University of KwaZulu Natal) and Hayley Martin (University of Cape Town)
    DISCLAIMER: This paper draws from our collective experiences, perspectives, and conversations with community builders in Africa who share similar experiences and perspectives.
    Introduction
    In our engagement with Effective Altruism (EA), we noted a distinct emphasis on Existential Risks, notably concerning Artificial Intelligence (AI), alongside a focus on animal welfare and veganism - issues that carry nuanced significance, differing between Western societies and Africa due to diverse factors in animal husbandry. However, our initial allure lay in EA's focus on Global Health and Development (GH&D). GH&D holds a special significance for us as it directly confronts the realities we, as African students, encounter daily.
    Amidst this, the need for balance arises: acknowledging existential risks while prioritising urgent issues like poverty and education. Our vision extends to an EA Africa initiative that blends bottom-up and top-down approaches for effective, contextually attuned change. However, challenges persist, including navigating a competitive altruistic landscape and balancing immediate impact with long-term prevention. A critical thread is Africa's self-sufficiency, with EA acting as a catalyst for local partnership, co-designed interventions, and self-reliance. The path forward involves forging strategic collaborations, knowledge sharing, and empowerment, all underpinned by a commitment to inclusivity, representation, and comprehensive change.
    Global Health and Development's Urgent Call to Address African Realities
    The challenges it addresses are not abstract concepts, but tangible issues that our communities and loved ones have grappled with. As university students, we acknowledge the privilege bestowed upon us and feel a profound responsibility to address the issues that plague our homeland. Our identity is intertwined with the principles of Ubuntu, which emphasise our shared humanity and interconnectedness. This cultural ethos, coupled with the weight of the "black tax," the financial responsibilities we bear for our families and communities, amplifies our desire to contribute meaningfully to the well-being of our people. Incorporating the principles of GH&D into our personal cause area is more than a mere pursuit; it's a calling driven by the urgent need to translate our empathy into action.
    By understanding the nuances of diplomatic engagement and the complexities of GH&D, we can channel our aspirations into effective strategies that uplift our communities while respecting our cultural values.
    GH&D resonates deeply with our African identity, our educational privilege, and our unwavering commitment to making a positive impact in the places we call home. While AI Alignment and Animal Welfare remains a critical concern, we acknowledge the complexities in communicating its relevance to African audiences. The messaging around AI Safety and Animal Welfare doesn't inherently speak with the immediate and intersecting challenges we face as a continent, although we recognise its significance in the broader global context (and acknowledge that there are people working on AI Alignment and Animal Advocacy on the continent as well). It's important to note that concentrating solely on existential risks could inadvertently diminish the urgency of current issues, such as poverty and education. Striving for a comprehensive approach that appreciates the distinctive dynamics of African contexts is paramount.
    It's worth contemplating the establishment of an EA Africa initiative, one that aspires to harmonise both bottom-up and...
    9 min
  • LW - The U.S. is mildly destabilizing by lc
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The U.S. is mildly destabilizing, published by lc on August 18, 2023 on LessWrong.
    We focus so much on arguing over who is at fault in this country that I think sometimes we fail to notice or alert on what's actually happening. I would just like to point out, without attempting to assign blame, that American political institutions appear to be losing common knowledge of their legitimacy, and abandoning certain important traditions of cooperative governance. It would be slightly hyperbolic, but not unreasonable to me, to term what has happened "democratic backsliding".
    Let's imagine America of 2012 was measured 0.8 on the fictionally accurate "legitimate democracy index", and Hungary of 2012 was measured 0.5. My thesis is that the we'd now be at 0.75, and that the regression seems to have calcified despite the culture war calming down since 2020. Within the last three or four years we have seen:
    The first presidential election in the history of the country ever contested by one of the main candidates; an election now considered probably or definitely illegitimate by nearly a third of Americans.
    The world's largest protest-riot ever, when measured by estimated damage to property or number of participants.
    Spontaneous mob assaults of the capitol building.
    The leader of the opposition party being arrested on a mix of real and recently-invented process crimes in several different jurisdictions a year before his campaign.
    Recent, and novel, movements by Republicans to fine and censure Democratic congressmen millions of dollars outside of the criminal justice system.
    Serious attempts at dramatically expanding political control over the civil service and, if you can permit me to speak anecdotally, serious and successful attempts at unprecedented political loyalty testing of appointed silovik.
    You can disagree with how any one political faction is characterizing the above events, or how I'm characterizing the above events. One take, for example, would be that Donald Trump is a clown and that all of his indictments are perfectly legitimate and that they ultimately demonstrate the dispassionate fairness of our nation's prosecutors. But even if that's the case, perception is the leading indicator for democratic stability and a large amount of Republicans do not agree with that interpretation. Since Republicans now believe that the arrests are politically motivated, and that Democrats are hitting "defect" by electing those prosecutors, they are pressuring their politicians to escalate and calling them traitors when they refuse to do so. This is itself bad.
    It's of course possible to exaggerate the danger. I do not expect the entire political system of the United States is going to change anytime soon. But since 1989 I think it has been appropriate to have a degree of knightian uncertainty in predicting the eternal dominance of this or that regime, on the basis that modern technology and secret police make resistance impossible. If you currently habitually round probabilities of serious repression or further democratic backsliding in the West to zero, I suggest moving that up to 1%-per-decade and spending a little bit of time thinking about what you'd do if this continues for five more years and your estimate increases to 5 or 10 percent.
    Presidential Election Poll, January 2021, Umass
    Adam Schiff Defeats effort to fine and censure him 15 million dollars
    Schedule F Appointment
    Trump's inner circle is secretly making plans to fire thousands government employees if he wins in 2024, report says
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • LW - 6 non-obvious mental health issues specific to AI safety. by Igor Ivanov
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 6 non-obvious mental health issues specific to AI safety., published by Igor Ivanov on August 18, 2023 on LessWrong.
    Intro
    I am a psychotherapist, and I help people working on AI safety. I noticed patterns of mental health issues highly specific to this group. It's not just doomerism, there are way more that are less obvious.
    If you struggle with a mental health issue related to AI safety, feel free to leave a comment about it and about things that help you with it. You might also support others in the comments. Sometimes such support makes a lot of difference and people feel like they are not alone.All the examples in this post are anonymized and changed in a way that it's impossible to recognize a specific person behind them.
    AI safety is a rather unusual field
    The problems described in this post arise because AI safety is not an ordinary field to work in. Many people within the AI safety community believe that it might be the most important field of work, but the general public mostly doesn't care that much. Also, the field itself is extremely competitive and newcomers often have hard time getting a job.No one really knows when we will create AGI, and whether we will be able to keep it aligned. If we fail to align AGI, the humanity might extinct, and even if we succeed, it will radically transform the world.
    Patterns
    AGI will either cause doom or create a utopia. Everything else seem unimportant and meaningless.
    Alex is an ML engineer working in a startup that fights with aging. He believes that AGI will either destroy humanity or bring a utopia, and among other things it will stop aging, so Alex thinks that his job is meaningless, and quits it. He also sometimes asks himself "Should I invest? Should I exercise? Should I even floss my teeth? This all seems meaningless."
    No one knows how the post-AGI world will look like. All predictions are wild speculations, and it's very hard to tell whether any actions unrelated to AI safety are meaningful. This uncertainty can cause anxiety and depressionThese problems are an exacerbated version of existential problem of meaninglessness of life, and the way to mitigate them is to rediscover meaning in the world that ultimately doesn't have meaning.
    I don't know when we will create AGI and if we will be able to align it, so I feel like I have no control over it.
    Bella is an anxious person, and she recently got interested in AI safety and she realized that nobody know for sure how to align AGI.She feels that AGI might pose an extreme danger, and there is nothing she can do. She even can't understand how much time do we have. A year? Five years? This uncertainty makes here even more anxious. And what if the takeoff will be so rapid that no one will understand what is going on?Bella is meeting a psychotherapist, but they treat her fear as something irrational. This doesn't help, and only makes Bella more anxious. She feels like even her therapist doesn't understand her.
    AI safety is a big part of my life, but others don't care that much about it. I feel alienated.
    Chang is an ML scientist working on mechanistic interpretability in AI lab. AI safety consumed all his life and became a part of his identity. He constantly checks AI safety influencers on Twitter, he spends a lot of time reading LessWrong and watching AI podcasts. He even made a tatoo of a paperclip.Chang lives outside of major AI safety hubs, and he feels a bit lonely because there is no one to talk about AI safety in person.Recently he attended his aunt's birthday party. He talked about alignment with his family. They were a bit curious about the topic, but didn't care that much. Chang feels like they just don't get it.
    Working on AI safety is so important that I neglected other parts of my life and burned-out.
    Dmitry is an undergrad student. He believes that AI safet...
    6 min
  • EA - Why we should fear any bioengineered fungus and give fungi research attention by emmannaemeka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we should fear any bioengineered fungus and give fungi research attention, published by emmannaemeka on August 18, 2023 on The Effective Altruism Forum.
    Someone asked me a question after my talk on climate change and emerging fungal pathogens. The question was, why should we give attention to fungi when we know the risk of bioengineering it can be low. Hence the weight given to viruses and bacteria. I thought through and wondered that it is interesting and the EA community should understand why this should be considered. I think in my opinion we need to take fungi very seriously for so many reasons:
    What we know that makes fungi pathogen interesting:
    There are only three known anitifungal drugs against major fungi pathogens which are not very effective. The available antifungals have narrow spectrus and high toxicity, and because of the plasticity of the fungi genome resistance is developed very easily. The seriousness of this issue was brought to fore during the COVID-19 pandemic, a number of secondary infections due to fungi pathogens were reported . The emergence of Azole resistance Aspergillus in Europe and environment is a concern as the mortality due to antibiotics resistance and limited drugs can get up to 100%.
    There are no vaccines for fungi. Currently, there are no immunotherapy or any vaccine available for any fungal infection. More research and funding is needed. Here is a nice paper that shows how close we are to finding a vaccine against any fungi pathogen.
    Candida auris the first fungi to have emerged as a result of climate change is a fungal pathogen. This is really interesting because this yeast behaves like bacteria and is naturally resistant to some antifungals. It is difficult to treat and led to the shutting down of hospitals . A Detroit hospital will stop taking patients temporarily as it tries to contain an outbreak of a rare, but potentially deadly and drug-resistant fungus. Find link to this new here. It was first discovered in 2009 and have now been reported in all the continents of the world. The CDC gives reasons why this pathogen is a problem
    Why is Candida auris a problem? source
    It causes serious infections. C. auris can cause bloodstream infections and even death, particularly in hospital and nursing home patients with serious medical problems. More than 1 in 3 patients with invasive C. auris infection (for example, an infection that affects the blood, heart, or brain) die.
    It's often resistant to medicines. Antifungal medicines commonly used to treat Candida infections often don't work for Candida auris. Some C. auris infections have been resistant to all three types of antifungal medicines.
    It's becoming more common. Although C. auris was just discovered in 2009, it has spread quickly and caused infections in more than a dozen countries.
    It's difficult to identify. C. auris can be misidentified as other types of fungi unless specialized laboratory technology is used. This misidentification might lead to a patient getting the wrong treatment.
    It can spread in hospitals and nursing homes. C. auris has caused outbreaks in healthcare facilities and can spread through contact with affected patients and contaminated surfaces or equipment. Good hand hygiene and cleaning in healthcare facilities is important because C. auris can live on surfaces for several weeks
    4. Fungi are the only species that have caused the complete extinction of a species. A newspaper reported thus "A deadly fungus that has driven more species to extinction than any other pathogen has spread across Africa unnoticed. Chytrid fungus, Batrachochytrium dendrobatidis, or Bd for short, is a highly infectious fungus that affects frogs, toads, salamanders and other amphibians(Source). Although various diseases, such as white-nose syndrome resulting from the European fungu...
    6 min
  • LW - Announcing Foresight Institute's AI Safety Grants Program by Allison Duettmann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Foresight Institute's AI Safety Grants Program, published by Allison Duettmann on August 18, 2023 on LessWrong.
    Foresight Institute has received funding to support projects in three AI safety areas we think are potentially under-explored:1. Neurotechnology, brain computer interface, whole brain emulation, and "lo-fi" uploading approaches to produce human-aligned software intelligence2. Computer security, cryptography, and related techniques to help secure AI systems3. Multi agent simulations, game theory and related techniques to create safe multipolar AI scenarios that avoid collusion and foster positive sum dynamicsThe grant application process is now open and accepts rolling applications. We expect to grant between $1 - 1.2 million per year across projects and look forward to receiving your project proposals that could make a significant difference for AI safety within short timelines. Please visit/ for full details and application instructions.
    I would appreciate it if you would consider sharing the opportunity with others who may benefit from applying.
    Feel free to comment here or email me at [email protected] with any questions, feedback, or ideas for collaborations.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • EA - How expensive is leaving your org? Squiggle Model by jessica mccurdy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How expensive is leaving your org? Squiggle Model, published by jessica mccurdy on August 18, 2023 on The Effective Altruism Forum.
    Disclaimer: I (Jessica) opted to post this without extra nuance as opposed to not sharing. Hopefully, it can still help spark some discussion and consideration but there are many additional considerations here.
    Ben West made this rough Squiggle model you can use to calculate an organization's costs of replacing an employee in months of productivity and rough monetary value.
    The model is pretty simple: total_cost = hiring_manager_time + hiring_support_time + time_without_employee + skill_up_time
    I wanted this to be shared because when Ben mentioned some of the extra costs of an employee leaving an org, I was surprised by how high it was. I think others might be underweighting this cost as well which could influence some career decisions.
    One highlighted point from the model I wasn't seriously considering:
    A new employee will generally take some amount of time to become as efficient as
    the previous employee.
    This comes from several sources:
    New people will often simply lack relevant skills/knowledge
    The coworkers of new staff will take a while to understand how to efficiently work with the new person, build up trust with them, understand their strengths and weaknesses, etc.
    New people will also almost certainly lack role-specific knowledge, e.g. "to send the newsletter you have to use this website and then get approval from this person except in these scenarios you actually get approval from different person" etc.
    Hope this is useful!
    To be clear, I definitely think there are times when people can have a lot more impact elsewhere and there are many biases that act in favor of staying in a job for too long. However, I feel like people in EA hop around particularly fast and sometimes face some social incentives to leave their current positions.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • EA - Personal Reflections on Longtermism by NatKiilu
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Personal Reflections on Longtermism, published by NatKiilu on August 18, 2023 on The Effective Altruism Forum.
    Disclaimer: This post is entirely based on my impressions of longtermism as informed by my interaction with the literature, other longtermists, and questions from curious non-longtermists. This post is not a critique of the work being done by longtermists currently but rather a push to widen the category of interventions being pursued under the longtermist umbrella. It may be that I am missing some key facts in which case I would greatly appreciate responses challenging my assumptions, claims, and conclusions. Finally, all views expressed here are solely mine and do not reflect any organizations I am affiliated with directly or indirectly. I take full responsibility for any harmful premises presented.
    [Summary]
    This post is a personal reflection on longtermism from my perspective as an African woman living and working in the continent. In the post, I make the claim that it seems as though pursuing the 'flourishing of future generations' such that all future beings lead dignified and worthwhile lives points to the need for us to pursue more interventions that feed into systemic change. I also make this point as a contribution to the of pursuit of good value lock-in. I contend that the values we currently hold substantially lower the quality of life for some groups of people who share traits which have historically been subjected to oppression and marginalization and that failing to work on these issues falls short of the claim that (some) longtermists seek not only the endurance of humanity but its flourishing too.
    [Earlier impressions]
    My initial response as a newcomer to EA was to reject longtermism. At the very least, I believed - incorrectly - that the widely accepted sustainability principle already covered its objectives by ensuring that the earth remains habitable for future generations. I also found strong longtermism - that in any set of decision situations, the best decision would be the one with the best consequences for the far-future conclusion - to be quite counterintuitive, especially given my surroundings. Finally, it was also difficult to fully comprehend the sheer numbers cited (scope insensitivity), difficult to believe that we could actually know what would count as protecting future generations (cluelessness), and even then whether these actions would actually lead to their intended consequences given that their effects are to be felt far off into the future (e.g., washing out and rapid diminution).
    The case for longtermism rests on the following premises: (a) that future people matter, i.e., they have equal moral worth as present people; (b) that by many estimates, they will by far outnumber us; (c) that there are ways in which we can make sure the longterm future goes well; and (d) (according to strong longtermists) that doing so is the greatest moral imperative of our lifetime given what is at stake. My understanding of the work and literature around longtermism seems to indicate that longtermism, broadly speaking, is concerned with two objectives: (a) ensuring that the longterm future exists and humanity's potential is not destroyed or significantly impaired (work here is heavily focused on preventing existential catastrophes) as well as, (b) ensuring that future beings lead flourishing lives (concerns here revolve around the quality of lives and wellbeing of sentient beings existing then, e.g., good value lock in and preventing suffering risks).
    I gradually came to embrace weak longtermism as a result of frequently exposing myself to discussions on the idea, a commitment to embracing the scout mindset, and becoming aware of some of the cognitive biases at work. Hence, I consider ensuring that the longterm future goes well a priority ...
    14 min
  • LW - What does it mean to "trust science"? by jasoncrawford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does it mean to "trust science"?, published by jasoncrawford on August 18, 2023 on LessWrong.
    And this, my children, is why we do not say things like "I believe in science". I mean, don't get me wrong, science definitely exists - I've seen it. But not everything that calls itself science is science, and even good science sometimes gets wrong results. -Megan McArdle
    Should we "trust science" or "believe in science"?
    I think this is a fuzzy idea that we would do well to make clear and precise. What does it mean to "trust science?"
    Does it mean "trust scientists"? Which scientists? They disagree, often vehemently. Which statements of theirs? Surely not all of them; scientists do not speak ex cathedra for "Science."
    Does it mean "trust scientific institutions"? Again, which ones?
    Does it mean "trust scientific papers"? Any one paper can be wrong in its conclusions or even its methods. The study itself could have been mistaken, or the writeup might not reflect the study.
    And it certainly can't mean "trust science news," which is notoriously inaccurate.
    More charitably, it could mean "trust the scientific process," if that is properly understood to mean not some rigid Scientific Method but a rational process of observation, measurement, evidence, logic, debate, and iterative revision of concepts and theories. Even in that case, though, what we should trust is not the particular output of the scientific process at any given time. It can make wrong turns. Instead, we should trust that it will find the truth eventually, and that it is our best and only method for doing so.
    The motto of science is not "trust us." (!) The true motto of science is the opposite. It is that of the Royal Society: nullius in verba, or roughly: "take no one's word."
    There is no capital-S Science - a new authority to substitute for God or King. There is only science, which is nothing more or less than the human faculty of reason exercised deliberately, systematically, methodically, meticulously to discover general knowledge about the world.
    So when someone laments a lack of "trust" in science today, what do they mean? Do they mean placing religion over science, faith over reason? Do they mean the growing distrust of elites and institutions, a sort of folksy populism that dismisses education and expertise in general? Or do they mean "you have to follow my favored politician / political program, because Science"? (That's the one to watch out for. Physics, chemistry and biology can point out problems, but we need history, economics and philosophy to solve them.)
    Anyway, here's to science - the system that asks you not to trust, but to think.
    Adapted from a 2019 Twitter thread.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - Corporate campaigns work: a key learning for AI Safety by Jamie Harris
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Corporate campaigns work: a key learning for AI Safety, published by Jamie Harris on August 18, 2023 on The Effective Altruism Forum.
    Negotiations and pressure campaigns have proven effective at driving corporate change across industries and movements. I expect that AI safety/governance can learn from this!
    The basic idea:
    The runaway success of effective animal advocacy has been sweeping corporate reform
    Similar tactics have been successful across social movements
    GovAI to ??? to PauseAI: Corporate campaigns need an ecosystem of roles and tactics
    Possible next steps
    Pragmatic research: ask prioritisation, user interviews, and message testing
    Start learning by doing
    Work extensively with volunteers (and treat them like staff members)
    Moral trade: longtermist money for experienced campaigner secondments
    The runaway success of effective animal advocacy has been sweeping corporate reform
    Animal advocates, funded especially by Open Philanthropy and other EA sources, have achieved startling success in driving corporate change over the past ~decade. As Lewis Bollard, Senior Program Officer at Open Philanthropy, writes:
    A decade ago, most of the world's largest food corporations lacked even a basic farm animal welfare policy. Today, they almost all have one. That's thanks to advocates, who won about 3,000 new corporate policies in the last ten years.
    In 2015-18, as advocates secured cage-free pledges from almost all of the largest American and European retailers, fast food chains, and foodservice companies. Advocates then extended this work globally, securing major pledges from Brazil to Thailand. Most recently, advocates won the first global cage-free pledges from 150 multinationals, including the world's largest hotel chains and food manufacturers.
    A major question was whether these companies would follow through on their pledges. So far, almost 1,000 companies have - that's 88% of the companies that promised to go cage-free by the end of last year. Another 75% of the world's largest food companies are now publicly reporting on their progress in going cage-free.
    Some advocates establish professional relationships with companies and encourage them to introduce improvements. Others use petitions, protests, and PR pressure to push companies over the line.
    Almost everyone who investigates these campaigns thoroughly seems to conclude that they're exceptionally cost-effective at making real improvements for animals, at least in the short term. There are both ethical and strategic reasons why some animal advocates doubt that these kinds of incremental welfare tactics are a good idea, but I lean towards thinking that the indirect effects are neutral to positive, while the direct effects are robustly good. There are other promising tactics that animal advocates can use, but the track record and evidence base for corporate welfare campaigns is unusually strong.
    Of course, animal advocacy is different to AI Safety. But something that has been so successful in one context seems worth exploring seriously in others. And oh wait, it has worked in more than one context already.
    Similar tactics have been successful across social movements
    In my research into other social movements' histories, I found strong evidence that pressure tactics can be effective at changing companies' behaviour or disrupting their processes:
    US anti-abortion activists seem to have successfully disrupted the supply of abortion services and may have reduced abortion rates.
    Anti-death penalty activists successfully disrupted the supply of lethal injection drugs.
    Pressure campaigns likely accelerated Starbucks and other chains' participation in Fair Trade certification schemes.
    Prison riots and strikes seem to have encouraged the creation of new procedures and rules for prisoners.
    There are lots of caveats, concerns,...
    14 min
  • AF - An Overview of Catastrophic AI Risks: Summary by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Overview of Catastrophic AI Risks: Summary, published by Dan H on August 18, 2023 on The AI Alignment Forum.
    We've recently published on our website a summary of our paper on catastrophic risks from AI, which we are cross-posting here. We hope that this summary helps to make our research more accessible and to share our policy recommendations in a more convenient format. (Previously we had a smaller summary as part of this post, which we found to be insufficient. As such, we have written this post and have removed that section to avoid being duplicative.)
    Executive summary
    Catastrophic AI risks can be grouped under four key categories which we explore below, and in greater depth in CAIS' linked paper:
    Malicious use: People could intentionally harness powerful AIs to cause widespread harm. AI could be used to engineer new pandemics or for propaganda, censorship, and surveillance, or released to autonomously pursue harmful goals. To reduce these risks, we suggest improving biosecurity, restricting access to dangerous AI models, and holding AI developers liable for harms.
    AI race: Competition could push nations and corporations to rush AI development, relinquishing control to these systems. Conflicts could spiral out of control with autonomous weapons and AI-enabled cyberwarfare. Corporations will face incentives to automate human labor, potentially leading to mass unemployment and dependence on AI systems. As AI systems proliferate, evolutionary dynamics suggest they will become harder to control. We recommend safety regulations, international coordination, and public control of general-purpose AIs.
    Organizational risks: There are risks that organizations developing advanced AI cause catastrophic accidents, particularly if they prioritize profits over safety. AIs could be accidentally leaked to the public or stolen by malicious actors, and organizations could fail to properly invest in safety research. We suggest fostering a safety-oriented organizational culture and implementing rigorous audits, multi-layered risk defenses, and state-of-the-art information security.
    Rogue AIs: We risk losing control over AIs as they become more capable. AIs could optimize flawed objectives, drift from their original goals, become power-seeking, resist shutdown, and engage in deception. We suggest that AIs should not be deployed in high-risk settings, such as by autonomously pursuing open-ended goals or overseeing critical infrastructure, unless proven safe. We also recommend advancing AI safety research in areas such as adversarial robustness, model honesty, transparency, and removing undesired capabilities.
    1. Introduction
    Today's technological era would astonish past generations. Human history shows a pattern of accelerating development: it took hundreds of thousands of years from the advent of Homo sapiens to the agricultural revolution, then millennia to the industrial revolution. Now, just centuries later, we're in the dawn of the AI revolution. The march of history is not constant - it is rapidly accelerating. World production has grown rapidly over the course of human history. AI could further this trend, catapulting humanity into a new period of unprecedented change.
    The double-edged sword of technological advancement is illustrated by the advent of nuclear weapons. We narrowly avoided nuclear war more than a dozen times, and on several occasions, it was one individual's intervention that prevented war. In 1962, a Soviet submarine near Cuba was attacked by US depth charges. The captain, believing war had broken out, wanted to respond with a nuclear torpedo - but commander Vasily Arkhipov vetoed the decision, saving the world from disaster. The rapid and unpredictable progression of AI capabilities suggests that they may soon rival the immense power of nuclear weapons. With ...
    26 min