Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Healthy Futures Global: Join Us in Tackling Syphilis in Pregnancy and Creating Healthier Futures, published by Nils on May 18, 2023 on The Effective Altruism Forum.

TLDR

Healthy Futures Global is a new global health charity originating from CE’s incubation program trying to prevent mother-to-child transmission of syphilis, founded by Keyur Doolabh (medical doctor with research experience) and Nils Voelker (M Sc in health economics and former strategy consultant)

Healthy Futures’ strategy is to elevate syphilis screening rates in antenatal clinics to the high levels of HIV screening rates by replacing HIV-only tests with a dual HIV/syphilis test

Keyur and Nils are currently exploring potential pilot countries, and will be in the Philippines and Tanzania soon - they invite you to subscribe to their newsletter and to reach out to volunteer, especially if you are in the Philippines or Tanzania or could connect us to people there

I. Introduction: Healthy Futures Global and its Origins

Keyur and Nils are excited to announce the launch of Healthy Futures Global, a new organisation originating from Charity Entrepreneurship's latest incubation programme, dedicated to making a positive impact on global health.

Healthy Futures’ mission is to improve maternal and newborn health by focusing on the elimination of congenital syphilis, a preventable but devastating disease that affects millions of families worldwide.

II. The Problem: Congenital Syphilis' Global Impact

Syphilis in pregnancy is a pressing global health issue. It causes approximately 60,000 newborn deaths and 140,000 (almost 10% of global) stillbirths annually, contributing up to 50% of all stillbirths in some regions (1, 2, 3, 4).

Antenatal syphilis also causes lifelong disabilities for many surviving children, often going unaddressed in many countries. This disability can include cognitive impairment, vision and hearing deficits, bone deformity, and liver dysfunction. If a pregnant woman has syphilis, her child has a 12% chance of neonatal death, 16% chance of stillbirth, and 25% chance of disability (5).

III. The Solution: Test and Treat Strategy

The theory of change (below) is a hybrid between direct intervention, technical assistance and policy work. It involves lobbying governments for policy support, and supporting governments, local NGOs and antenatal clinics to roll out dual HIV/Syphilis tests.

The key components of the approach involve rapid testing (RDTs) during antenatal care and immediate treatment with antibiotics (BPG) for positive cases.

The main strengths of Healthy Futures are:

Cost effectiveness: The strategy has the potential to cost-effectively save lives, prevent disabilities, and reduce the burden on health systems. Our analysis gives us an expected value of $2,400 per life saved and ~10x the cost-effectiveness of direct cash transfers. The medical evidence for positive effects of treating the pregnant woman and her baby is strong (6).

Monitoring and evaluation: The direct nature of this intervention offers quick feedback loops, allowing us to re-evaluate our strategy accordingly.

Track record: Keyur is a medical doctor and Nils brings a background of consulting for pharmaceutical companies and global health organisations. Their backgrounds are a good fit for this type of intervention.

Organisational fuel: Healthy Futures benefits from ongoing mentoring from Charity Entrepreneurship and previously incubated charities.

IV. Challenges, Risks and Mitigation

The challenges Healthy Futures plans to address are (ranked according to severity x likelihood of occurring - highest on top):

Implementational challenges:

Sustainability: The long-term success and cost-effectiveness of this intervention depend on governments making the necessary changes in the health care system and establishing mech...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: U.S. Regulatory Updates to Benefit-Cost Analysis: Highlights and Encouragement to Submit Public Comments, published by DannyBressler on May 18, 2023 on The Effective Altruism Forum.

On April 6, 2023, the U.S. Office of Management and Budget released a draft of the first update to the Federal benefit-cost analysis (BCA) guidelines in 20 years. I saw a nice article in Vox Future Perfect and a nice EA forum post that covered this. These posts cover some of the key points, but I think there are other important updates that might be overlooked. I will highlight some of those below.

The key new documents stipulating the new draft BCA guidelines are:

Circular A-4

The Circular A-4 Preamble

Circular A-94

Why is the update important? Since 1993, U.S. agencies have been required to conduct a regulatory analysis of all significant regulatory actions (the definition of which was just revised from a rule with an annual impact of more than $100 million to $200 million), which includes an assessment of the costs and benefits of the action. Essentially, all major regulatory actions in the U.S. are subject to BCA, guided by Circular A-4.

However, these updated guidance documents are still drafts. They are subject to public comment and peer review and may be changed significantly in light of the feedback received in this process.

If you think that some of these highlights or other parts of the new A-4 and A-94 are a good idea (or if you don’t) I’d highly recommend submitting a public comment via the Regulations.gov system (A-4 link and A-94 link). The deadline for public comments is June 6th.

Positive comments that support the approach taken in the document are equally and often more useful/impactful than critical comments. If everyone who dislikes something criticizes it, and everyone who supports something doesn’t bother mentioning their support, it looks like everyone who had an opinion opposed it! So, if you like the approach taken (or don’t), please write a comment! Also, note that comments supported with the analytical reasons why the approach is (or is not) justified are generally more useful and taken more seriously.

Now on to the highlights:

Short-Term Discount Rate

As the Vox article mentioned, the new update to Circular A-4 significantly lowers the discount rate to a 1.7% near-term discount rate. This of course is a large change from the previous 3% rate, but this comes from just using a similar method to the previous 2003 A-4 method with more recent Treasury yield data. The preamble has a deeper dive into this calculation and asks the public a number of questions about whether there is a better approach, for those who are interested.

The draft Circular A-4 continues to take a “descriptive” approach to discounting in which market data is used to determine the observed tradeoffs people make between money now and money in the future. The discount rate is now lower simply because yields have been steadily declining for the last 20 years.

There are good reasons to believe that rates will continue to be low, but it’s also important to emphasize that if rates are not low in the future, then this near-term discount rate will go up again. This is why from the perspective of placing more weight on the future, the next bullets may be more important.

Long-Term Declining Discount Rate

A related important change (and more robust to future interest rate fluctuations) is that A-4 and A-94 endorse the general concept of declining discount rates, and the A-4 preamble proposed and asked for comment on a specific declining discount rate schedule, which discounts the future at progressively lower rates to account for future interest rate uncertainty. This is in line with the approach recommended in the literature based on the best available economics, and also ends up placing larger weight on t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some background for reasoning about dual-use alignment research, published by Charlie Steiner on May 18, 2023 on The AI Alignment Forum.

This is pretty basic. But I still made a bunch of mistakes when writing this, so maybe it's worth writing. This is background to a specific case I'll put in the next post.

It's like a a tech tree

If we're looking at the big picture, then whether some piece of research is net positive or net negative isn't an inherent property of that research; it depends on how that research is situated in the research ecosystem that will eventually develop superintelligent AI.

Consider this toy game in the picture. We start at the left and can unlock technologies, with unlocks going faster the stronger our connections to prerequisites. The red and yellow technologies in the picture are superintelligent AI - pretend that as soon as one of those technologies is unlocked, the hastiest fraction of AI researchers are immediately going to start building it. Your goal is for humanity to unlock yellow technology before a red one.

This game would be trivial if everyone agreed with you. But there are many people doing research, and they have all kinds of motivations - some want as many nodes to be unlocked as possible (pure research - blue), some want to personally unlock a green node (profit - green), some want to unlock the nearest red or yellow node no matter which it is (blind haste - red), and some want the same thing as you (beneficial AI - yellow) but you have a hard time coordinating with them.

In this baseline tech tree game, it's pretty easy to play well. If you're strong, just take the shortest path to a yellow node that doesn't pass too close to any red nodes. If you're weak, identify where the dominant paradigm is likely to end up, and do research that differentially advantages yellow nodes in that future.

The tech tree is wrinkly

But of course there are lots of wrinkles not in the basic tech tree, which can be worth bearing in mind when strategizing about research.

Actions in the social and political arenas. You might be motivated to change your research priorities based on how it could change peoples' minds about AI safety, or how it could affect government regulation.

Publishing and commercialization. If a player publishes, they get more money and prestige, which boosts their ability to do future research. Other people can build on published research. Not publishing is mainly useful to you if you're already in a position of strength, and don't want to give competitors the chance to outrace you to a nearby red node (and of course profit-motivated players will avoid publishing things that might help competitors beat them to a green node).

Uncertainty. We lack exact knowledge of the tech tree, which makes it harder to plan long chains of research in advance. Uncertainty about the tech tree forces us to develop local heuristics - ways to decide what to do based on information close at hand. Uncertainty adds a different reason you might not publish a technology: if you thought it was going to be a good idea to research when you started, but then you learned new things about the tech tree and changed your mind.

Inhomogeneities between actors and between technologies. Different organizations are better at researching different technologies - MIRI is not just a small OpenAI.

Ultimately, which technologies are the right ones to research depends on your model of the world / how you expect the future to go. Drawing actual tech trees can be a productive exercise for strategy-building, but you might also find it less useful than other ways of strategizing.

We're usually mashing together definitions

I'd like to win the tech tree game. Let's define a "good" technology as one that would improve our chances of winning if it was unlocked for free, given the st...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Unexpected Clanging, published by Chris Leong on May 18, 2023 on The AI Alignment Forum.

There are two boxes in front of you. In one of them, there is a little monkey with a cymbal, whilst the other box is empty. In precisely one hour the monkey will clang its cymbal.

While you wait, you produce an estimate of the probability of the monkey being in the first box. Let's assume that you form your last estimate, p, three seconds before the monkey clangs its cymbal. You can see the countdown and you know that it's your final estimate, partly because you're slow at arithmetic.

Let Omega be an AI that can perfectly simulate your entire deliberation process. Before you entered the room, Omega predicted what your last probability estimate would be and decided to place the monkey in a box such as to mess with you. Let q be the probability of Omega placing the monkey in the first box. In particular, Omega, sets q=p/2, unless p=0 or you haven't formed a probability estimate, in which case q=1.

What probability should you expect that the monkey is in the first box?

I think it's fairly clear that this is a no-win situation. No matter what the final probability estimate you form before clanging, as soon as you've locked it in, you know that it is incorrect, even if you haven't heard the clanging yet. You can try to escape this, but there's no reason that the universe has to play nice.

This problem can be seen as a variation on Death in Damascus. I designed this problem to reveal that the core challenge Death in Damascus poses isn't just that another process in the world can depend upon your decision, but that it can depend upon your expectations even if you don't actually make a decision based upon those expectations.

I also find this problem as a useful intuition pump as I think it's clearer that it's a no-win situation than in other similar problems. In Newcomb's problem, it's easy to get caught up thinking about the Principle of Dominance. In Death in Damascus, you can confuse yourself trying to figure out whether CDT recommends staying or fleeing. At least to me, in this problem it is clearer it is a dead end and that there's no way to beat Omega.

This is also a useful intuition pump for the Evil Genie Puzzle. When I first discovered this puzzle, I felt immensely confused that no matter which decision that you made you would immediately regret it. However, the complexity of the puzzle made it complicated for me to figure out exactly what to make of it, so when trying to solve it I came up with this problem as something easier to grok. I guess my position after considering the Unexpected Clanging is that you just have to accept that a sufficiently powerful agent may be able to mess with you like this and that you just have to deal with it. (I'll leave a more complete analysis to a future post).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Presenting: 2023 Incubated Charities (Round 1) - Charity Entrepreneurship, published by CE on May 18, 2023 on The Effective Altruism Forum.

We are thrilled to announce the launch of four new nonprofit organizations through our February/March 2023 Incubation Program.

Executive summary

The 2023 Round 1 incubated charities are:

Animal Policy International - Ensuring animal welfare standards are upheld in international trade policy

Ansh - Empowering mothers to save newborn lives by building healthcare capacity for adoption of Kangaroo Care

Healthy Futures Global - Preventing mother-to-child transmission of syphilis through testing and treatment

HealthLearn - Providing the world’s best online training to health workers in developing countries

Two more organizations got started during the program, but are not officially incubated by CE. We believe that the interventions, chosen by the solo founders, are promising (one was recommended by us as a top idea). We have provided support to both organizations through mentorship and benefits similar to those offered to our incubated projects.

These organizations are:

The Mission Motor - Building a more evidence-driven animal cause area by training and supporting organizations to use monitoring and evaluation to improve the impact of their interventions

Upstream Policies - Driving responsible fishing practices by championing bait fish prohibition

Context: The Charity Entrepreneurship Incubation Program February/March 2023

The February/March 2023 program focused on global health and animal advocacy. Our generous donors from the CE Seed Network have enabled us to provide these initiatives with $590,000 in grant funding to kickstart their interventions.

In addition to our seed grants, we are dedicated to providing our founders with comprehensive support. This includes continuous mentorship, operational assistance, free co-working spaces at our London office, and access to an ever-growing network of founders, donors, and highly specialized mentors. We have offered several tailored safety nets for those who have decided not to found a charity this program, such as career mentorship, connections to job opportunities, a two-month stipend, or another chance to found a charity through one of our upcoming programs. Our aim is to ensure that all program participants pursue high-impact careers, regardless of whether they found a charity in the given round.

We are also pleased to share with you a recently-published video, which showcases program participants sharing their insights on the challenges and benefits of the program. They discuss their motivations for applying, as well as what they found most useful and enjoyable. The footage was filmed during an in-person week held at our London office, and we believe it provides valuable insights into what makes our program unique. We hope you take a moment to watch it.

Feel free to learn more about the program here. The next application phase will start on July 10, and close on September 30, 2023. You'll have the opportunity to apply for both the February/March 2024 and July/August 2024 Incubation Programs. For the February/March 2024 program, our focus will be on: mass-media interventions, and preventative animal advocacy. To receive notifications when we start accepting applications, sign up here.

Our new charities in detail

ANIMAL POLICY INTERNATIONAL

Co-founders and Co-Executive Directors: Mandy Carter, Rainer KravetsWebsite: animalpolicyinternational.orgEmail address: [email protected] Incubation Grant: $110,000

Description of the intervention

Animal Policy International is working with policymakers in regions with higher levels of animal welfare legislation to advocate for responsible imports that are in line with domestic laws. By applying equal standards, they aim ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Collective Identity, published by Nicholas Kees Dupuis on May 18, 2023 on The AI Alignment Forum.

Thanks to Simon Celinder, Quentin Feuillade--Montixi, Nora Ammann, Clem von Stengel, Guillaume Corlouer, Brady Pelkey and Mikhail Seleznyov for feedback on drafts. This post was written in connection with the AI Safety Camp.

Executive Summary:

This document proposes an approach to corrigibility that focuses on training generative models to function as extensions to human agency. These models would be designed to lack independent values/preferences of their own, because they would not have an individual identity; rather they would identify as part of a unified system composed of both human and AI components.

The selfless soldier: This section motivates the difference between two kinds of group centric behavior, altruism (which is based in individual identity) and collective identity.

Modeling groups vs individuals: Here we argue that individuals are not always the most task-appropriate abstraction, and that it often makes sense to model humans on the group level.

Generative predictive models: This section describes how generative predictive models will model themselves and their environment, and motivates the importance of the “model of self” and its connection to personal identity.

Strange identities: There are several ways (in humans) in which the one-to-one correspondence between a neural network and its model of self breaks down, and this section discusses three of those examples in order to suggest that identity is flexible enough that an AI’s identity need not be individual or individuated.

Steps toward identity fusion: Here we aim to clarify the goal of this agenda and what it would mean for an AI to have an identity based on a human-AI system such that the AI component extends the human’s agency. While we don’t give a clear plan for how to bring about this fusion, we do offer an antithetical example of what kind of training would clearly fail.

Relevance for corrigibility: This section concludes the document by drawing more direct connections to corrigibility, and by offering a series of open questions for how this research might be made more concrete.

The selfless soldier

In the heat of battle a grenade is tossed into the middle of a troop of soldiers. One soldier throws themself on top of the grenade, sacrificing themself for the survival of the troop. There are two main ways to frame what just happened.

Altruism (individual identity): The soldier has the personal value/preference of protecting their troop from harm. Reasoning (quickly) from this value, the soldier deduces that they must sacrifice themselves in order to bring about the future where their fellow soldiers are safe.

Collective Identity: The individual soldier is not the most important abstraction to explain/predict this situation, rather it is the troop as a whole. The troop cares about its own survival, and this manifests itself in the decision to sacrifice one of its members to protect itself from further harm (even though the cognition, at least at the moment of decision, happens entirely within one brain). While the individual soldier could theoretically use clever arguments to escape this conclusion, they do not (because as a component of the troop, this is not their function).

The problem of alignment is often framed as trying to ensure that the values of an AI system are aligned with humanity, ideally imbuing them with a certain kind of perfect altruism toward humankind. The problem of corrigibility is often framed as ensuring that even when those values are not (yet) perfectly aligned with our own, an adversarial relationship does not develop between the AI and its human designers (such that it would resist shutdown attempts or changes to its source code).

This approach tries instead to expl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Animal Welfare Library 🦉, published by arvomm on May 17, 2023 on The Effective Altruism Forum.

TL;DR

Introducing the Animal Welfare Library (AWL🦉): a web repository of animal welfare resources. We hope the library is useful for:

People who want to learn about animal welfare.

More advanced users seeking references or organisations related to animal ethics and effective animal advocacy projects.

The library contains:

Books

Articles

Organisations

Videos and Films

Repositories

Advocates

News

Podcasts

And more, like newsletters, coming soon.

We want AWL to be a link you can easily share with someone wanting to learn more about animal welfare. At 212 entries right now, the library is still expanding and is by no means exhaustive (suggest additions here or in the comments!).

We found having a searchable and interactive overview of what is out there really helpful, and we hope you do too! We really value any feedback you might have!

Our Story

Today we just launched the Animal Welfare Library or AWL (pronounced owl /aʊl/ 🦉!). AWL is our answer to the question "what is the go-to place for finding high-quality animal welfare resources?". Here's our story.

Arvo: When I began my journey to learn more about animal welfare, I realised that there was an overwhelming amount of information on the subject, varying substantially in quality. Over the months and years that followed, I came across several websites and organisations that seemed to be doing incredible work and sharing valuable insights, and I frequently found myself wishing I had known about these resources from the start. I also started to see the interdisciplinary nature of the endeavour which made me realise that it would be particularly beneficial to develop a tool to catalogue some of the knowledge we possess and build a home for a centralised repository of valuable resources.

Eleos: The plight of badly-off humans and other animals has been a priority of mine for many years. There has always been a sense of urgency behind my thinking: to me, practical philosophy isn’t simply a set of interesting puzzles, but a fundamentally important enterprise to make the world a better place. A few years ago, I embarked on my path to study how I could apply my compassion more systematically in a way that helps me be more effective at making this world a better place. In summer 2021, I started gathering resources on animal ethics and helping animals that would help others make a difference, which resulted in this compilation (and this forum post) that would later help ground this project.

In 2022, we met and found we had an aligned vision for a project like AWL. We joined forces and decided to make the library happen.

This is how this project was born. This website is the resource we wish we had had in our hands when we started learning about animal welfare and humanity’s role in helping end the neglected suffering of millions of animals.

Go to the library here: Animal Welfare Library.

Structure of the Site

The website is structured into home, library and act now pages. (We are also drafting a `why care’ page summarising key arguments in the animal welfare space). The home page has some highlighted resources and organisations and it prompts visitors to explore the library. The Library has all the resources we compiled and the options to filter and search (desktop only) for specific items, themes or key terms. The act now is split into four parts: career (which redirects to Animal Advocacy Careers), donations (redirecting to Animal Charity Evaluators), expanding our social influence and eating plant-based (each of these last two redirects to a subset of relevant resources within the library). When possible, we tried to keep things minimalistic.

Feedback

This is all still work in progress - we wanted to put it out there be...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let’s use AI to harden human defenses against AI manipulation, published by Tom Davidson on May 17, 2023 on The AI Alignment Forum.

Views my own not my employers.

Summary

tldr: AI may manipulate humans; we can defend against that risk better by optimising AIs to manipulate humans, seeing what manipulation techniques they use, and learning to detect those techniques.

It’s critical that humans can detect manipulation from AIs for two reasons. Firstly, so that we don’t reward AIs for manipulative behaviour (outer alignment). Secondly, so that we can block attempts at AI takeover that run through manipulating humans.

Many standard techniques in alignment can be directed towards this goal. Using debate, we can reward one AI for persuading a human that another AI is being manipulative. The first AI could use techniques from interpretability and cross examination.

This post discusses a complementary approach, where AIs do “gain of function” research to i) discover techniques for manipulating humans and ii) harden human defenses against these techniques (with AI assistance). In other words, AIs are optimised to find “human vulnerabilities” and then patch those vulnerabilities.

The positive outcomes are:

Humans (with AI assistance) have stronger defenses against manipulation.

AIs are trained to avoid being manipulative and to instead use persuasion techniques that favour finding true conclusions, e.g. deductive reasoning.

As a consequence, AI debates favour truth more strongly.

By getting AI to research manipulation techniques, this approach takes advantage of the fact that, as we approach TAI, AIs will be able to do lots of cognitive work.

How might this look concretely? Here’s a very brief example:

Optimize a (possibly superhuman) “persuader-AI” to convince humans of true and false statements, where we know the ground truth.

The persuader-AI should manipulate humans if this makes it more convincing.

This step should be conducted in a secure environment to ensure that the manipulative persuader-AI doesn’t escape.

Optimize a (possibly superhuman) analyser-AI to identify human-understandable manipulation techniques that were used in #1.

This step can also be done in a secure environment, allowing us to use a powerful analyser-AI at lower risk.

Train limited detector-AIs to recognise the manipulation techniques found in #2.

Detector-AIs can then:

help humans detect manipulation techniques during AI debates, pushing the outcome towards the truth

monitor for AI manipulation in situations where full-blown debate isn’t practical (e.g. chatbots)

give AIs higher reward for using non-manipulative persuasion techniques and lower reward for using manipulative persuasion techniques

The techniques are all human-understandable, so humans can randomly check the accuracy of the AI outputs in #2, #3 and #4.

Some benefits of this approach:

Analyser-AIs, who identify manipulation techniques, can be trained using “process-based” optimisation, which is probably safer than “outcomes-based” optimisation.

Detector-AIs, who detect manipulation techniques, can be myopic (planning only over short time horizons) and limited in other ways. This makes collusion between AIs harder. (Collusion is a big potential pitfall of debate.)

This process for hardening human defenses is inherently modular. It involves multiple steps, and AIs performing each step need not understand the full process. This makes collusion harder.

Human defenses can stay “one step ahead” of AI offense even as each generation of AIs enables new manipulation techniques. For each new generation of AIs:

Discover the new manipulation techniques enabled by the new generation of AIs:

Optimize the new generation of AIs to manipulate humans in a secure test environment.

Identify the new manipulation techniques they use

Train detector-AIs to recog...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why AGI systems will not be fanatical maximisers (unless trained by fanatical humans), published by titotal on May 17, 2023 on The Effective Altruism Forum.

[Disclaimer: While I have dabbled in machine learning, I do not consider myself an expert.]

Introduction

When introducing newcomers to the idea of AI existential risk, a typical story of destruction will involve some variation of the “paperclip maximiser” story. The idea is that some company wishes to use an AGI to perform some seemingly simple and innocuous task, such as producing paperclips. So they set the AI with a goal function of maximizing paperclips. But, foolishly, they haven’t realized that taking over the world and killing all the humans would allow it to maximize paperclips, so it deceives them into thinking it’s friendly until it gets a chance to defeat humanity and tile the universe with paperclips (or wiggles that the AI interprets as paperclips under it's own logic).

What is often not stated in these stories is an underlying assumption about the structure of the AI in question. These AI’s are fixed goal utility function maximisers, hellbent on making an arbitrary number as high as possible, by any means necessary. I’ll refer to this model as “fanatical” AI, although I’ve seen other posts refer to them as “wrapper” AI, referring to their overall structure.

Increasingly, the assumption that AGI’s will be fanatical in nature is being challenged. I think this is reflected in the “orthodox” and “reform” Ai split. This post was mostly inspired by Nostalgebraist's excellent “why optimise for fixed goals” post, although I think there is some crossover with the arguments of the “shard theory” folks.

Humans are not fanatical AI. They do have goals, but the goals change over time, and can only loosely be described by mathematical functions. Traditional programming does not fit this description, being merely a set of instructions executed sequentially. None of the massively successful recent machine-learning based AI fits this description, as I will explain in the next section. In fact, nobody even knows how to make such a fanatical AI.

These days AI is being designed by trial-and-error techniques. Instead of hand designing every action it makes, we’re jumbling it's neurons around and letting it try stuff until it finds something that works. The inner working of even a very basic machine learning model is somewhat opaque to us. What is ultimately guiding the AI development is some form of evolution: the strategies that work survive, the strategies that don’t get discarded.

This is ultimately why I do not believe that most AI will end up as fanatical maximisers. Because in the world that an AI grows up in, trying to be a fanatical global optimizer is likely to get you killed.

This post relies on two assumptions: that there will be a fairly slow takeoff of AI intelligence, and that world takeover is not trivially easy. I believe both to be true, but I won't cover my reasons here for the sake of brevity.

In part 1, I flesh out the argument for why selection pressures will prevent most AI from becoming fanatical. In part 2, I will point out some ways that catastrophe could still occur, if AI is trained by fanatical humans.

Part 1: Why AI won't be fanatical maximisers

Global and local maxima

[I've tried to keep this to machine learning 101 for easy understanding].

Machine learning, as it exists today, can be thought of as an efficient trial and error machine. It contains a bazillion different parameters, such as the “weights” of a neural network, that go into one gigantic linear algebra equation. You throw in an input, compute the output of the equation, and “score” the result based on some goal function G. So if you were training an object recognition program, G might be “number of objects correctly identified”. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing The Long Game Project: Tabletop Exercises for a Resilient Tomorrow, published by Dr Dan Epstein on May 17, 2023 on The Effective Altruism Forum.

“No pilot would dare to fly a commercial airliner without significant training in a flight simulator . yet decision-makers are expected to make critical decisions relying on ‘theory’, ‘experience’, ‘intuition’, ‘gut feeling’, or less” - de Suarez et al 2012.

tl;dr

The Long Game Project is an organisation focused on helping institutions improve decision-making, enhance team collaboration, and build resilience. We aim to do this by a) Producing tools and resources to improve access to effective tabletop exercising; b) Providing advice, thought leadership and full-suite consultation to organisations interested in tabletop exercises; c) Encouraging a culture of running organisational games and building a community around tabletop games. Follow us on twitter and LinkedIn, play with our beta scenario-generator tool, provide feedback for some cash prizes and spread the word!

Set the scene

Imagine a world where organisations are prepared for the unexpected, where decision-makers can more confidently navigate crises and create positive outcomes even in the face of chaos because they have regular, simulated experience. Dynamic decision-making under uncertain conditions is a skill that can and should, be practised.

Experience matters.

Play it now,

before you live it later.

Roll for Initiative: Introducing The Long Game Project

Our mission is to help organisations improve decision-making (IIDM), enhance team collaboration, and build resilience using tabletop exercises that are rules-light but experience heavy.

Why we exist

The world is becoming increasingly complex and unpredictable, with organisations facing various challenges. Institutional decision-making in practice is often ill-equipped to handle such complexity, leading to sub-optimal decision quality and severe failure modes. 80,000 hours lists IIDM as one of the best ways to improve existing levers, interventions and practical suggestions for our long-term future and lists this as a neglected issue. While IIDM is a complex cause area, requiring some disentangling, there are existing levers, interventions and practical suggestions that are under-utilised in practice. The Long Game Project exists to bridge this gap by applying several levers to a diverse range of sectors, helping organisations adapt and thrive in an ever-changing landscape.

Our toolkit of levers includes; tabletop exercises, role-playing, future simulations, facilitation, probabilistic thinking games, goal and value alignment, decision design and other methods that combine game design and ideas from behavioural economics, psychology, and organisational theory.

Critical Hit: Our organisational aims

Be the place to point to for tabletop scenario advice and tools- Provide expert-level thought leadership and consultation on running serious tabletop exercises.

Lower the entry bar to effective tabletop exercising by producing tools, giving guidance and providing expertise.

Empower institutions to tackle complex challenges effectively by transforming how they plan, react, and adapt in an increasingly uncertain world.

Focus on scenarios of the world's most pressing problems and long-term horizons.

Encourage a culture of running organisational games and building a community around tabletop games.

Theory of Change

Our theory of change is rooted in the belief that effective decision-making is a learnable skill. By providing organisations with simulated experience, we aim to create a positive feedback loop in which participants develop stronger decision-making abilities, leading to better outcomes and increased resilience.

This, in turn, creates more efficient, effective, and adaptable organisations, ultimately c...