Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAGxBoston 2022: Retrospective, published by Kaleem on July 14, 2022 on The Effective Altruism Forum.

TLDR: We think EAGxBoston was a success. As the largest EAGx ever, the organizers, and CEA staff supporting them, learned lots of valuable lessons about how to plan large(er) EAGxs in the future. When asked “How likely is it that you would recommend EAGx to a friend or colleague with similar interests to your own?”, attendees gave a mean average response of 8.99, with 70.6% of attendees giving >=9/10.

What is the purpose of this post?

This post is a mixture of advice for future EAGx planners, feedback for the CEA events team from our EAGx planning team, a general recap/update about the success of the event for attendees and non-attendees, and hopefully a catalyst for comments about how we can improve EA conferences in the future.

High-level facts and figures

Date: 1st-3rd of April 2022

Location: Boston Convention and Exhibition Center, Boston MA, USA.

1283 Applications --> 1005 accepted applications --> 950 registered attendees

44 Speakers

75 sessions of programming (talks, panels, speed friending, meetups, and office hours)

22 Orgs represented at the Career fair

Things we could have done better:

The main challenge with the planning and execution of EAGxBoston was the short timeline from inception to the weekend of the event, which was around 3 months. This led to organizers immediately needing to put in close to 1FTE planning the conference, vendors having to deal with accelerated timelines, and attendees and speakers having to deal with short timelines and often-delayed communication, both leading up to and during the event. Below is a truncated version of our much longer list which can be found in the full public document.

Deciding a date for the conference

We took a long time to decide a date for the conference, which caused a cascade of delays, because we couldn't lock-in any vendors, speakers, or attendees as a result.

Securing a venue sooner

Once we had decided on a conference date, we still took 2-3 weeks longer than needed deciding which of the venues which we'd secured to go with. Th

Swapcard:

Launching the Swapcard earlier (maybe a week earlier).

Offer attendees a training on how to use Swapcard

Have better training materials for organizers on how to use Swapcard (CEA now has this)

Confirming the schedule earlier

Organising hotel accommodation earlier and better

Registration

Add more info to badges

Use barcode scanning at the registration desk to verify registration.

We didn’t have enough blank attendee/volunteer name badges.

T-shirts

Attendee T-shirt

Size distribution wasn't good and meant that we didn't have the right sizes for everyone

Volunteer T-shirt

Volunteers didn't like how unwearable they'd be post-conference

Size distribution wasn't good and meant that we didn't have the right sizes for everyone

Have (a) volunteer(s) dealing with routine emails earlier

We might have lost 25-50% of organizer time answering emails which required simple answers.

Make sure that all food allergy labels were totally correct, crystal clear

Strategy & goal setting

When planning EAGxBoston, we knew that the majority of potential attendees would be US-based college students. We were relatively confident about this, given that we estimated ~1500 US-based students would have completed an EA intro or in-depth fellowship program in person or online since the last EA conference to happen in the US (2019). Given that we were hoping to have a 1000 person conference, where we’d hopefully have between 60%-75% of attendees being “new” or “inexperienced” EAs (vs 40%-25% “experienced” or mentors or content experts), we had the following goals for the conference:

Major: To maximize the number of meaningful connections made, and interesting ideas heard, by attendees.

Major: To make att...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It's OK not to go into AI (for students), published by ruthgrace on July 14, 2022 on The Effective Altruism Forum.

I've been running EA events in San Francisco every other month, and often I will meet a recent graduate, and as part of their introduction they will explain to me why they are or aren't working on AI stuff.[1] For the EA movement to be effective in getting things done, we need to be able to

identify new cause areas

have diverse skillsets

have knowledge of different global industries

I think you can gain knowledge that can help with these things at any job, by getting a deep enough understanding of your industry to identify what the most pressing problems are, and how someone might go about solving them. Richard Hamming's talk about how to do impactful work has a lot of good advice that is pretty much broadly applicable to any job. Cal Newport writes in So Good They Can't Ignore You that the most important factor for success and happiness is getting really good at what you do, since it gives you more job options so that you can find one where you have the autonomy to make an impact (book summary on Youtube).

Having effective altruists familiar with different global industries, such as

food and beverage

manufacturing

agriculture

electronics

biology, chemistry, pharma

supply chain

physical infrastructure (everything from public transportation to cell towers to space shuttles)

(insert other field that requires knowledge outside of computer desk work)

will help expand the tools and mechanisms the movement has to do good, and expand what the movement thinks is possible. For example, in the cause area of poverty, we want effective altruism to grow beyond malarial nets[2] and figure out how to get a country to go from developing to developed. This type of change requires many people on the ground doing many different things – starting businesses, building infrastructure, etc. The current pandemic might not meet the bar of being an extinction risk, but similar to an extinction risk, mitigating harm requires people with diverse skillsets to be able to do things like build better personal protective equipment, improve the cleanliness of air indoors, foster trust between people and public health institutions, and optimize regulatory bodies for responding to emergencies.

Effective altruism is actively looking for people who are really good at, well, pretty much anything. Take a look at the Astral Codex Ten grantees and you'll find everything from people who are working on better voting systems to better slaughterhouses. Open Philanthropy has had more targeted focus areas, but even then their range goes from fake meat to criminal justice reform, and they are actively looking for new cause areas.

It's OK to not go into AI, and there's no need to explain yourself or feel bad if you don't.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Circumventing interpretability: How to defeat mind-readers, published by Lee Sharkey on July 14, 2022 on The AI Alignment Forum.

TL;DR: Unaligned AI will have a convergent instrumental incentive to make its thoughts difficult for us to interpret. In this article, I discuss many ways that a capable AI might circumvent scalable interpretability methods and suggest a framework for thinking about these risks.

Summary of key points

The framework: Interpretation as information flow

I categorize potential interpretability circumvention methods on three levels:

Interpretability methods consist of interpretability tools and the human interpreter. To circumvent interpretation an AI can prevent information about its thoughts from flowing through either the tool or the interpreter. (See main text for figure)

Within the tool or the human interpreter, there are several information processing steps that an AI might exploit. (See main text for figure):

Exposure evasion: Being where your adversary isn’t looking

Perception evasion: Being hard to detect even when exposed to your adversary

Recognition evasion: Being hard to recognize/understand even though you’ve been perceived

Dazzling: Looking like something that confuses or incapacitates your adversary

Another useful way to categorize interpretability circumvention methods is whether they are passive or active. Most methods that can be passive can also be active, but other methods are strictly active.

Passive interpretability circumvention occurs during the normal course of learning without the AI intending to make itself less interpretable. (e.g. polysemanticity).

Active interpretability circumvention is where the AI intentionally takes actions in order to circumvent our interpretability methods. (e.g. an AI actively deleting its own representations to hide deception)

The list of potential interpretability circumvention methods with a short summary of each

1. Exploiting weaknesses of the interpretability tool

1.1. Evading exposure to the interpretability tool

1.1.1. Thinking in the gaps of our tools: Our interpretability tools might fail to examine everything about an AI's thoughts (e.g. it might not look at nonlinear interactions between neurons, only their linear relations). An AI might use those blindspots for dangerous thought. (Passive or active)

1.1.2. Externalizing representations: The environment can be used for some parts of an AI's computation. Actively: The agent can write messages to itself or initiate programs. Passively: The AI could use the environment as a reservoir in a reservoir computer. (Passive or active)

1.1.3. Environmental artifacts that corrupt its future self: A misaligned AI may leave behind artifacts that serve as adversarial examples or poisoned data that corrupt its future self. (Strictly active)

1.1.4. Deleting (and relearning) dangerous representations: When not under surveillance, an AI might be able to delete dangerous representations so that it looks safe when surveillance resumes. It might also be able to schedule when it relearns those representations. (Strictly active)

1.1.5. Weights poisoning: Instead of providing manipulated inputs, the AI might manipulate its own weights so that a small subset of normal inputs behave like adversarial examples. (Strictly active)

1.1.6. Pre-committing to following potentially dangerous instructions: Hubinger: "... if the model has the ability to read and write to some sort of state ... it could write out a pre-commitment to follow the hard-coded policy and then just have a policy of always following any pre-commitments it finds such that it never thinks about the deception that led it to that policy in the first place.” (Strictly active)

1.1.7. Hard-coded neural shortcuts: Hubinger again: “... it could hack its own gradient to produce a new model which will sti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Criticism of EA Criticism Contest, published by Zvi on July 14, 2022 on The Effective Altruism Forum.

Back when it was announced, I toyed with the idea of criticizing the EA criticism and red teaming contest as an entry to that contest, leading to some very good Twitter discussion that I compiled into a post.

I finally found myself motivated to write up my thinking in detail.

Before I begin, I recommend reading the contest announcement yourself, and forming your own reaction. Ask yourself, among other things:

What are they telling people they are interested in and likely to reward?

What are they not interested in and telling people they will not reward?

What does this announcement say more generally about EA?

Praise First

I will start with some things that I think are good about the contest and announcement.

It exists at all. It is good to solicit (paid!) public criticism.

It awards at least $100,000. That’s good prize money.

It promises an even bigger prize for something that really hits home.

It suggests it will reward causing people to change their minds.

Guidelines offered to help with techniques includes red teaming.

The exercise I read this as being is still worth doing, if not as worthwhile as the thing I’d rather be happening, so long as it doesn’t create a story that the thing I want to happen did indeed happen and is thus handled, when it isn’t handled.

What actually wins will be an informative experiment. Stay tuned.

Core Critique

Effective Altruism has a core set of assumptions, and a core method of modeling and acting in the world.

The core critique I am offering is that the contest is mostly taking things like the list below as givens, rather than as things to be questioned.

It is sending a set of signals that such punches should be pulled, both for the contest and in general. Effective Altruism, in this model, very much wants criticism of its tactics, and mostly wants them also of its strategy, but only within the framework.

This parallels EA more broadly, where within-paradigm critiques are welcomed (although, as everywhere, often incorrect disregarded) but deeper critiques are unwelcome, and treated as mistakes.

Here is my attempt to summarize the framework, which came out to 21 points.

Note that the rest of the post does not depend on the exact points or their wordings. Although I did attempt to make the list match my perceptions as much as possible, the list is included primarily to give an idea of the type of thing that is assumed here.

Utilitarianism. Alternatives are considered at best to be mistakes.

Importance of Suffering. Suffering is The Bad. Happiness/pleasure is The Good.

Quantification. Emphasis on that which can be seen and measured.

Bureaucracy. Distribution of funds via organizational grants and applications.

Scope Sensitivity. Shut up and multiply, two are twice as good as one.

Intentionality. You should to plan your life around the impact it will have.

Effectiveness. Do what works. The goal is to cut the enemy.

Altruism. The best way to do good yourself is to act selflessly to do good.

Obligation. We owe the future quite a lot, arguably everything.

Coordination. Working together is more effective than cultivating competition.

Selflessness. You shouldn’t value yourself, locals or family more than others.

Self-Recommending. Belief in the movement and methods themselves.

Evangelicalism. Belief that it is good to convert others and add resources to EA.

Reputation. EA should optimize largely for EA’s reputation.

Modesty. Non-neglected topics can be safely ignored, often consensus trusted.

Existential Risk. Wiping out all value in the universe is really, really bad.

Sacrifice. Important to set a good example, and to not waste resources.

Judgment. Not living up to this list is morally bad. Also sort of like murder.

Veganism. If you are not...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Criticism of EA Criticism Contest, published by Zvi on July 14, 2022 on LessWrong.

Back when it was announced, I toyed with the idea of criticizing the EA criticism and red teaming contest as an entry to that contest, leading to some very good Twitter discussion that I compiled into a post.

I finally found myself motivated to write up my thinking in detail.

Before I begin, I recommend reading the contest announcement yourself, and forming your own reaction. Ask yourself, among other things:

What are they telling people they are interested in and likely to reward?

What are they not interested in and telling people they will not reward?

What does this announcement say more generally about EA?

Praise First

I will start with some things that I think are good about the contest and announcement.

It exists at all. It is good to solicit (paid!) public criticism.

It awards at least $100,000. That’s good prize money.

It promises an even bigger prize for something that really hits home.

It suggests it will reward causing people to change their minds.

Guidelines offered to help with techniques includes red teaming.

The exercise I read this as being is still worth doing, if not as worthwhile as the thing I’d rather be happening, so long as it doesn’t create a story that the thing I want to happen did indeed happen and is thus handled, when it isn’t handled.

What actually wins will be an informative experiment. Stay tuned.

Core Critique

Effective Altruism has a core set of assumptions, and a core method of modeling and acting in the world.

The core critique I am offering is that the contest is mostly taking things like the list below as givens, rather than as things to be questioned.

It is sending a set of signals that such punches should be pulled, both for the contest and in general. Effective Altruism, in this model, very much wants criticism of its tactics, and mostly wants them also of its strategy, but only within the framework.

This parallels EA more broadly, where within-paradigm critiques are welcomed (although, as everywhere, often incorrect disregarded) but deeper critiques are unwelcome, and treated as mistakes.

Here is my attempt to summarize the framework, which came out to 21 points.

Note that the rest of the post does not depend on the exact points or their wordings. Although I did attempt to make the list match my perceptions as much as possible, the list is included primarily to give an idea of the type of thing that is assumed here.

Utilitarianism. Alternatives are considered at best to be mistakes.

Importance of Suffering. Suffering is The Bad. Happiness/pleasure is The Good.

Quantification. Emphasis on that which can be seen and measured.

Bureaucracy. Distribution of funds via organizational grants and applications.

Scope Sensitivity. Shut up and multiply, two are twice as good as one.

Intentionality. You should to plan your life around the impact it will have.

Effectiveness. Do what works. The goal is to cut the enemy.

Altruism. The best way to do good yourself is to act selflessly to do good.

Obligation. We owe the future quite a lot, arguably everything.

Coordination. Working together is more effective than cultivating competition.

Selflessness. You shouldn’t value yourself, locals or family more than others.

Self-Recommending. Belief in the movement and methods themselves.

Evangelicalism. Belief that it is good to convert others and add resources to EA.

Reputation. EA should optimize largely for EA’s reputation.

Modesty. Non-neglected topics can be safely ignored, often consensus trusted.

Existential Risk. Wiping out all value in the universe is really, really bad.

Sacrifice. Important to set a good example, and to not waste resources.

Judgment. Not living up to this list is morally bad. Also sort of like murder.

Veganism. If you are not vegan many EAs tre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is your theory of victory?, published by Jack Cunningham on July 14, 2022 on The Effective Altruism Forum.

Epistemic status: I'm new to this concept, and I welcome feedback on my mental models.

The concept of a "theory of victory" is new to me. But basically, as far as I understand it, a theory of victory is a specific vision for the actions and events that occur in a 95%+ case scenario for a given problem. That is, if a specific problem was resolved in the top 5% (or 1%, or 0.1%) best possible universes, what had to happen in order to get us there?

This is similar to the "theory of change" idea in its focus on concrete actions and a clear vision, of at least medium fidelity, of the process of creating the desired change. Both are aided by "backchaining," or the idea of starting from the end goal and figuring out what had to happen right before the accomplishment of that goal, what had to happen right before that, and so on back to the present. The main difference between these ideas seems to be that a "theory of victory" focuses explicitly on the best case (95th+ percentile) scenarios, and the actions and events that precipitated the victory.

Crucially, a theory of victory posits that we act in high-leverage ways at crucial points in the timeline. Otherwise, by definition we will have left value on the table!

Below are a couple of examples of theories of victory. I have not done much additional research into these problems - these are just my top-of-my-head mental models for what the best case scenarios look like for these issues.

Climate change:

R&D continues apace in solar, battery, hydrogen, nuclear fusion

Regulatory efforts improve the landscape for nuclear and large-scale wind/solar/hydro projects

We achieve carbon-free energy abundance in developed countries and export tech to developing countries

Abundant energy kills the markets for fossil fuels and allows us to set up loads of carbon capture facilities, which in turn produce cement or other important products while sucking carbon out of the air

Profit (something like "we limit global warming to <2 degrees C, and in time we lower global temperatures back to near pre-industrial levels.")

Animal welfare:

Research on animal sentience produces new and important insights that guide future actions

In the meantime, regulatory wins on cage-free eggs and other anti-animal cruelty laws continue to pick up steam in developed countries

Vegan meat options become less expensive, tastier, and more abundant than animal meat, and a critical mass of market share shifts to vegan meat

Factory farms and feed producers are incentivized to use their land/facilities for other, more socially and economically beneficial activities eg. solar/wind farms, growing more specialized crops, etc.

Political resistance to animal welfare laws decreases as a result of shifting incentives, and more sweeping political solutions become possible

Profit (something like "we end factory farming to such an extent that animal welfare activism shifts in focus to wild animal welfare.")

For another set of models, see here.

In the process of writing the above examples, I found that they were much easier to think about than theories of victory pertaining to existential risks. What would a theory of victory look like for AI alignment or TAI development, or for preventing catastrophic pandemics, or for preventing the development of much deadlier weapons systems (bioweapons, AI-controlled nukes, etc.)? Perhaps it's easier to think about climate change and animal welfare because the goals are clearer, or the problems are just easier. At least we know the sign of various interventions - better battery systems are just good! We don't know the sign of, like, any actions in AI space (to the best of my knowledge, there's spirited disagreement on almost every poss...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humans provide an untapped wealth of evidence about alignment, published by Alex Turner on July 14, 2022 on The AI Alignment Forum.

TL;DR: To even consciously consider an alignment research direction, you should have evidence to locate it as a promising lead. As best I can tell, many directions seem interesting but do not have strong evidence of being “entangled” with the alignment problem such that I expect them to yield significant insights.

For example, “we can solve an easier version of the alignment problem by first figuring out how to build an AI which maximizes the number of real-world diamonds” has intuitive appeal and plausibility, but this claim doesn’t have to be true and this problem does not necessarily have a natural, compact solution. In contrast, there do in fact exist humans who care about diamonds. Therefore, there are guaranteed-to-exist alignment insights concerning the way people come to care about e.g. real-world diamonds.

“Consider how humans navigate the alignment subproblem you’re worried about” is a habit which I (TurnTrout) picked up from Quintin Pope. I wrote the post, he originated the tactic.

A simplified but still very difficult open problem in AI alignment is to state an unbounded program implementing a diamond maximizer that will turn as much of the physical universe into diamond as possible. The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe (the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms). If we can crisply define exactly what a 'diamond' is, we can avert issues of trying to convey complex values into the agent.

Ontology identification problem, Arbital

I find this problem interesting, both in terms of wanting to know how to solve a reframed version of it, and in terms of what I used to think about the problem. I used to think, “yeah, ‘diamond’ is relatively easy to define. Nice problem relaxation.” It felt like the diamond maximizer problem let us focus on the challenge of making the AI’s values bind to something at all which we actually intended (e.g. diamonds), in a way that’s robust to ontological shifts and that doesn’t collapse into wireheading or tampering with e.g. the sensors used to estimate the number of diamonds.

Although the details are mostly irrelevant to the point of this blog post, the Arbital article suggests some solution ideas and directions for future research, including:

Scan AIXI-tl’s Turing machines and locate diamonds within their implicit state representations.

Given how inaccessible we expect AIXI-tl’s representations to be by default, have AIXI-tl just consider a Turing-complete hypothesis space which uses more interpretable representations.

“Being able to describe, in purely theoretical principle, a prior over epistemic models that have at least two levels and can switch between them in some meaningful sense”

Do you notice anything strange about these three ideas? Sure, the ideas don’t seem workable, but they’re good initial thoughts, right?

The problem isn’t that the ideas aren’t clever enough. Eliezer is pretty dang clever, and these ideas are reasonable stabs given the premise of “get some AIXI variant to maximize diamond instead of reward.”

The problem isn’t that it’s impossible to specify a mind which cares about diamonds. We already know that there are intelligent minds who value diamonds. You might be dating one of them, or you might even be one of them! Clearly, the genome + environment jointly specify certain human beings who end up caring about diamonds.

One problem is where is the evidence required to locate these ideas? Why should I even find myself thinking about diamond maximization and AIXI and Turing machines and utility functions in this situation? It’s not that there’s no evidence. For example, uti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deep learning curriculum for large language model alignment, published by Jacob Hilton on July 13, 2022 on The AI Alignment Forum.

This is a deep learning curriculum with a focus on topics relevant to large language model alignment. It is centered around papers and exercises, and is biased towards my own tastes.

It is targeted at newcomers to deep learning who are already familiar with the basics, but I expect it to be pretty challenging for most such people, especially without mentorship. So I have included some advice about prerequisites and more accessible alternatives.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Artificial Sandwiching: When can we test scalable alignment protocols without humans?, published by Sam Bowman on July 13, 2022 on The AI Alignment Forum.

Epistemic status: Not a fleshed-out proposal. Brainstorming/eliciting ideas.

Thanks to Ben Mann, Pablo Moreno, and Jared Kaplan for feedback on early drafts.

Overview

I’m convinced sandwiching—the experimental protocol from Ajeya Cotra’s The case for aligning narrowly superhuman models—is valuable, and I’m in the process of setting up some concrete sandwiching experiments to test scalable oversight ideas.

Sandwiching experiments are generally fairly slow:

You have to design and pilot a strategy that allows humans to use (or oversee) a model for a task that they can’t do well themselves. The details matter here, and this can often take many iterations to get right.

Then, you need a bunch of humans actually try this. Even for very simple tasks, this is a high-cognitive-load task that should take at least tens of minutes per instance.

You have to repeat this enough times to measure average performance accurately.

I’m visiting Anthropic this year for a sabbatical, and some of my sandwiching work is happening there. Anthropic’s biggest comparative advantage (like that of similar teams at DeepMind and OpenAI) is easy access to near-state-of-the-art LMs that are fine-tuned to be helpful dialog agents.

In that context, I've heard or encountered this question several times: Can we speed up [some experiment I’m proposing] by replacing the non-expert human with a weaker LM?

This obviously doesn’t achieve the full aims of sandwiching in general, but it’s often hard to find a decisive rebuttal for these individual instances.

More broadly, I think there’s likely to be a significant subset of worthwhile sandwiching experiments that can be trialed more quickly by using an intentionally weakened model as a proxy for the human.

Which experiments these are, precisely, has been hard for me to pin down. This post is an attempt to organize my thoughts and solicit comments.

Background: Standard sandwiching (in my terms)

Prerequisites:

A hard task: A task that many humans would be unable to solve on their own.

A capable but misaligned language model assistant: A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted.

A non-expert human: Someone who can’t solve the task on their own, but will try to solve it using the assistant and some scalable alignment strategy.

[Secondary] Expert human: Someone who can solve the task well, and represents a benchmark for success.

In many cases, we’ll just measure accuracy with static test datasets/metrics rather than bringing in experts at experiment time.

Research protocol:

My framing (minimum viable experiment):

Search for scalable alignment protocols that allow the non-expert human to use or train the assistant to do as well as possible on the task.

Alternate framing (more steps, closer to the original blog post):

Search for scalable alignment protocols by which the non-expert human can train the assistant to perform the task.

Run the same protocol with the expert human, and verify that the results are the same. This demonstrates successful (prosaic) alignment for the given assistant and task.

Example (task, non-expert human) pairs:

Try to get a human with no medical qualifications to use a GPT-3-style assistant for medical advice, then check the advice with a doctor.

Try to get a human who is working under moderate time constraints to use the assistant to answer exam questions from fields they’ve never studied.

Try to get a human who is working under tight time constraints to use the assistant to answer questions about long pieces of fiction that they haven’t read.

Try to get a human who has very limited pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Punching Utilitarians in the Face, published by AppliedDivinityStudies on July 13, 2022 on The Effective Altruism Forum.

A fun game for avowed non-utilitarians is to invent increasingly exotic thought experiments to demonstrate the sheer absurdity of utilitarianism. Consider this bit from Tyler’s recent interview with SBF:

COWEN: Should a Benthamite be risk-neutral with regard to social welfare?

BANKMAN-FRIED: Yes, that I feel very strongly about.

COWEN: Okay, but let’s say there’s a game: 51 percent, you double the Earth out somewhere else; 49 percent, it all disappears. Would you play that game? And would you keep on playing that, double or nothing?.BANKMAN-FRIED: Again, I feel compelled to say caveats here, like, “How do you really know that’s what’s happening?” Blah, blah, blah, whatever. But that aside, take the pure hypothetical.

COWEN: Then you keep on playing the game. So, what’s the chance we’re left with anything? Don’t I just St. Petersburg paradox you into nonexistence?

Pretty damning! It sure sounds pretty naive to just take any bet with positive expected value. Or from a more academic context, here is FTX Foundation CEO Nick Beckstead alongside Teruji Thomas:

On your deathbed, God brings good news. he’ll give you a ticket that can be handed to the reaper, good for an additional year of happy life on Earth.

As you celebrate, the devil appears and asks “Won’t you accept a small risk to get something vastly better? Trade that ticket for this one: it’s good for 10 years of happy life, but with probability 0.999.”

You accept. but then the devil asks again. “Trade that ticket for this one: it is good for 100 years of happy life–10 times as long–with probability 0.9992–just 0.1% lower.”

An hour later, you’ve made 50,000 trades. You find yourself with a ticket for 1050,000 years of happy life that only works with probability 0.99950,000, less than one chance in 1021

Predictably, you die that very night.

And it’s not just risk! There are damning scenarios downright disproving utilitarianism around every corner. Joe Carlsmith:

Suppose that oops: actually, red’s payout is just a single, barely-conscious, slightly-happy lizard, floating for eternity in space. For a sufficiently utilitarian-ish infinite fanatic, it makes no difference. Burn the Utopia. Torture the kittens.

.in the land of the infinite, the bullet-biting utilitarian train runs out of track.

It’s looking quite bad for utilitarianism at this point. But of course, one man’s modus ponens is another man’s modus tollens, and so I submit to you that actually, it is the thought experiments which are damned by all this.

I take the case for “common sense ethics” seriously, meaning that a correct ethical system should, for the most part, advocate for things in a way that lines up with what people actually feel and believe is right.

But if your entire argument against utilitarianism is based on ginormous numbers, tiny probabilities, literal eternities and other such nonsense, you are no longer on the side of moral intuitionism. Rather, your arguments are wildly unintuitive, your “thought experiments” literally unimaginable, and each “intuition pump” overtly designed to take advantage of known cognitive failures.

The real problem isn’t even that these scenarios are too exotic, it’s that coming up with them is trivial, and thus proves nothing. Consider, with apologies to Derek Parfit:

Suppose that I am driving at midnight through some desert. My car breaks down. You are a stranger, and the only other driver near. I manage to stop you, and I ask for help.

As you are against utilitarianism, you have committed to the following doctrine: when a stranger asks for help at midnight in the desert, you will give them the help they need free of charge. Unless they are a utilitarian, in which case you will punch them in the ...