The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLM Generality is a Timeline Crux, published by Egg Syntax on June 24, 2024 on The AI Alignment Forum.

Short Summary

LLMs may be fundamentally incapable of fully general reasoning, and if so, short timelines are less plausible.

Longer summary

There is ML research suggesting that LLMs fail badly on attempts at general reasoning, such as planning problems, scheduling, and attempts to solve novel visual puzzles. This post provides a brief introduction to that research, and asks:

Whether this limitation is illusory or actually exists.

If it exists, whether it will be solved by scaling or is a problem fundamental to LLMs.

If fundamental, whether it can be overcome by scaffolding & tooling.

If this is a real and fundamental limitation that can't be fully overcome by scaffolding, we should be skeptical of arguments like Leopold Aschenbrenner's (in his recent 'Situational Awareness') that we can just 'follow straight lines on graphs' and expect AGI in the next few years.

Introduction

Leopold Aschenbrenner's recent 'Situational Awareness' document has gotten considerable attention in the safety & alignment community. Aschenbrenner argues that we should expect current systems to reach human-level given further scaling[1], and that it's 'strikingly plausible' that we'll see 'drop-in remote workers' capable of doing the work of an AI researcher or engineer by 2027. Others hold similar views.

Francois Chollet and Mike Knoop's new $500,000 prize for beating the ARC benchmark has also gotten considerable recent attention in AIS[2]. Chollet holds a diametrically opposed view: that the current LLM approach is fundamentally incapable of general reasoning, and hence incapable of solving novel problems. We only imagine that LLMs can reason, Chollet argues, because they've seen such a vast wealth of problems that they can pattern-match against.

But LLMs, even if scaled much further, will never be able to do the work of AI researchers.

It would be quite valuable to have a thorough analysis of this question through the lens of AI safety and alignment. This post is not that[3], nor is it a review of the voluminous literature on this debate (from outside the AIS community). It attempts to briefly introduce the disagreement, some evidence on each side, and the impact on timelines.

What is general reasoning?

Part of what makes this issue contentious is that there's not a widely shared definition of 'general reasoning', and in fact various discussions of this use various terms. By 'general reasoning', I mean to capture two things. First, the ability to think carefully and precisely, step by step. Second, the ability to apply that sort of thinking in novel situations[4].

Terminology is inconsistent between authors on this subject; some call this 'system II thinking'; some 'reasoning'; some 'planning' (mainly for the first half of the definition); Chollet just talks about 'intelligence' (mainly for the second half).

This issue is further complicated by the fact that humans aren't fully general reasoners without tool support either. For example, seven-dimensional tic-tac-toe is a simple and easily defined system, but incredibly difficult for humans to play mentally without extensive training and/or tool support.

Generalizations that are in-distribution for humans seems like something that any system should be able to do; generalizations that are out-of-distribution for humans don't feel as though they ought to count.

How general are LLMs?

It's important to clarify that this is very much a matter of degree. Nearly everyone was surprised by the degree to which the last generation of state-of-the-art LLMs like GPT-3 generalized; for example, no one I know of predicted that LLMs trained on primarily English-language sources would be able to do translation between languages. Some in the field argued as...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 35 Interactive Learning Modules Relevant to EAs / Effective Altruism (that are all free), published by spencerg on June 24, 2024 on The Effective Altruism Forum.

Hi all! For quite a number of years now at Clearer Thinking we've been developing free, interactive, online learning modules on a wide variety of topics. I wanted to create a list of which of these tools are most relevant to EAs and to Effective Altruism itself, in case you find them useful (for yourself, or for sharing with others). Here's the list, organized by topic:

1. Calibration, prediction and knowledge practice

Calibrate Your Judgment - this program contains thousands of question sets to help you become adept at making well-calibrated judgments. It also tracks your progress over time so you can see how you improve!

Predict Psychological Correlations - test how accurate your understanding of psychology is by practicing and improving your ability to make predictions.

Retrocaster - take the challenge to identify the correct global trends on a variety of topics - from Africa's future population to the evolution of CO2 emissions in the US.

Biosafety Quiz - uncover the history of pandemics, test your biosafety knowledge, and learn essential steps to safeguard humanity's future.

Long-Term Future Quiz - get a glimpse into the opportunities that may await humanity if we can avoid disasters such as nuclear war and extreme climate change. Invite your imagination to take flight with 11 multiple-choice questions.

Artificial Intelligence Quiz - test your knowledge of artificial intelligence and some of the risks it poses.

The Common Misconceptions Test - Separate fact from B.S. in this fun quiz that asks you to identify the misconceptions among 30 common beliefs.

World's Biggest Problems Quiz - test your knowledge of some of the world's most pressing problems: global health, animal welfare, and existential risks to humanity.

Political Bias Test - learn to spot your political biases and test your policy knowledge

Charity Effectiveness - see if you can guess which social interventions had a positive effect, a negative effect, or no effect, based on a short description.

Guess Which Experiments Replicate - Can you guess which of these high-profile experiments were legit?

Improve Your Frequency Predictions - a quick quiz to test your ability to estimate the frequency of events and compare your scores to others'.

2. Sharpening Your Thinking

Nuanced Thinking Techniques - Learn to recognize 3 common binary thinking traps and learn the nuanced thinking techniques you can use to combat them.

The Question of Evidence - there is more to interpreting evidence than you might think. Learn to improve your predictions by understanding evidence on a deeper level using Bayes' rule.

Can You Detect Weak Arguments? - Find out how well you can spot misleading rhetoric and recognize your own bias using real-world examples

Faulty Reasoning Quiz - learn about five types of faulty arguments and take a quiz to see how well you can spot them

Rhetorical Fallacies - train yourself to detect common reasoning errors that often go unnoticed.

The Belief Challenger - our beliefs determine our behaviors and how we interact with others. Use this program to refine your beliefs and form more accurate views.

The Planning Fallacy - ever notice that projects usually take longer than people predict? Understand the science behind this phenomenon and become a more reliable planner.

Mental Traps - scientists have discovered a slew of systematic errors in human thought. Learn about a few of the mind's mistakes in this brief introduction.

3. Communication

Productive Disagreements - learn techniques to have constructive arguments

Become A Great Listener - this tool teaches the skill of Active Listening, a strategy commonly used by therapists, helpline volunteers,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten minutes to speak up for ~4.5 million caged chickens, published by Ben Stevenson on June 24, 2024 on The Effective Altruism Forum.

(You can read this post as a Google Doc. You might find this easier to share with animal-sympathetic non-EAs. Also: I work at Rethink Priorities, but I'm writing in a personal capacity.)

A few weeks ago, I shared some suggested responses for a Defra consultation on welfare labelling. I was really pleased to hear that many EAs and animal rights advocates wrote in to support the proposal. I'm now sharing a similar opportunity: ~ten minutes of your time to speak up for ~4.5 million caged chickens.

If you want to help, you have less than 48 hours to get this done!

What's the public consultation?

The Scottish government is running a public consultation on their proposal to ban cages for layer hens. Agriculture is a devolved power so Holyrood have responsibility for Scottish animal welfare.

Let's be clear: cage-free chickens have dramatically better lives than caged chickens. Liberating caged chickens alleviates a great deal of suffering.[1]

Over 80% of Scottish layer hens are already cage-free. A greater proportion than across the whole UK, this is a testament to hard work from animal welfare campaigners.

Nevertheless, a ban still seems really important:

Corporate commitments might not last forever. A ban can lock in campaigners' hard-won victories and pre-empt businesses backsliding

Corporate commitments and welfare schemes might not always be met or enforced. A recent investigation from Animal Rising revealed that at least one English farm kept chickens in cages despite being RSPCA Assured. We could have greater trust that chickens are truly free from cages if they were banned

The pro-animal movement will have another win, which builds political momentum for our cause. The Scottish government notes that Luxembourg and Austria have already banned cages; that Germany, the Czech Republic and Slovakia are phasing them out; and that Australia aims to ban cages by 2036. If Scotland is attentive to other governments' bans, then other governments might pay attention to a Scottish ban

18% of Scottish layer hens are still so many - over a million - birds. None of them deserve to live their whole life condemned to a cage.

The government are considering three options:

1. 2030 ban on enriched cage production

2. 2030 ban on installing new cages, and 2034 ban on enriched cage production

3. "Information campaign aimed at consumers to encourage all retailers who have not signed up to the retailer pledge and caterers to commit to stop stocking/selling eggs or egg products from enriched cages by 2034"[2]

Option (2) is the default because the government thinks this best balances improving bird welfare and supporting the egg industry. This proposal is what the public consultation is about.

I think pursuing (2) rather than (1) is really disappointing. Four more years of cages means up to 4.5 million layer hens will live their lives behind bars.[3] And there are only fifteen farms currently keeping caged layer hens in Scotland; do they really need four more years of warning? An instant ban on installing new cages and a 2030 ban on all enriched cage production seem eminently reasonable to me.

How you can help

You can access the public consultation here and submit your response by 25 June 2024. You don't have to be Scottish to respond.

The consultation is divided into several pages. Only the 'About you' page is necessary.

Page

This is about

Number of questions

Animal welfare

Will a ban on cages help chickens express natural behaviours and improve other welfare outcomes?

2 questions

Policy proposal

Do you agree with the proposal?

5 questions

Capital costs

Estimating cost to convert cage farms to cage-free

1 question

Production costs

Estimating cost per egg under c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Different senses in which two AIs can be "the same", published by Vivek Hebbar on June 24, 2024 on The AI Alignment Forum.

Sometimes people talk about two AIs being "the same" or "different" AIs. We think the intuitive binary of "same vs. different" conflates several concepts which are often better to disambiguate. In this post, we spell out some of these distinctions. We don't think anything here is particularly novel; we wrote this post because we think it's probably mildly helpful for people to think this through, and because Buck fairly often wants to reference it.

Some example distinctions:

In a debate protocol[1], people often refer to the judge and debaters as three "different" AIs because their tokens are subject to different training incentives. But you could also say that the two debaters are "the same AI" because their prompts and context look similar. Or you could say that all three are the "same AI" because they were fine-tuned from the same weights.

In a control setting, an untrusted monitor might be fine-tuned from the same base model as the model it's monitoring, leaving it intuitively unclear whether they are "the same AI".

We think that it's often important to be able to distinguish different senses of "personal identity" when discussing various issues in AI safety, especially issues related to collusion. We frame each sense of personal identity as telling us whether two actions[2] belong to the "same AI" or "different AIs". Here are the senses we distinguish in this post:

1. Model weights: Were the two actions produced by two different models?

1. Pretrain identity: Models are different if they come from separate pretraining runs

2. Parameter identity: Models are different even if they are fine-tunings of the same pretrained model

2. Shared context: Do the two actions have shared LLM context?

3. Shared activations: Are the two actions downstream of shared activations?

4. Shared memory: In a scaffold, did the processes producing the two actions have access to a shared memory bank?

5. Shared reward: In an RL setting, do the actions share the same rewards?

6. Shared role in training: In protocols like debate, the training incentive can be different in nature for different classes of prompts/contexts. Do the two actions come from contexts in the same class?

Model identity

Model identity is based on which model parameters were run to produce a given action. "Pretrain identity" conflates all models finetuned from the same pretraining run. Whereas "parameter identity" requires the weights to be identical.

Some reasons to care about model identity:

The presence of issues such as scheming[3] is more correlated across models with shared training history

Models with shared training history might have similar misaligned goals, making them more inclined to coordinate

Similarities between their internal ontologies might make it easier to coordinate on Schelling signals or steganographically encode information

Shared activations & shared context

For "shared activations", we consider two actions to belong to the same AI when they are downstream of the same activations. For a deterministic[4] unidirectional transformer, two actions share activations if they are generated by the same model and their contexts share a common prefix. However, future architectures might break the close correspondence between shared activations and shared context.

For example, consider this setup: An LLM sees [prompt 1], generates some [python code], then [prompt 2] is appended instructing the same LLM to critique the code for security flaws, and it generates [critique]. If the activations on the [prompt 1] tokens contain a signal saying that it's time for coordinated defection, the model can see those activations both while generating the code and while generating the critique.

If we ins...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applying Force to the Wrong End of a Causal Chain, published by silentbob on June 23, 2024 on LessWrong.

There's a very common thing that humans do: a person makes an observation about something they dislike, so they go ahead and make an effort to change that thing. Sometimes it works, and sometimes it doesn't.

If it doesn't work, there can be a variety of reasons for that - maybe the thing is very difficult to change, maybe the person lacks the specific skills to change the thing, maybe it depends on the behavior of other people and the person is not successful in convincing them to act differently. But there's also one failure mode which, while overlapping with the previous ones, is worthy to highlight: imagine the thing the person dislikes is the outcome of a reasonably complex process.

The person observes primarily this outcome, but is partially or fully ignorant of the underlying process that causes the outcome. And they now desperately want the outcome to be different. In such a situation they are practically doomed to fail - in all likelihood, their attempts to change the outcome will not be successful, and even if they are, the underlying cause is still present and will keep pushing in the direction of the undesired outcome.

Three Examples

Productivity in a Company

A software company I worked for once struggled with a slow development cycle, chronic issues with unmet deadlines, and generally shipping things too slowly. The leadership's primary way of addressing this was to repeatedly tell the workforce to "work faster, be more productive, ship things more quickly".

In principle, this approach can work, and to some degree it probably did speed things up. It just requires that the people you're pushing have enough agency, willingness and understanding to take it a step further and take the trip down the causal chain, to figure out what actually needs to happen in order to achieve the desired outcome.

But if middle management just forwards the demand to "ship things more quickly" as is, and the employees below them don't have enough ownership to transform that demand into something more useful, then probably nothing good will happen.

The changed incentives might cause workers to burn themselves out, to cut corners that really shouldn't be cut, to neglect safety or test coverage, to set lower standards for documentation or code quality - aspects that are important for stable long term success, but take time to get right.

To name one very concrete example of the suboptimal consequences this had: The company had sent me a new laptop to replace my old one, which would speed up my productivity quite a bit. But it would have taken a full work day or two to set the new laptop up. The "we need to be faster" situation caused me to constantly have more pressing things to work on, meaning the new, faster laptop sat at the side of my desk, unused, for half a year.

Needless to say, on top of all that, this time was also highly stressful for me and played a big role in me ultimately leaving the company.

Software development, particularly when multiple interdependent teams are involved, is a complex process. The "just ship things more quickly" view however seems to naively suggest that the problem is simply that workers take too long pressing the "ship" button.

What would have been a better approach? It's of course easy to armchair-philosophize my way to a supposedly better solution now. And it's also a bit of a cop-out to make the meta comment that "you need to understand the underlying causal web that causes the company's low velocity".

However, in cases like this one, I think one simple improvement is to make an effort for nuanced communication, making clear that it's not (necessarily) about just "working faster", but rather asking everyone to keep their eyes open for cause...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing the AI for Animals newsletter, published by Max Taylor on June 23, 2024 on The Effective Altruism Forum.

Following the success of the

AI, Animals, and Digital Minds conference earlier this month,

AI for Animals is a new newsletter of resources and updates around the intersection of AI and animals. Each month or so, we'll give a brief overview of a specific topic relating to AI and animals, followed by recent news and other useful resources. While the main focus will be on biological animals, we'll also include resources relevant to digital minds, in recognition of the

feasibility of their development and the

enormous potential ethical implications.

The newsletter will typically comprise the following sections:

1. Overview of a specific AI and animals-related topic

2. Updates

3. Opportunities

4. Recent news

5. Research publications

6. Resources

Our first edition ('

What AI could mean for animals') provides a broad overview of AI's implications for animals. In future editions, we'll hone in on more specific topics, such as the role of AI in intensive animal farming, its potential to accelerate alternative protein development, and its applications for helping animals living in the wild.

Click here if you'd like to subscribe. We're always open to ideas, questions, and feedback - just comment on this post or email us at

[email protected]. Feel free to get in touch any time to suggest topics to cover, news stories and other resources to include, or anything else that could help make this a useful resource!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We should value income doublings equally across time and place (Founders Pledge), published by NicoT on June 23, 2024 on The Effective Altruism Forum.

Hi! I'm Nico and I'm on the research team at Founders Pledge. We noticed that the way we compare current to future income benefits is in tension with how we compare income benefits across interventions. However, aligning these two comparisons - choosing the same function for utility from consumption for both - might lead to large changes in our CEAs. So, we are now thinking about how to choose the right approach.

Since our framework is based on GiveWell's, which is used by other organisations, too, I expect that we're facing the same issues. I'm posting here as a way of thinking out loud and with the hope of getting input from others.

Summary

Founders Pledge and GiveWell both use different values of η (elasticity of marginal utility from consumption) when modelling isoelastic utility from consumption depending on the context. Across interventions, we assume η=1. Over time within an intervention, we assume η1.59. We should choose the same η for both models as having different η values leads us to prefer doubling the incomes of richer people relative to poorer people.

Practically, this inconsistency leads to strange conclusions in existing CEAs. Taking GiveWell's Unlimit Health (deworming) CEA as a stylised example:

For two people in Madagascar, we value doubling the income of someone who makes $2,500 30% as much as for someone who makes $500.

When the person making $2,500 lives in Côte d'Ivoire, however, we value doubling their income the same (100% as much) as for the person in Madagascar who makes $500.

Resolving this isn't straightforward and has large implications for our prioritisation. For example:

Using η=1 everywhere - which implies that income doublings have the same value regardless of absolute income levels - doubles the cost-effectiveness of education and deworming programs and makes economic growth and poverty graduation interventions look substantially better.

Using η=1.87, which is implied by our discount rate, everywhere requires our evaluations to take into account the income levels of recipients and prioritise lower-income regions more. An income doubling in Malawi would be worth roughly 1.9x as much as in Ethiopia, 3.4x as much as in Kenya, 6.4x as much as in Egypt, and 75x as much as in the US. The same is true within countries: in India, an income doubling in Bihar would be worth 3.4x as much as an income doubling in Andhra Pradesh.

I'm hoping this post will start a conversation around what the right value of η is.

Our inconsistent η values

Summary: GiveWell's framework, which we use, explicitly uses log-utility from consumption, which implies isoelastic utility with η=1. However, our (and GiveWell's) discount rate uses η1.59. We use η=1 when comparing between interventions/places. But we use η1.59 for comparisons across time, where income doublings are worth 2.6% less in a year from now solely because incomes will be higher then. We should use the same η for comparisons across time and place.

Not doing so leads us to prefer doubling the incomes of richer vs poorer people (see next section).

We use η=1 (log-utility) to compare the value of income benefits across people or interventions[1]. That assumption is convenient because it allows us to disregard absolute income levels: an income doubling is as valuable from $250 to $500 as it is from $2.5k to $5k.

Because of that, we can make statements like "the value of a 10% income increase from a deworming program in India equals the value of a 10% income increase from a cash transfer program in Kenya" without knowing the incomes of the recipients.

At the same time, we use η1.59 when comparing the value of income benefits in different years within an inte...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Enriched tab is now the default LW Frontpage experience for logged-in users, published by Ruby on June 23, 2024 on LessWrong.

In the past few months, the LessWrong team has been making use of the latest AI tools (given that they unfortunately exist[1]) for art, music, and deciding what we should all be reading.

Our experiments with the latter, i.e. the algorithm that chooses which posts to show on the frontpage, has produced results sufficiently good that at least for now, we're making Enriched the default for logged-in users[2]. If you're logged in and you've never switched tabs before, you'll now be on the Enriched tab. (If you don't have an account, making one takes 10 seconds.)

To recap, here are the currently available tabs (subject to change):

Latest: 100% post from the Latest algorithm (using karma and post age to sort[3])

Enriched (new default): 50% posts from the Latest algorithm, 50% posts from the recommendations engine

Recommended: 100% posts from the recommendations engine, choosing posts specifically for you based on your history

Subscribed: a feed of posts and comments from users you have explicitly followed

Bookmarks: this tab appears if you have bookmarked any posts

Note that posts which are the result of the recommendation engine have a sparkle icon after the title (on desktop, space permitting):

Posts from the last 48 hours have their age bolded:

Why make Enriched the default?

To quote from my earlier post about frontpage recommendation experiments:

A core value of LessWrong is to be timeless and not news-driven. However, the central algorithm by which attention allocation happens on the site is the Hacker News algorithm[2], which basically only shows you things that were posted recently, and creates a strong incentive for discussion to always be centered around the latest content.

This seems very sad to me. When a new user shows up on LessWrong, it seems extremely unlikely that the most important posts for them to read were all written within the last week or two.

I do really like the simplicity and predictability of the Hacker News algorithm. More karma means more visibility, older means less visibility. Very simple. When I vote, I basically know the full effect this has on what is shown to other users or to myself.

But I think the cost of that simplicity has become too high, especially as older content makes up a larger and larger fraction of the best content on the site, and people have been becoming ever more specialized in the research and articles they publish on the site.

We found that a hybrid posts list of 50% Latest and 50% Recommended lets us get the benefits of each algorithm[4].

The Latest component of the list allows people to stay up to date with the most recent new content, provides predictable visibility for new posts, and is approximately universal in that everyone sees those posts which makes posts a bit more common-knowledge-y.

The Recommended component of the list allows us to present content that's predicted to be most interesting/valuable to a user from across thousands of posts from the last 10+ years, not being limited to just recent stuff.

Shifting the age of posts

When we first implemented recommendations, they were very recency biased. My guess is that's because the data we were feeding it was of people reading and voting on recent posts, so it knew those were the ones we liked. In a manner less elegant than I would have prefered, we constrained the algorithm to mostly serving content 30 or 365 days older. You can see the evolution of the recommendation engine, on the age dimension, here:

I give more detailed thoughts about what we found in the course of developing our recommendation algorithm in this comment below.

Feedback, please

Although we're making Enriched the general default, this feature direction is still expe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bed Time Quests & Dinner Games for 3-5 year olds, published by Gunnar Zarncke on June 23, 2024 on LessWrong.

I like these games because they are playful, engage the child and still achieve the objective of getting the child to bed/eat dinner etc. Requires creativity and some slack.

Excerpt from Shohannah's post:

Recently I had the bright idea to give up on being a regular parent. Mostly cause regular parenting practices melt my brain. But then I wondered … does it have to be boring?

[...]

But no. It's all culture and it's all recent culture and you can decide to do Something Else Instead. Really.

So as someone who craves mental stimulation above the pay grade of the 3 to 5 revolutions around the sun my daughters have managed so far … I figured I'd just make up New Rules.

All the time.

So far we've been going for two weeks and the main areas are bedtime routines for my eldest (5) and dinner games for all of us (5, 3, and myself). I noticed I seem to have an easy time generating new and odd rule sets every day, and then started wondering if maybe more parents would enjoy this type of variety in their childcare routines and would want to tap into some of the ideas I've been coming up with. So in case that's you, here is what I've found so far!

[...]

Magic Time

Kiddo is the parent. You are the kiddo. Except, the kiddo is still bringing themselves to bed and not you. They get to tell you what to do and take care of you. You will have to listen. I completely recommend performing a lot of obstructive behavior and misunderstanding basic instructions. This was one of the most popular games and may show some insight into how your child would prefer to be parented, or feels about your parenting.

and fourteen games/rulesets more.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Impartialist Sentientism and Existential Anxiety about Moral Circle Explosion, published by Rafael Ruiz on June 22, 2024 on The Effective Altruism Forum.

Background: I'm currently writing a PhD in moral philosophy on the topic of moral progress and moral circle expansion (you can read a bit about it here and here). I recently was at three EA-related conferences: EA Global London 2024, a workshop on AI, Animals, and Digital Minds at the London School of Economics, and the Oxford Workshop on Global Priorities Research.

Particularly this year, I learned a lot and was mostly shaken up by the topics of invertebrate welfare and AI sentience, which are the focus of this post. I warn you, the ethics are going to get weird.

Introduction.

This is post is a combination of academic philosophy and a more personal reflection on the anxious, dizzying existential crisis I'm facing recently, due to the weight of suffering that I believe might be contained in the universe, which I believe is morally and overwhelmingly important, and for which I feel mostly powerless about. It is a mix of philosophical ideas and a personal, subjective experience trying to digest very radical changes to my view of the world.

The post will be as follows. First, I explain my moral view: classical utilitarianism, which you could also call impartialist sentientism. Then I broaden and weaken the view, and explain that even if you don't subscribe to this moral view, you should still worry about these things. The next section very briefly explains the empirical case for invertebrate and AI sentience.

Although I'm riddled with uncertainty, I think it is likely that some invertebrate species are sentient, and that future AIs could be sentient. This combines in weird ways with moral impartiality and longtermism, since I don't think it matters where and when this happens, they're still morally weighty.

This worry then explodes into moral concern about very weird topics: astronomical numbers of alien invertebrate welfare across the universe, and astronomical numbers of sentient AI developed by either us in the future or aliens, and other very weird topics that I find hard to grasp.

I talk about these topics in a more personal way, particularly ideas of alienation and anxiety over these topics. Then I worry about potential avenues of where this "train to crazy town" is headed, such as: blowing up the universe, extreme moral offsetting by spreading certain invertebrate species across the universe, tiling the universe with hedonium, infinite ethics, and worries about Pascal's Mugging.

Then I take a step back, compose myself, and get back to work. How should these ideas change how we ought to act, in terms of charitable donations, or lines of research and work? What are some actionable points?

Finally, I recommend some readings if you're interested in these topics, since there's a lot of research coming up on this topic. I conclude that the future of normative ethics will be very weird.

Impartialist Sentientism.

Currently, the moral theory I place the highest credence on is total hedonist utilitarianism (what you would call classical, Singerite, or Benthamite utilitarianism), but I believe a lot of these arguments go through even if you aren't, because the amount of value posited by impartialist hedonist utilitarianism tends to trump other considerations, among other things. Just in case you're not up to speed with moral philosophy, let me give a broad background overview on the moral philosophy.

If you're well acquainted, feel free to skip to the next section.

This theory breaks down into the following elements:

By "pain" or "suffering" I mean "any negatively valenced experience", that is, any experience judged to be negative by the subject undergoing it. By "pleasure" or "happiness" I mean "any positively valenced ex...