Audio version available at Cold Takes (or search Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio")

Human civilization is thousands of years old. What's our report card? Whatever we've been doing, has it been "working" to make our lives better than they were before? Or is all our "progress" just helping us be nastier to others and ourselves, such that we need a radical re-envisioning of how the world works?

I'm surprised you've read this far instead of clicking away (thank you). You're probably feeling bored: you've heard the answer (Yes, life is getting better) a zillion times, supported with data from books like Enlightenment Now and websites like Our World in Data and articles like this one and this one.

I'm unsatisfied with this answer, and the reason comes down to the x-axis. Look at any of those sources, and you'll see some charts starting in 1800, many in 1950, some in the 1990s ... and only very few before 1700.1.

This is fine for some purposes: as a retort to alarmism about the world falling apart, perhaps as a defense of the specifically post-Enlightenment period. (And I agree that recent trends are positive.) But I like to take a very long view of our history and future, and I want to know what the trend has been the whole way.

In particular, I'd like to know whether improvement is a very deep, robust pattern - perhaps because life fundamentally tends to get better as our species accumulates ideas, knowledge and abilities - or a potentially unstable fact about the weird, short-lived time we inhabit.

So I'm going to put out several posts trying to answer: what would a chart of "average quality of life for an inhabitant of Earth look like, if we started it all the way back at the dawn of humanity?"

This is a tough and frustrating question to research, because the vast majority of reliable data collection is recent - one needs to do a lot of guesswork about the more distant past. (And I haven't found any comprehensive study or expert consensus on trends in overall quality of life over the long run.) But I've tried to take a good crack at it - to find the data that is relatively straightforward to find, understand its limitations, and form a best-guess bottom line.

In future pieces, I'll go into detail about what I was able to find and what my bottom lines are. But if you just want my high-level, rough take in one chart, here's a chart I made of my subjective guess at average quality of life for humans2 vs. time, from 3 million years ago to today:

Sorry, that wasn't very helpful, because the pre-agriculture period (which we know almost nothing about) was over 100x as long as everything else.3.

(I think it's mildly reality-warping for readers to only ever see charts that are perfectly set up to look sensible and readable. It's good to occasionally see the busted first cut of a chart, which often reveals something interesting in its own right.)

But here's a chart with cumulative population instead of year on the x-axis. The population has exploded over the last few hundred years, so this chart has most of the action going on over the last few hundred years. You can think of this chart as "If we lined up all the people who have ever lived in chronological order, how does their average quality of life change as we pan the camera from the early ones to the later ones?"

In other words:

We don't know much at all about life in the pre-agriculture era. Populations were pretty small, and there likely wasn't much in the way of technological advancement, which might (or might not) mean that different chronological periods weren't super different from each other.5

My impression is that life got noticeably worse with the start of agriculture some thousands of years ago, although I'm certainly not confident in this.

It's very unclear what happened in between the Neolithic Revolution (start of agriculture) and Industrial Revolution a couple hundred years ago.

Life got rapidly better following the Industrial Revolution,...

Audio version available at Cold Takes (or search Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio")

Human civilization is thousands of years old. What's our report card? Whatever we've been doing, has it been "working" to make our lives better than they were before? Or is all our "progress" just helping us be nastier to others and ourselves, such that we need a radical re-envisioning of how the world works?

I'm surprised you've read this far instead of clicking away (thank you). You're probably feeling bored: you've heard the answer (Yes, life is getting better) a zillion times, supported with data from books like Enlightenment Now and websites like Our World in Data and articles like this one and this one.

I'm unsatisfied with this answer, and the reason comes down to the x-axis. Look at any of those sources, and you'll see some charts starting in 1800, many in 1950, some in the 1990s ... and only very few before 1700.1.

This is fine for some purposes: as a retort to alarmism about the world falling apart, perhaps as a defense of the specifically post-Enlightenment period. (And I agree that recent trends are positive.) But I like to take a very long view of our history and future, and I want to know what the trend has been the whole way.

In particular, I'd like to know whether improvement is a very deep, robust pattern - perhaps because life fundamentally tends to get better as our species accumulates ideas, knowledge and abilities - or a potentially unstable fact about the weird, short-lived time we inhabit.

So I'm going to put out several posts trying to answer: what would a chart of "average quality of life for an inhabitant of Earth look like, if we started it all the way back at the dawn of humanity?"

This is a tough and frustrating question to research, because the vast majority of reliable data collection is recent - one needs to do a lot of guesswork about the more distant past. (And I haven't found any comprehensive study or expert consensus on trends in overall quality of life over the long run.) But I've tried to take a good crack at it - to find the data that is relatively straightforward to find, understand its limitations, and form a best-guess bottom line.

In future pieces, I'll go into detail about what I was able to find and what my bottom lines are. But if you just want my high-level, rough take in one chart, here's a chart I made of my subjective guess at average quality of life for humans2 vs. time, from 3 million years ago to today:

Sorry, that wasn't very helpful, because the pre-agriculture period (which we know almost nothing about) was over 100x as long as everything else.3.

(I think it's mildly reality-warping for readers to only ever see charts that are perfectly set up to look sensible and readable. It's good to occasionally see the busted first cut of a chart, which often reveals something interesting in its own right.)

But here's a chart with cumulative population instead of year on the x-axis. The population has exploded over the last few hundred years, so this chart has most of the action going on over the last few hundred years. You can think of this chart as "If we lined up all the people who have ever lived in chronological order, how does their average quality of life change as we pan the camera from the early ones to the later ones?"

In other words:

We don't know much at all about life in the pre-agriculture era. Populations were pretty small, and there likely wasn't much in the way of technological advancement, which might (or might not) mean that different chronological periods weren't super different from each other.5

My impression is that life got noticeably worse with the start of agriculture some thousands of years ago, although I'm certainly not confident in this.

It's very unclear what happened in between the Neolithic Revolution (start of agriculture) and Industrial Revolution a couple hundred years ago.

Life got rapidly better following the Industrial Revolution,...

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

RULESET

Code is available here for those who are interested.

CHARACTER STATS

A character has two stats: an Element and a Power Level.

18 of the 19 characters are power levels 1-6 of the elements Fire, Water and Earth:

The remaining character, the Nullifying Nightmare, has a Power Level of 5 with the unique element of Void.

The NPC team consists of Fire 5, Water 6, Earth 3, Earth 4, Earth 6.

Congratulations here to abstractapplic, who was the first to figure out the elements.

HOW CHARACTERS FIGHT

A fight between two teams is composed of fights between the individual characters. When two characters fight one another, it works as follows:

Some elements counter others:

Fire is countered by Water

Water is countered by Earth

Earth is countered by Fire

If one character's element is countered by the other, that character loses the fight (regardless of power level). For example, if Oil Ooze (Fire 2) fights Greenery Giant (Earth 6), Oil Ooze will win.

If the characters are the same element, the higher-power one will win. For example, if Phoenix Paladin (Fire 5) fights Fire Fox (Fire 3), Phoenix Paladin will win.

If the characters are the same element and the same power, each has a 50% chance to win.

There are two special cases:

The 1 of each element counters the 6 of the same element, and beats it rather than losing to it. So if Volcano Villain (Fire 1) fights Blaze Boy (Fire 6), Volcano Villain will win. (Congratulations to Yonge, who I think is the first person to have explicitly noticed one of these counters).

The Nullifying Nightmare has Power Level 5, and the unique element of Void. Void does not counter any elements, and is not countered by any elements - the Nightmare just fights with Power Level directly, as if it were the same element as its opponent. So it will lose a fight to any Power 6, have a 50% chance against any Power 5, and beat any Power 1-4.

HOW TEAMS FIGHT

To find the outcome of a game between two teams:

Choose a random character from each team.

Those two characters fight.

The loser is KOd and removed from their team.

The winner sticks around.

Repeat this process until one team has run out of characters. That team loses.

This ruleset encourages balanced teams. For example, a team of 5 Fire characters will lose to a team of 4 Earth and 1 Water characters. Even though 4/5 of character matchups favor the Fire characters, nothing on their team can beat the one Water character, and eventually it will work its way through their whole team and win.

Once you understand how the rules work, general strategy is:

Choose high-Power characters.

Try to be reasonably balanced elementally.

Try to counter the enemy team (with 1s against their 6s, a tilt towards the right elements, etc).

The elemental counter mechanic does not always work as you might expect it to at the team level - rather than thinking about countering your opponent's common elements, you should think of it as targeting your opponent's weak elements. If your opponent has 3 Earth and 2 Water characters, for example, rather than saying 'that team has lots of Earth characters, I should bring high-power Fire characters to beat them' (which will get you wiped out by the Water characters), you want to say 'that team has no Fire characters, I should bring high-power Earth characters it won't be able to beat'.

DATASET GENERATION

The games you have access to were played by players grouped together by the games auto-party functionality. These players do not build coordinated teams, so there's no correlation between characters.

However, some characters are more popular than others (most notably the Siren Sorceress, whose infamous costume has won her a large and...enthusiastic...fan base).

Overall, Water characters are somewhat more common and Earth characters somewhat less common. This doesn't affect the game itself, but it...

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

RULESET

Code is available here for those who are interested.

CHARACTER STATS

A character has two stats: an Element and a Power Level.

18 of the 19 characters are power levels 1-6 of the elements Fire, Water and Earth:

The remaining character, the Nullifying Nightmare, has a Power Level of 5 with the unique element of Void.

The NPC team consists of Fire 5, Water 6, Earth 3, Earth 4, Earth 6.

Congratulations here to abstractapplic, who was the first to figure out the elements.

HOW CHARACTERS FIGHT

A fight between two teams is composed of fights between the individual characters. When two characters fight one another, it works as follows:

Some elements counter others:

Fire is countered by Water

Water is countered by Earth

Earth is countered by Fire

If one character's element is countered by the other, that character loses the fight (regardless of power level). For example, if Oil Ooze (Fire 2) fights Greenery Giant (Earth 6), Oil Ooze will win.

If the characters are the same element, the higher-power one will win. For example, if Phoenix Paladin (Fire 5) fights Fire Fox (Fire 3), Phoenix Paladin will win.

If the characters are the same element and the same power, each has a 50% chance to win.

There are two special cases:

The 1 of each element counters the 6 of the same element, and beats it rather than losing to it. So if Volcano Villain (Fire 1) fights Blaze Boy (Fire 6), Volcano Villain will win. (Congratulations to Yonge, who I think is the first person to have explicitly noticed one of these counters).

The Nullifying Nightmare has Power Level 5, and the unique element of Void. Void does not counter any elements, and is not countered by any elements - the Nightmare just fights with Power Level directly, as if it were the same element as its opponent. So it will lose a fight to any Power 6, have a 50% chance against any Power 5, and beat any Power 1-4.

HOW TEAMS FIGHT

To find the outcome of a game between two teams:

Choose a random character from each team.

Those two characters fight.

The loser is KOd and removed from their team.

The winner sticks around.

Repeat this process until one team has run out of characters. That team loses.

This ruleset encourages balanced teams. For example, a team of 5 Fire characters will lose to a team of 4 Earth and 1 Water characters. Even though 4/5 of character matchups favor the Fire characters, nothing on their team can beat the one Water character, and eventually it will work its way through their whole team and win.

Once you understand how the rules work, general strategy is:

Choose high-Power characters.

Try to be reasonably balanced elementally.

Try to counter the enemy team (with 1s against their 6s, a tilt towards the right elements, etc).

The elemental counter mechanic does not always work as you might expect it to at the team level - rather than thinking about countering your opponent's common elements, you should think of it as targeting your opponent's weak elements. If your opponent has 3 Earth and 2 Water characters, for example, rather than saying 'that team has lots of Earth characters, I should bring high-power Fire characters to beat them' (which will get you wiped out by the Water characters), you want to say 'that team has no Fire characters, I should bring high-power Earth characters it won't be able to beat'.

DATASET GENERATION

The games you have access to were played by players grouped together by the games auto-party functionality. These players do not build coordinated teams, so there's no correlation between characters.

However, some characters are more popular than others (most notably the Siren Sorceress, whose infamous costume has won her a large and...enthusiastic...fan base).

Overall, Water characters are somewhat more common and Earth characters somewhat less common. This doesn't affect the game itself, but it...

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

Force model use and then detect it

Parent project: this is a subproject of the value learning project.

Background

I've seen human values residing, at least in part, in our mental models. We have a mental model of what might happen in the world, and we grade these outcomes as good or bad. In order to learn what humans value, the AI needs to be able to access the mental models underlying our thought processes.

Before starting on humans, with our messy brains, it might be better to start on artificial agents, especially neural-net based ones that superficially resemble ourselves.

The problem is that deep learning RL agents are generally model-free. Or, when they are model-based, they are generally constructed with a model explicitly, so that identifying their model is as simple as saying "the model is in this sub-module, the one labelled 'model'."

Setup

The idea here is to force a neural net to construct a model within itself - a model that we can somewhat understand.

I can think of several ways of doing that. We could get a traditional deep learning agent that performs on a game. But we might also force it to answer questions about various aspects of the game, identifying the values of certain features we have specified in advance ("how many spaceships are there on the screen currently?"). We can then use multi-objective optimisation with a strong simplicity prior/regulariser. This may force the agent to use the categories it has constructed to answer the questions, in order to play the game.

Or we could be more direct. We could, for instance, have the neural net pass on instructions or advice to another entity that actually plays the game. The neural net sees the game state, but the other entity can only react in terms of the features we've laid down. So the neural net has to translate the game state into the features (this superficially looks like an autoencoder; those might be another way of achieving the aim).

Ideally, we may discover ways of forcing an agent to use a model without specifying the model ourselves; some approaches to transfer learning may work here, and it's possible that GPT-3 and other transformer-based architectures already generate something that could be called an "internal model".

Then, we go looking for that model within the agent. Here the idea is to use something like the OpenAI microscope. That approach allows people to visualise what each neuron in an image classifier is reacting to, and how the classifier is doing its job. Similarly, we'd want to identify where the model resides, how it's encoded and accessed, and similar questions. We can then modify the agent's architecture to test if these characteristics are general, or particular to the agent's specific design.

Research aims

See how feasible it is to force a neural net based RL agent to construct mental models.

See how easy it is to identify these mental models within the neural net, and what characteristics they have (are they spread out, are they tightly localised, how stable are they, do they get reused for other purposes?).

See how the results of the first two aims might lead to more research, or might be applied to AI-human interactions directly.

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

Force model use and then detect it

Parent project: this is a subproject of the value learning project.

Background

I've seen human values residing, at least in part, in our mental models. We have a mental model of what might happen in the world, and we grade these outcomes as good or bad. In order to learn what humans value, the AI needs to be able to access the mental models underlying our thought processes.

Before starting on humans, with our messy brains, it might be better to start on artificial agents, especially neural-net based ones that superficially resemble ourselves.

The problem is that deep learning RL agents are generally model-free. Or, when they are model-based, they are generally constructed with a model explicitly, so that identifying their model is as simple as saying "the model is in this sub-module, the one labelled 'model'."

Setup

The idea here is to force a neural net to construct a model within itself - a model that we can somewhat understand.

I can think of several ways of doing that. We could get a traditional deep learning agent that performs on a game. But we might also force it to answer questions about various aspects of the game, identifying the values of certain features we have specified in advance ("how many spaceships are there on the screen currently?"). We can then use multi-objective optimisation with a strong simplicity prior/regulariser. This may force the agent to use the categories it has constructed to answer the questions, in order to play the game.

Or we could be more direct. We could, for instance, have the neural net pass on instructions or advice to another entity that actually plays the game. The neural net sees the game state, but the other entity can only react in terms of the features we've laid down. So the neural net has to translate the game state into the features (this superficially looks like an autoencoder; those might be another way of achieving the aim).

Ideally, we may discover ways of forcing an agent to use a model without specifying the model ourselves; some approaches to transfer learning may work here, and it's possible that GPT-3 and other transformer-based architectures already generate something that could be called an "internal model".

Then, we go looking for that model within the agent. Here the idea is to use something like the OpenAI microscope. That approach allows people to visualise what each neuron in an image classifier is reacting to, and how the classifier is doing its job. Similarly, we'd want to identify where the model resides, how it's encoded and accessed, and similar questions. We can then modify the agent's architecture to test if these characteristics are general, or particular to the agent's specific design.

Research aims

See how feasible it is to force a neural net based RL agent to construct mental models.

See how easy it is to identify these mental models within the neural net, and what characteristics they have (are they spread out, are they tightly localised, how stable are they, do they get reused for other purposes?).

See how the results of the first two aims might lead to more research, or might be applied to AI-human interactions directly.

Our Tundra is an inhospitable[1] environment. The only significant food available to herbivores is Lichen, which has a tiny nutritional value of 1. The Tundra is cold too. Staying warm requires the cold tolerance adaptation, which costs +2 size.

An organism must expend 20% of its energy just to survive. A herbivore foraging for lichen cannot have a size greater than 5 or else it will expend more energy in metabolism than it is possible to acquire from eating Lichen.

All organisms have base size 0.1. The cold adaptation (+2) plus the Lichen digestive tract (+1) costs a total of +3 size. A Tundra herbivore has a minimum size of 3.1. A herbivore with size 5.1 is untenable since it expends more energy (1.02) than is possible to obtain from Lichen (1.00).

Players submitted 39 species native to the Tundra. Only 4 of them were viable herbivores: Micropas, Arctic Slug, Northern Nibbler and "lichen" (not to be confused with the foragable "Lichen"). (Multicore's Arctic Fox was a carnivore.)

These species could support little in the way of weapons, armor and speed. They were defenseless. In the first 8 turns, all four of our viable foragers are eaten to extinction.

After the viable herbivores were eliminated, total ecological collapse was inevitable.

The Frostwing Snipper

An honorable mention goes to Nem's Frostwing Snipper, a Speed 10 species that could digest both Lichen and Seeds. The maximum speed made the Frostwing Snipper immune to predation which let it survive the initial carnage. The ability to digest seeds meant that the Frostwing Snipper did consume enough energy on average to more than replace itself.

However, "on average" is not enough. The Tundra's carrying capacity of Frostwing Snippers was too small. Random fluctuations eventually knocked the Frostwing Snipper into extinction.

Winners

None

Eratta

Simon notes that I used 50,000 instead of 1,000 for each species initial energy. Using the correct value of 1,000 doesn't really change anything except everyone dies faster and MatIsk's Porostozer Mamuci (which eats Carrion) takes the place of Frostwing Snipper.

Our Tundra is an inhospitable[1] environment. The only significant food available to herbivores is Lichen, which has a tiny nutritional value of 1. The Tundra is cold too. Staying warm requires the cold tolerance adaptation, which costs +2 size.

An organism must expend 20% of its energy just to survive. A herbivore foraging for lichen cannot have a size greater than 5 or else it will expend more energy in metabolism than it is possible to acquire from eating Lichen.

All organisms have base size 0.1. The cold adaptation (+2) plus the Lichen digestive tract (+1) costs a total of +3 size. A Tundra herbivore has a minimum size of 3.1. A herbivore with size 5.1 is untenable since it expends more energy (1.02) than is possible to obtain from Lichen (1.00).

Players submitted 39 species native to the Tundra. Only 4 of them were viable herbivores: Micropas, Arctic Slug, Northern Nibbler and "lichen" (not to be confused with the foragable "Lichen"). (Multicore's Arctic Fox was a carnivore.)

These species could support little in the way of weapons, armor and speed. They were defenseless. In the first 8 turns, all four of our viable foragers are eaten to extinction.

After the viable herbivores were eliminated, total ecological collapse was inevitable.

The Frostwing Snipper

An honorable mention goes to Nem's Frostwing Snipper, a Speed 10 species that could digest both Lichen and Seeds. The maximum speed made the Frostwing Snipper immune to predation which let it survive the initial carnage. The ability to digest seeds meant that the Frostwing Snipper did consume enough energy on average to more than replace itself.

However, "on average" is not enough. The Tundra's carrying capacity of Frostwing Snippers was too small. Random fluctuations eventually knocked the Frostwing Snipper into extinction.

Winners

None

Eratta

Simon notes that I used 50,000 instead of 1,000 for each species initial energy. Using the correct value of 1,000 doesn't really change anything except everyone dies faster and MatIsk's Porostozer Mamuci (which eats Carrion) takes the place of Frostwing Snipper.

Published on September 29, 2021 3:47 PM GMT

Many thanks to Lauren Mee, David Reinstein, Brenton Mayer, Aaron Gertler, Alex Holness-Tofts, Lynn Tan, Vaidehi Agarwalla, David Moss, and Renee Bell for providing feedback on drafts of this writeup, as well as all who provided feedback on the studies themselves.

Summary

Animal Advocacy Careers (AAC) ran two longitudinal studies aiming to compare and test the cost-effectiveness of our one-to-one advising calls and our online course. Various forms of these two types of careers advice service have been used by people seeking to build the effective altruism (EA) movement for years, and we expect the results to be informative to EA movement builders, as well as to AAC.

We interpret the results as tentative evidence of positive effects from both services, but the effects of each seem to be different. Which is more effective overall depends on your views about which sorts of effects are most important; our guess is that one-to-one calls are slightly more effective per participant, but not by much. One-to-one calls seem substantially more costly per participant, which makes the service harder to scale.

There therefore seems to be a tradeoff between costs and apparent effects per participant. We’d guess that the online course was (and will be, once scaled up) slightly more cost-effective, all things considered, but the services might just serve different purposes, especially since the applicants might be different for the different services.

Background

Animal Advocacy Careers (AAC) ran a longitudinal study testing the effects of our ~1 hour one-to-one careers advising calls, which operated in a similar style to calls given by 80,000 Hours and the organisers of local effective altruism (EA) groups across the world. Over roughly the same time period, we ran a second study using very similar methodology that tested the effects of our ~9 week online course, which taught some core content about effective animal advocacy, effective altruism, and impact-focused career strategy and culminated in support to develop a career plan, either via a group workshop or by redirecting to planning materials by 80,000 Hours.

Each study was designed as a randomised controlled trial,[1] and pre-registered on the Open Science Framework (here and here), although a few methodological difficulties mean that we shouldn’t interpret the results as giving very conclusive answers. Despite these difficulties, we think that the studies provide useful evidence both for AAC and others focusing on building the effective altruism movement (i.e. the community striving to help others as much as possible using the best evidence available) to help us prioritise our time and resources. We’ll be sharing more about the methodological lessons from the studies in a forthcoming post called “EA movement building: Should you run an experiment?”

The findings are also written up in the style of a formal academic paper, viewable here. That version provides more detail on the methodology (participants, procedure, and instruments) and contains extensive appendices (predictions, full results, anonymised raw data, R code, and more). In the rest of this post, we summarise some of the key results and takeaways.

Which service has larger effects?

The ideal evaluation of whether a career advice intervention genuinely increases a participant’s expected impact for altruistic causes would be very challenging and expensive.[2] So instead, we designed and collected data on four metrics that we expected to be useful indicators of whether people were making changes in promising directions:

“Attitudes,” e.g. views on cause prioritisation, inclination towards effective altruism.

“Career plans,” e.g. study plans, internship plans, job plans, and long-term plans.

“Career-related behaviours,” e.g. secured a new role, applied to one or more positions or programmes, joined the effective animal advocacy directory.

“Self-assessed expected impact for altruistic caus...

Published on September 29, 2021 3:47 PM GMT

Many thanks to Lauren Mee, David Reinstein, Brenton Mayer, Aaron Gertler, Alex Holness-Tofts, Lynn Tan, Vaidehi Agarwalla, David Moss, and Renee Bell for providing feedback on drafts of this writeup, as well as all who provided feedback on the studies themselves.

Summary

Animal Advocacy Careers (AAC) ran two longitudinal studies aiming to compare and test the cost-effectiveness of our one-to-one advising calls and our online course. Various forms of these two types of careers advice service have been used by people seeking to build the effective altruism (EA) movement for years, and we expect the results to be informative to EA movement builders, as well as to AAC.

We interpret the results as tentative evidence of positive effects from both services, but the effects of each seem to be different. Which is more effective overall depends on your views about which sorts of effects are most important; our guess is that one-to-one calls are slightly more effective per participant, but not by much. One-to-one calls seem substantially more costly per participant, which makes the service harder to scale.

There therefore seems to be a tradeoff between costs and apparent effects per participant. We’d guess that the online course was (and will be, once scaled up) slightly more cost-effective, all things considered, but the services might just serve different purposes, especially since the applicants might be different for the different services.

Background

Animal Advocacy Careers (AAC) ran a longitudinal study testing the effects of our ~1 hour one-to-one careers advising calls, which operated in a similar style to calls given by 80,000 Hours and the organisers of local effective altruism (EA) groups across the world. Over roughly the same time period, we ran a second study using very similar methodology that tested the effects of our ~9 week online course, which taught some core content about effective animal advocacy, effective altruism, and impact-focused career strategy and culminated in support to develop a career plan, either via a group workshop or by redirecting to planning materials by 80,000 Hours.

Each study was designed as a randomised controlled trial,[1] and pre-registered on the Open Science Framework (here and here), although a few methodological difficulties mean that we shouldn’t interpret the results as giving very conclusive answers. Despite these difficulties, we think that the studies provide useful evidence both for AAC and others focusing on building the effective altruism movement (i.e. the community striving to help others as much as possible using the best evidence available) to help us prioritise our time and resources. We’ll be sharing more about the methodological lessons from the studies in a forthcoming post called “EA movement building: Should you run an experiment?”

The findings are also written up in the style of a formal academic paper, viewable here. That version provides more detail on the methodology (participants, procedure, and instruments) and contains extensive appendices (predictions, full results, anonymised raw data, R code, and more). In the rest of this post, we summarise some of the key results and takeaways.

Which service has larger effects?

The ideal evaluation of whether a career advice intervention genuinely increases a participant’s expected impact for altruistic causes would be very challenging and expensive.[2] So instead, we designed and collected data on four metrics that we expected to be useful indicators of whether people were making changes in promising directions:

“Attitudes,” e.g. views on cause prioritisation, inclination towards effective altruism.

“Career plans,” e.g. study plans, internship plans, job plans, and long-term plans.

“Career-related behaviours,” e.g. secured a new role, applied to one or more positions or programmes, joined the effective animal advocacy directory.

“Self-assessed expected impact for altruistic caus...