Published on September 14, 2021 8:43 PM GMT

Epistemic status: Half-baked at best

I have often been skeptical of the value of a) critiques against effective altruism and b) fully general arguments that seem like they can apply to almost anything. However, as I am also a staunch defender of hypocrisy, I will now hypocritically attempt to make the case for applying a fully general critique to effective altruism.

In this post, I will claim that:

Motivated reasoning inhibits our ability to acquire knowledge and form reasoned opinions.

Effective altruism should not be assumed to be above these biases. Moreover, there are strong reasons to believe that incentive structures and institutions in effective altruism exacerbate rather than alleviate these biases.

Observed data and experiences in effective altruism support this theory; they are consistent with an environment where motivated reasoning and selection biases are rampant.

To the extent that these biases (related to motivated reasoning) are real, we should expect the harm done to our ability to form reasoned opinions to also seriously harm the project of doing good.

I will use the example of cost-effectiveness analyses as a jumping board for this argument. (I understand that effective altruism, especially outside of global health and development, has largely moved away from explicit expected value calculations and cost-effectiveness analyses. However, I do not believe this change invalidates my argument (see Appendix B)).

I also list a number of tentative ways to counteract motivated reasoning and selection bias in effective altruism:

Encourage and train scientific/general skepticism in EA newcomers.

Try marginally harder to accept newcomers, particularly altruistically motivated ones with extremely high epistemic standards

As a community, fund and socially support external (critical) cost-effectiveness analyses and impact assessments of EA orgs

Within EA orgs, encourage and reward dissent of various forms

Commit to individual rationality and attempts to reduce motivated reasoning

Maybe encourage a greater number of people to apply and seriously consider jobs outside of EA or EA-adjacent orgs

Maintain or improve the current culture of relatively open, frequent, and vigorous debate

Foster a bias towards having open, public discussions of important concepts, strategies, and intellectual advances

Motivated reasoning: What it is, why it’s common, why it matters

By motivated reasoning, I roughly mean what Julia Galef calls “soldier mindset” (H/T Rob Bensinger):

In directionally motivated reasoning, often shortened to "motivated reasoning", we disproportionately put our effort into finding evidence/reasons that support what we wish were true.

Or, from Wikipedia:

emotionally biased reasoning to produce justifications or make decisions that are most desired rather than those that accurately reflect the evidence

I think motivated reasoning is really common in our world. As I said in a recent comment:

My impression is that my interactions with approximately every entity that perceives themself as directly doing good outside of EA is that they are not seeking truth, and this systematically corrupts them in important ways. Non-random examples that come to mind include public health (on covid, vaping, nutrition), bioethics, social psychology, developmental econ, climate change, vegan advocacy, religion, US Democratic party, and diversity/inclusion. Moreover, these problems aren't limited to particular institutions: these problems are instantiated in academia, activist groups, media, regulatory groups and "mission-oriented" companies.

What does motivated reasoning look like in practice? In the field of cost-effectiveness analyses, it might look like this comment on a blog post about scientific conflicts of interest:

Back in the 90’s I did some consulting work for a startup that was developing a new medical device. They were honest people–they never pressured me. My contract stipulated that ...

Published on September 3, 2021 9:58 AM GMT

In short: There are many methods to pool forecasts. The most commonly used is the arithmetic mean of probabilities. However, there are empirical and theoretical reasons to prefer the geometric mean of the odds instead. This is particularly important when some of the predictions have extreme values. Therefore, I recommend defaulting to the geometric mean of odds to aggregate probabilities.

Epistemic status: My impression is that geometric mean of odds is the preferred pooling method among most researchers who have looked into the topic. That being said, I only cite one study with direct empirical evidence supporting my recommendation.

One key piece of advice I would give to people keen on forming better opinions of the world is to pay attention to many experts, and to reason things through a few times using different assumptions. In the context of quantitative forecasting, this results in many predictions, that together paint a more complete picture.

But, how can we aggregate the different predictions? Ideally, we would like a simple heuristic that pools the many experts and models we have considered and produce an aggregate prediction [1].

There are many possible choices for such a heuristic. A common one is to take the arithmetic mean of the individual predictions :

We see an example of this approach in this article from Luisa Rodriguez, where it is used to aggregate some predictions about the chances of a nuclear war in a year.

A different heuristic, which I will argue in favor of, is to take the geometric mean of the odds:

Whereas the arithmetic mean adds the values together and divides by the number of values, the geometric mean multiplies all the values and then takes the N-th root of the product (where N = number of values).

And the odds equal the probability of an event divided by its complement, o=p/(1−p).[2]

For example, in Rodriguez's article we have four predictions from different sources [3]:

Rodriguez takes as an aggregate the arithmetic mean of the probabilities [1.40%+2.21%+0.39%+0.40%]/4=1.10%, which corresponds to pooled odds of about 1:91. If we take the geometric mean of the odds instead we will end with pooled odds of (1:70⋅1:44⋅1:255⋅1:249)1/4≈1:118, which corresponds to a pooled probability of about 0.84%.

In the remainder of the article I will argue that the geometric mean of odds is both empirically more accurate and has some compelling theoretical properties. In practice, I believe we should largely prefer to aggregate probabilities using the geometric mean of odds.

The (extremized) geometric mean of odds empirically results in more accurate predictions

(Satopää et al, 2014) empirically explores different aggregation methods, including average probabilities, the median probability, and the geometric mean of the odds - as well as some more complex methods of aggregating forecasts like and extremized version of the geometric mean of the odds and the beta transformed linear opinion pool. They aggregate responses from 1300 forecasters over 69 questions on geopolitics [4].

In summary, they find that the extremized geometric mean of odds performs best in terms of the Brier score of its predictions. The non-extremized geometric mean of odds robustly outperforms the arithmetic mean of probabilities and the median, though it performs worse than some of the more complex methods.

We haven't quite explained what extremizing is, but it involves raising the pooled odds to a power a>1 [5]:

In their dataset, the extremizing parameter a that attains the best Brier score falls between 1.161 and 3.921. As a handy heuristic, when extremizing I suggest using a power of 2.5 in practice. My intuition is that extremizing makes most sense when aggregating data from underconfident experts, and I would not use it when aggregating personal predictions derived from different approaches. This is because extremizing is meant to be a correction for forecaster underconfidence [6]. That bei...

Published on September 3, 2021 9:58 AM GMT

In short: There are many methods to pool forecasts. The most commonly used is the arithmetic mean of probabilities. However, there are empirical and theoretical reasons to prefer the geometric mean of the odds instead. This is particularly important when some of the predictions have extreme values. Therefore, I recommend defaulting to the geometric mean of odds to aggregate probabilities.

Epistemic status: My impression is that geometric mean of odds is the preferred pooling method among most researchers who have looked into the topic. That being said, I only cite one study with direct empirical evidence supporting my recommendation.

One key piece of advice I would give to people keen on forming better opinions of the world is to pay attention to many experts, and to reason things through a few times using different assumptions. In the context of quantitative forecasting, this results in many predictions, that together paint a more complete picture.

But, how can we aggregate the different predictions? Ideally, we would like a simple heuristic that pools the many experts and models we have considered and produce an aggregate prediction [1].

There are many possible choices for such a heuristic. A common one is to take the arithmetic mean of the individual predictions :

We see an example of this approach in this article from Luisa Rodriguez, where it is used to aggregate some predictions about the chances of a nuclear war in a year.

A different heuristic, which I will argue in favor of, is to take the geometric mean of the odds:

Whereas the arithmetic mean adds the values together and divides by the number of values, the geometric mean multiplies all the values and then takes the N-th root of the product (where N = number of values).

And the odds equal the probability of an event divided by its complement, o=p/(1−p).[2]

For example, in Rodriguez's article we have four predictions from different sources [3]:

Rodriguez takes as an aggregate the arithmetic mean of the probabilities [1.40%+2.21%+0.39%+0.40%]/4=1.10%, which corresponds to pooled odds of about 1:91. If we take the geometric mean of the odds instead we will end with pooled odds of (1:70⋅1:44⋅1:255⋅1:249)1/4≈1:118, which corresponds to a pooled probability of about 0.84%.

In the remainder of the article I will argue that the geometric mean of odds is both empirically more accurate and has some compelling theoretical properties. In practice, I believe we should largely prefer to aggregate probabilities using the geometric mean of odds.

The (extremized) geometric mean of odds empirically results in more accurate predictions

(Satopää et al, 2014) empirically explores different aggregation methods, including average probabilities, the median probability, and the geometric mean of the odds - as well as some more complex methods of aggregating forecasts like and extremized version of the geometric mean of the odds and the beta transformed linear opinion pool. They aggregate responses from 1300 forecasters over 69 questions on geopolitics [4].

In summary, they find that the extremized geometric mean of odds performs best in terms of the Brier score of its predictions. The non-extremized geometric mean of odds robustly outperforms the arithmetic mean of probabilities and the median, though it performs worse than some of the more complex methods.

We haven't quite explained what extremizing is, but it involves raising the pooled odds to a power a>1 [5]:

In their dataset, the extremizing parameter a that attains the best Brier score falls between 1.161 and 3.921. As a handy heuristic, when extremizing I suggest using a power of 2.5 in practice. My intuition is that extremizing makes most sense when aggregating data from underconfident experts, and I would not use it when aggregating personal predictions derived from different approaches. This is because extremizing is meant to be a correction for forecaster underconfidence [6]. That bei...

Published on September 12, 2021 9:15 PM GMT

Good stories are a key part of how EA has grown since its beginning. Three examples:

Harry Potter and the Methods of Rationality, which introduced many readers to important ideas about good epistemics and AI risk

500 Million, But Not A Single One More, which has been read aloud in conference halls for hundreds of people and is a part of the EA Handbook (it’s also “creative nonfiction”, but it still counts as a good story)

The Fable of the Dragon-Tyrant, which seems to have been extremely influential for one of the world’s most prominent entrepreneurs (who has since given tens of millions of dollars to various EA-adjacent causes)

There’s a lot of “rational fiction” out there — stories about people thinking clearly to solve problems. Many of those stories also incorporate EA themes. But they tend to reveal their ideas over dozens of chapters, making it hard for someone to pick up on those themes unless they’re willing to dedicate many hours of time.

We’d like to see creative work that “gets to the point” quickly — stories that, in a single sitting, might inspire someone to find out more about effective altruism, whether that means the whole movement or a single idea/cause area/intervention.

So we’re running a contest! We want to see you write or share stories and creative nonfiction with EA themes. And we’ve added prizes to sweeten the deal.[1]

Notably, your work doesn’t have to use EA jargon or cover a popular cause area, as long as it gets across the core idea of "using evidence and reason to help others effectively".

That said, it doesn't hurt if the work references popular EA topics in some way, or tries to directly inspire readers to find out more about EA. For example, HPMOR includes a note along the lines of “to learn what Harry knows, read the LessWrong Sequences”. We’d be happy to see stories that would justify the note “to learn what X knows, join a Virtual Program”.[2]

What kinds of content can I submit?

We’ll have two categories:

Fictional stories, like “The Fable of the Dragon-Tyrant”

Creative nonfiction, like “500 Million, But Not A Single One More”

No need to include the category in your submission.

What are the prizes?

We’re offering $10,000 in total prize money. Each of the two categories will have a $2,500 top prize. We’ll also choose five honorable mentions, for a $1,000 prize each.

Honorable mentions can come from either category, in case one category has many more submissions than the other.

What’s the deadline?

Entries must be published on the EA Forum no later than Friday, October 29 (initial deadline extended by two weeks after a few people said it seemed short).

If you’d like to be kind to the judges and let us space out our reading over time, you can publish earlier :-)

How do I submit content?

All stories must be published on the EA Forum and tagged with Creative Writing Contest.

We want lots of people to read and discuss your submissions — we think the Forum will be a really fun place if good stories start showing up. However, we won’t use upvotes or comments as part of our process for choosing a winner.

If you’re wary of sharing your work in public, remember that winning the contest guarantees your work being shared in public (with many, many people). That said, you are welcome to use a pseudonym if you’d prefer!

If you’re submitting someone else’s work, please crosspost it onto the Forum as a full post. Not just a linkpost — people should be able to stay on the Forum as they read submissions.

The exception is writing covered by copyright. In that case, we still encourage you to add an excerpt in addition to a link; in our experience, this makes people more likely to read.

If the work is already on the Forum, add the Creative Writing Contest tag and send me a message to let me know you’ve done so.

Will CEA make use of the winning entries in some way?

We don’t have any concrete plans yet. But it’s very likely that we’ll share some of the best submissions ...

Published on September 12, 2021 9:15 PM GMT

Good stories are a key part of how EA has grown since its beginning. Three examples:

Harry Potter and the Methods of Rationality, which introduced many readers to important ideas about good epistemics and AI risk

500 Million, But Not A Single One More, which has been read aloud in conference halls for hundreds of people and is a part of the EA Handbook (it’s also “creative nonfiction”, but it still counts as a good story)

The Fable of the Dragon-Tyrant, which seems to have been extremely influential for one of the world’s most prominent entrepreneurs (who has since given tens of millions of dollars to various EA-adjacent causes)

There’s a lot of “rational fiction” out there — stories about people thinking clearly to solve problems. Many of those stories also incorporate EA themes. But they tend to reveal their ideas over dozens of chapters, making it hard for someone to pick up on those themes unless they’re willing to dedicate many hours of time.

We’d like to see creative work that “gets to the point” quickly — stories that, in a single sitting, might inspire someone to find out more about effective altruism, whether that means the whole movement or a single idea/cause area/intervention.

So we’re running a contest! We want to see you write or share stories and creative nonfiction with EA themes. And we’ve added prizes to sweeten the deal.[1]

Notably, your work doesn’t have to use EA jargon or cover a popular cause area, as long as it gets across the core idea of "using evidence and reason to help others effectively".

That said, it doesn't hurt if the work references popular EA topics in some way, or tries to directly inspire readers to find out more about EA. For example, HPMOR includes a note along the lines of “to learn what Harry knows, read the LessWrong Sequences”. We’d be happy to see stories that would justify the note “to learn what X knows, join a Virtual Program”.[2]

What kinds of content can I submit?

We’ll have two categories:

Fictional stories, like “The Fable of the Dragon-Tyrant”

Creative nonfiction, like “500 Million, But Not A Single One More”

No need to include the category in your submission.

What are the prizes?

We’re offering $10,000 in total prize money. Each of the two categories will have a $2,500 top prize. We’ll also choose five honorable mentions, for a $1,000 prize each.

Honorable mentions can come from either category, in case one category has many more submissions than the other.

What’s the deadline?

Entries must be published on the EA Forum no later than Friday, October 29 (initial deadline extended by two weeks after a few people said it seemed short).

If you’d like to be kind to the judges and let us space out our reading over time, you can publish earlier :-)

How do I submit content?

All stories must be published on the EA Forum and tagged with Creative Writing Contest.

We want lots of people to read and discuss your submissions — we think the Forum will be a really fun place if good stories start showing up. However, we won’t use upvotes or comments as part of our process for choosing a winner.

If you’re wary of sharing your work in public, remember that winning the contest guarantees your work being shared in public (with many, many people). That said, you are welcome to use a pseudonym if you’d prefer!

If you’re submitting someone else’s work, please crosspost it onto the Forum as a full post. Not just a linkpost — people should be able to stay on the Forum as they read submissions.

The exception is writing covered by copyright. In that case, we still encourage you to add an excerpt in addition to a link; in our experience, this makes people more likely to read.

If the work is already on the Forum, add the Creative Writing Contest tag and send me a message to let me know you’ve done so.

Will CEA make use of the winning entries in some way?

We don’t have any concrete plans yet. But it’s very likely that we’ll share some of the best submissions ...

Published on September 11, 2021 10:05 PM GMT

There’s lots of cool data floating around in EA: grant databases, survey results, growth metrics, etc. I’m a data scientist and enjoy data visualisation, so thought it would be a fun project to build a website which aggregates EA data into interactive plots.

The website is now live at EffectiveAltruismData.com. Source code is available on Github.

This project is still a work in progress: the data is pretty out of date and I’ve got lots of future work planned. But it’s far enough along that I’d like some public feedback.

The website is responsive and should look good on any desktop or tablet screen. If you’re viewing on a phone it will probably look ok in landscape mode.

Here are a few screenshots:

Implementation Details

The website is mostly coded in Python. The main libraries I used were:

Pandas for data handling.

Plotly for creating the interactive plots.

Dash for the web framework.

I also wrote a bunch of vanilla CSS for the frontend styling.

The web server is currently deployed with Heroku, which costs $7/month.

I have vague ambitions to re-implement the frontend with D3.js or Chart.js. This should cut down the loading time and give me more control over how the visualisations work.

Design Philosophy

I aimed to follow the data visualisation principles laid out in Information Dashboard Design and Storytelling with Data. These include:

Minimise the “ink-to-data” (or “pixel-to-data”) ratio to avoid distracting clutter.

Encoding data in length or distance is much higher fidelity than area or angle.

Avoid pie charts, stacked area plots, radar charts, violin plots.

Stick to bar charts, scatter plots, and line graphs as much as possible.

Don’t make the reader rotate their head.

Use horizontal bar charts rather than vertical ones.

Minimise the total length the eye has to travel to take in all the data.

Avoid legends.

On line graphs, put the labels directly on the ends of the lines.

Why I Did This

I said earlier that this project was motivated by my fancy for data visualisation. But I do think there’s a lot of scope for valuable data visualisation and data wrangling work within Effective Altruism.

For example, I’ve found it difficult to get a sense of the scale of donations within EA. Are total donations basically Open Philanthropy plus a rounding error? Or do donations from all the little guys like me actually make a difference in the big picture?

This isn’t just an interesting question in itself: it also informs my life decisions. If the total donations of people in my reference class is enough to make a noticeable change to AMF funding, then I’m more likely to steadily earn-to-give on a moderately affluent career path. If my reference class is totally overwhelmed by a handful of mega-donors, then I’m more likely to drop everything and spend a year figuring out if I can contribute to AI safety.

This was some of the motivation behind the first panel of EffectiveAltruismData.com. Ultimately, I’d like to have a plot which puts all the major stocks and flows of EA money on a common scale and puts my personal earning-to-give into perspective.

Another example: I have a vague sense that EAs are getting more diverse over time. Is this true? Currently, answering this question would require going through all the EA survey reports, reading numbers from images of plots, and typing them into a spreadsheet. It would be nice if all the data was easily accessible from some central repository and ready for analysis.

Data visualisation is a great tool for getting lots of people quickly up to speed on quantitative facts. Gapminder and Our World in Data do this to great effect. If we want EA to be an efficient machine for turning smart people into utils, then we should make full use of data visualisation’s affordances.

This Took Surprisingly Long

I started this project in June last year, a full 14 months ago. This is why the survey data is from 2019. The git repository has about 170 commits. I reckon ea...

Published on September 11, 2021 10:05 PM GMT

There’s lots of cool data floating around in EA: grant databases, survey results, growth metrics, etc. I’m a data scientist and enjoy data visualisation, so thought it would be a fun project to build a website which aggregates EA data into interactive plots.

The website is now live at EffectiveAltruismData.com. Source code is available on Github.

This project is still a work in progress: the data is pretty out of date and I’ve got lots of future work planned. But it’s far enough along that I’d like some public feedback.

The website is responsive and should look good on any desktop or tablet screen. If you’re viewing on a phone it will probably look ok in landscape mode.

Here are a few screenshots:

Implementation Details

The website is mostly coded in Python. The main libraries I used were:

Pandas for data handling.

Plotly for creating the interactive plots.

Dash for the web framework.

I also wrote a bunch of vanilla CSS for the frontend styling.

The web server is currently deployed with Heroku, which costs $7/month.

I have vague ambitions to re-implement the frontend with D3.js or Chart.js. This should cut down the loading time and give me more control over how the visualisations work.

Design Philosophy

I aimed to follow the data visualisation principles laid out in Information Dashboard Design and Storytelling with Data. These include:

Minimise the “ink-to-data” (or “pixel-to-data”) ratio to avoid distracting clutter.

Encoding data in length or distance is much higher fidelity than area or angle.

Avoid pie charts, stacked area plots, radar charts, violin plots.

Stick to bar charts, scatter plots, and line graphs as much as possible.

Don’t make the reader rotate their head.

Use horizontal bar charts rather than vertical ones.

Minimise the total length the eye has to travel to take in all the data.

Avoid legends.

On line graphs, put the labels directly on the ends of the lines.

Why I Did This

I said earlier that this project was motivated by my fancy for data visualisation. But I do think there’s a lot of scope for valuable data visualisation and data wrangling work within Effective Altruism.

For example, I’ve found it difficult to get a sense of the scale of donations within EA. Are total donations basically Open Philanthropy plus a rounding error? Or do donations from all the little guys like me actually make a difference in the big picture?

This isn’t just an interesting question in itself: it also informs my life decisions. If the total donations of people in my reference class is enough to make a noticeable change to AMF funding, then I’m more likely to steadily earn-to-give on a moderately affluent career path. If my reference class is totally overwhelmed by a handful of mega-donors, then I’m more likely to drop everything and spend a year figuring out if I can contribute to AI safety.

This was some of the motivation behind the first panel of EffectiveAltruismData.com. Ultimately, I’d like to have a plot which puts all the major stocks and flows of EA money on a common scale and puts my personal earning-to-give into perspective.

Another example: I have a vague sense that EAs are getting more diverse over time. Is this true? Currently, answering this question would require going through all the EA survey reports, reading numbers from images of plots, and typing them into a spreadsheet. It would be nice if all the data was easily accessible from some central repository and ready for analysis.

Data visualisation is a great tool for getting lots of people quickly up to speed on quantitative facts. Gapminder and Our World in Data do this to great effect. If we want EA to be an efficient machine for turning smart people into utils, then we should make full use of data visualisation’s affordances.

This Took Surprisingly Long

I started this project in June last year, a full 14 months ago. This is why the survey data is from 2019. The git repository has about 170 commits. I reckon ea...

Published on September 7, 2021 11:19 AM GMT

I am approaching the end of my AI governance PhD, and I’ve spent about 2.5 years as a researcher at FHI. During that time, I’ve learnt a lot about the formula for successful early-career research.

This post summarises my advice for people in the first couple of years. Research is really hard, and I want people to avoid the mistakes I’ve made.

My argument: At the early stage of your research career, you should think of yourself as an entrepreneur trying to build products (i.e. research outputs) that customers (i.e. the broader community) want to consume.

You might be thinking: but I thought I was trying to maximise my impact? Sure, but at this stage in your career, you don’t know what’s impactful. You should be epistemically humble and responsive to feedback from people you respect. You should be opportunistic, willing to pivot quickly.

I am calling this the “lean startup” approach to research. By now, everyone knows that most startup ideas are bad, and that founders should embrace this: testing minimal versions of the product, getting feedback from customers, iterating, and making dramatic changes where necessary. When you’re starting out in research, it’s the same.

Early-stage researchers have two big problems. Number one, all your project ideas are bad. Number two, once you’ve written something, nobody is going to read it. It’s like an app that nobody downloads. It is possible to avoid these pitfalls, but that requires active effort. I will list many strategies that I’ve found helpful. At the end of the post, I’ll give a few examples from my own career.

A lot of this advice is stolen from people who have helped me over the years. I encourage you to try it out.

EDIT: I am most familiar with AI governance. I'm not sure how well my views generalise to other fields. (Thanks to the commenters who raised this issue.)

Problem 1: your project ideas are bad

In the early stage of your research career, 80-100% of your project ideas are bad. You’ll feel like your favourite project idea is great, but then years later, you’ll ask yourself: “what was I thinking?”

Executing an idea requires a large time investment. You don’t want to waste that time on a bad idea.

By “project idea” I mean not just a topic, but some initial framing of the problem, and some promising ideas for what kind of arguments or results you might produce. So, how do you find a good one of those?

Ideally, someone senior tells you what to work on. But this is time-expensive for them, and they don’t want to give away their best ideas to somebody who might execute them badly. So more realistically.

Write out at least 10 project ideas, and ask somebody more senior to rank the best few. Always keep this list and add to it over time. This is a tried-and-tested method and it works very well. If you are pushing just one, single project idea, you might be able to arouse some minor, polite interest from other people, but this is a much less meaningful feedback process.

Notice when people are genuinely interested. Sometimes you will get a cue that a person is actually interested in a puzzle or argument that you’ve formulated. You notice that they’ve been nerd sniped. That’s a very valuable feedback signal. It is also a reason to recentre the project around the exact question that nerd sniped them. Because you don’t yet have a well-developed sense of what issues are most interesting, you should update heavily on this kind of feedback. (As you get more experienced, you can allow yourself to get nerd sniped by your own ideas.)

Fit into an established paradigm. While at FHI I have gradually absorbed a sense of the implicit worldviews of senior people, and what kinds of problems they think are important and interesting. You can get some of this from reading papers. One good strategy is to extend a line of inquiry that existing research has begun. Read existing literature and think: can I add anything to this? It’s tempting to look for an id...

Published on September 7, 2021 11:19 AM GMT

I am approaching the end of my AI governance PhD, and I’ve spent about 2.5 years as a researcher at FHI. During that time, I’ve learnt a lot about the formula for successful early-career research.

This post summarises my advice for people in the first couple of years. Research is really hard, and I want people to avoid the mistakes I’ve made.

My argument: At the early stage of your research career, you should think of yourself as an entrepreneur trying to build products (i.e. research outputs) that customers (i.e. the broader community) want to consume.

You might be thinking: but I thought I was trying to maximise my impact? Sure, but at this stage in your career, you don’t know what’s impactful. You should be epistemically humble and responsive to feedback from people you respect. You should be opportunistic, willing to pivot quickly.

I am calling this the “lean startup” approach to research. By now, everyone knows that most startup ideas are bad, and that founders should embrace this: testing minimal versions of the product, getting feedback from customers, iterating, and making dramatic changes where necessary. When you’re starting out in research, it’s the same.

Early-stage researchers have two big problems. Number one, all your project ideas are bad. Number two, once you’ve written something, nobody is going to read it. It’s like an app that nobody downloads. It is possible to avoid these pitfalls, but that requires active effort. I will list many strategies that I’ve found helpful. At the end of the post, I’ll give a few examples from my own career.

A lot of this advice is stolen from people who have helped me over the years. I encourage you to try it out.

EDIT: I am most familiar with AI governance. I'm not sure how well my views generalise to other fields. (Thanks to the commenters who raised this issue.)

Problem 1: your project ideas are bad

In the early stage of your research career, 80-100% of your project ideas are bad. You’ll feel like your favourite project idea is great, but then years later, you’ll ask yourself: “what was I thinking?”

Executing an idea requires a large time investment. You don’t want to waste that time on a bad idea.

By “project idea” I mean not just a topic, but some initial framing of the problem, and some promising ideas for what kind of arguments or results you might produce. So, how do you find a good one of those?

Ideally, someone senior tells you what to work on. But this is time-expensive for them, and they don’t want to give away their best ideas to somebody who might execute them badly. So more realistically.

Write out at least 10 project ideas, and ask somebody more senior to rank the best few. Always keep this list and add to it over time. This is a tried-and-tested method and it works very well. If you are pushing just one, single project idea, you might be able to arouse some minor, polite interest from other people, but this is a much less meaningful feedback process.

Notice when people are genuinely interested. Sometimes you will get a cue that a person is actually interested in a puzzle or argument that you’ve formulated. You notice that they’ve been nerd sniped. That’s a very valuable feedback signal. It is also a reason to recentre the project around the exact question that nerd sniped them. Because you don’t yet have a well-developed sense of what issues are most interesting, you should update heavily on this kind of feedback. (As you get more experienced, you can allow yourself to get nerd sniped by your own ideas.)

Fit into an established paradigm. While at FHI I have gradually absorbed a sense of the implicit worldviews of senior people, and what kinds of problems they think are important and interesting. You can get some of this from reading papers. One good strategy is to extend a line of inquiry that existing research has begun. Read existing literature and think: can I add anything to this? It’s tempting to look for an id...

Published on September 1, 2021 10:55 AM GMT

Over the last year, I have found myself giving feedback on various drafts, something that I'm generally quite happy to do. Recently, I got to give two variations of this same feedback in quick succession, so I noticed the commonalities, and then realized that these commonalities were also present on past pieces of feedback. I thought I'd write the general template up, in case others might find it valuable.

High level comments

You are working at the wrong level of abstraction and depth / you are biting more than you can chew / being too ambitious.

In particular, the questions that you analyze are likely to have many cruxes, i.e, factors that might change the conclusion completely. But you only identify a few such cruxes, and thus your analysis doesn't seem likely to be that robust.

I guess that the opposite error is possible—focus too much on one specific scenario which isn't that likely to happen. I just haven't seen it as much, and it doesn't seem as crippling when it happens.

Because you're being too ambitious, you don't have the tools necessary to analyze what you want to analyze, and to some extent those tools may not exist.

Compare with: Forecasting transformative AI timelines using biological anchors, Report on Semi-informative Priors on Transformative AI or Invertebrate Sentience: Summary of findings, which are much more constrained and have specific technical/semi-technical intellectual tool suited to the job (comparison with biological systems, variations on Laplace's law and other priors, markers of consciousness like reaction to harmful stimuli). You don't have an equivalent technical tool.

There is a missing link between the individual facts you outline, and the conclusions you reach (e.g., about [redacted] and [redacted]). I think that the correct thing to do here is to sit with the uncertainty, or to consider a range of scenarios, rather than to reach one specific conclusion. Alternatively, you could highlight that different dynamics could still be possible, but that on the balance of probabilities, you personally think that your favored hypothesis is more likely.

But in that case, it's be great if you more clearly defined your concepts and then expressed your certainty in terms of probabilities, because those are easier to criticize or put to the test, or even notice that there is a disagreement to be had.

Judgment calls

I get the impression that you rely too much on secondary sources, rather than on deeply understanding what you're talking about.

You are making the wrong tradeoff between formality and ¿clarity of thought?

Your report was difficult to read because of the trappings of scholarship—formal tone, long sentences and paragraphs, etc.) An index would have helped.

Your classification scheme is not exhaustive, and thus less useful.

This seems particularly important when considering intelligent adversaries.

I get the impression that you are not deeply familiar with the topic you are talking about. For example, when giving your overview, you don't consider [redacted], which is really the company working on this space.

In particular, I expect that the funders or decision-makers (for instance, Open Philanthropy) whom you might be attempting to influence or inform will be more familiar with the topic than you, and would thus not outsource their intellectual labor to your report.

I don't really know whether you are characterizing the literature faithfully, whether you're just citing the top few most salient experts that you found, or whether there are other factors at play. For instance, maybe the people who [redacted] don't want to be talking about it. Even if you are representing the academic consensus fairly, I don't know how much to trust it. Like, I get that it's an academic field, but I don't particularly expect it to have good models of the world.

It's unclear who is the "we" who will implement the measures you propose in the text.

This might be an acceptabl...