Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Published on October 14, 2021

Key take-aways

Sam Bankman-Fried was able to make over $20 billion in just 4 years.

By staying in quant trading, he’d have made several orders of magnitude less money over that period.

Presumably, (way?) fewer than 100 EAs tried something similar.

If that’s our base rate, then maybe more EAs should try to found multi-billion dollar companies.

Note on this post:

This is my first EA Forum post. One stumbling block in the past has been to have very high expectations toward potential post ideas and then not write them up at all. So to counteract that, here I've tried writing up my thoughts on one specific thing without having as broad a scope as might be ideal.

Who am I talking to?

This post focuses on the expected value of founding a start-up. Of course, there are many reasons not to found a start-up other than beliefs about the expected value. To start with, it requires high levels of sacrifice and probably the most likely outcome is failure. Furthermore, the qualities required to make a good founder are quite rare and if you don’t have them, that’s no reason to feel bad! In fact, there’s probably lots of other ways you can do a ton of good!

So here I will focus on the people who think they might be a good fit and who would consider founding a start-up if they were convinced it was the highest-impact thing for them to do.

This is only one way to look at this

There are several approaches one could take to this question and this is only one of them. That’s why the post is titled “An update in favor of trying to ...” and not “You should probably try to .” (although that could still be true!). The most important factor is probably your inside view. But there are also other outside views! See these two posts from Brian Tomasik and Applied Divinity Studies for instance[1]. They both consider the success rate of companies that got into YCombinator among other things. A paragraph I particularly liked from the latter post was this:

I understand that the odds of becoming a billionaire are low, but it doesn’t matter if you only consider the conditional probabilities. What’s the cost/benefit of taking 6 months off your day job to work on a startup? Conditional on being successful there, what’s the cost/benefit of trying very hard to seek out venture capital? Given that you’ve raised money, what’s the cost/benefit of trying to make a billion dollars? The bet sounds insane to begin with, but at each step you’re taking on a very reasonable level of risk.

The argument

Sam Bankman-Fried is an example of a person motivated by EA to earn as much as he could in order to give it all away. It’s going pretty well. His net worth is estimated to be around $22.5 billion and he’s the world’s richest person under thirty. It took him about 4 years (!) to build most of that wealth, from founding Alameda Research in October 2017 to today.

Lincoln Quirk, co-founder of Wave, which is now valued at $1.7B, is also worth mentioning here. He too has been interested in EA for a long time. I think for the purposes of this post though, it’s fine to focus on Sam. Reasoning in this footnote[2].

Can we conclude anything from Sam’s story? I think so! Specifically, I wanna lay out an argument that on current margins, more people should take bets to make tens of billions of dollars. That is, if you are in a reference class similar to Sam’s before founding Alameda Research and you are currently considering founding a start-up, then this post aims to cause you to update in favor of that.

Let’s first ask how many EAs tried to found companies that would potentially be worth billions. Let’s call this number N.

David Moss estimates there are about 2300 highly engaged EAs. A little under 40% of them are earning to give. How many of those are trying to found multi-billion dollar companies? I don’t know...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 14, 2021

The pandemic has become predictable, which is the world we much prefer to live in. Cases and deaths continue their slow but steady declines, vaccine approvals for boosters and children continue to slowly move forward. Mandates continue to make people angry but mostly work when used, while also not that often being used. Same old, same old.

The big news story was that one of the studies of Long Covid was revealed to be vastly overestimating frequency due to a methodological ‘error.’ It’s an important update.

Executive Summary

Cases and deaths declining as expected.

Approvals progressing slowly as expected.

Long Covid study had important error, see discussion later in post.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 560k cases (-10%) and 11,500 deaths (-8%).

Results: 547k cases (-12%) and 11,051 deaths (-12%).

Prediction for next week: 481k cases (-12%) and 9,835 deaths (-11%).

We are still in a low-variance situation, with no reason to expect any big surprises for at least several more weeks.

Deaths

The exact rate of decline here was a pleasant surprise, but we’ve gotten to the point where this is a lagging indicator, so the times when it is surprising are mostly about data gathering issues, either holidays or changes in the ways in which we do testing. Expect this trend to continue, and continue to mirror cases with a lag.

Cases

This is pretty much exactly what we would have expected. The Northeast number is slightly disappointing, others were slightly better than expected, but no real surprises.

Vaccinations

The rate of vaccinations continues to not change much, and it’s clear that the positive bump wasn’t the start of a surge. That continues not to tell us much about how much work the mandates are doing here. My impression is that the mandates that have teeth work, but that few of them have teeth.

In his podcast interview this week with FiveThirtyEight, former FDA Commissioner Dr. Scott Gottlieb asks where the cost-benefit analyses are on the vaccine mandates, since there don’t seem to be any. He estimates that we could use mandates to go from something like 80% adult vaccination rates to 84%, and asks how much that really accomplishes.

My answer is that this change would be worth quite a lot, but how much depends heavily on how we would react to the two scenarios.

Cutting the number of unvaccinated adults by 20% wouldn’t cut transmission by 20%, since there is some spread by non-adults and some spread by the vaccinated. It’s still a major percent drop in transmission if behavior didn’t adjust, at least 10%. Cutting transmission by 10% means a 10% cut in cases every four days or so if behavior doesn’t adjust, which is a rather big game, cutting cases in half each month.

Behaviors would of course adjust, which is a big benefit. We’d get to increase our equilibrium amount of Covid-19 risk by 10%, which is a substantial percentage of the way back to regular life. If we used this to eliminate ‘stupid prevention’ selectively the win would be even bigger.

Alternatively, if we didn’t adjust our behaviors, then we get full suppression before too long, or damn close to it, at which point the adjustments happen no matter what any ‘public health experts’ or government officials might want. If we’re looking at under 10k cases per day, as we likely are within a few months, people outside of some particular institutions going to go back to normal, and even in those few places there’d be increasing pressure over time.

The intuition pump here is that putting us over the top is a huge deal, and that this is plausibly exactly what we need to get over the top, and if this doesn’t get us there it still gives us a bunch of extra slack to spend. Plus it creates new ‘safe spaces’ where everyone is vaccinated. And of course, it lowers av...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 14, 2021

The pandemic has become predictable, which is the world we much prefer to live in. Cases and deaths continue their slow but steady declines, vaccine approvals for boosters and children continue to slowly move forward. Mandates continue to make people angry but mostly work when used, while also not that often being used. Same old, same old.

The big news story was that one of the studies of Long Covid was revealed to be vastly overestimating frequency due to a methodological ‘error.’ It’s an important update.

Executive Summary

Cases and deaths declining as expected.

Approvals progressing slowly as expected.

Long Covid study had important error, see discussion later in post.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 560k cases (-10%) and 11,500 deaths (-8%).

Results: 547k cases (-12%) and 11,051 deaths (-12%).

Prediction for next week: 481k cases (-12%) and 9,835 deaths (-11%).

We are still in a low-variance situation, with no reason to expect any big surprises for at least several more weeks.

Deaths

The exact rate of decline here was a pleasant surprise, but we’ve gotten to the point where this is a lagging indicator, so the times when it is surprising are mostly about data gathering issues, either holidays or changes in the ways in which we do testing. Expect this trend to continue, and continue to mirror cases with a lag.

Cases

This is pretty much exactly what we would have expected. The Northeast number is slightly disappointing, others were slightly better than expected, but no real surprises.

Vaccinations

The rate of vaccinations continues to not change much, and it’s clear that the positive bump wasn’t the start of a surge. That continues not to tell us much about how much work the mandates are doing here. My impression is that the mandates that have teeth work, but that few of them have teeth.

In his podcast interview this week with FiveThirtyEight, former FDA Commissioner Dr. Scott Gottlieb asks where the cost-benefit analyses are on the vaccine mandates, since there don’t seem to be any. He estimates that we could use mandates to go from something like 80% adult vaccination rates to 84%, and asks how much that really accomplishes.

My answer is that this change would be worth quite a lot, but how much depends heavily on how we would react to the two scenarios.

Cutting the number of unvaccinated adults by 20% wouldn’t cut transmission by 20%, since there is some spread by non-adults and some spread by the vaccinated. It’s still a major percent drop in transmission if behavior didn’t adjust, at least 10%. Cutting transmission by 10% means a 10% cut in cases every four days or so if behavior doesn’t adjust, which is a rather big game, cutting cases in half each month.

Behaviors would of course adjust, which is a big benefit. We’d get to increase our equilibrium amount of Covid-19 risk by 10%, which is a substantial percentage of the way back to regular life. If we used this to eliminate ‘stupid prevention’ selectively the win would be even bigger.

Alternatively, if we didn’t adjust our behaviors, then we get full suppression before too long, or damn close to it, at which point the adjustments happen no matter what any ‘public health experts’ or government officials might want. If we’re looking at under 10k cases per day, as we likely are within a few months, people outside of some particular institutions going to go back to normal, and even in those few places there’d be increasing pressure over time.

The intuition pump here is that putting us over the top is a huge deal, and that this is plausibly exactly what we need to get over the top, and if this doesn’t get us there it still gives us a bunch of extra slack to spend. Plus it creates new ‘safe spaces’ where everyone is vaccinated. And of course, it lowers av...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Published on October 14, 2021

[Spoiler note: I have used only examples from relatively early on in the new Thrawn Trilogy and ones which I believe won’t spoil your enjoyment of the story; however, if you’d like to go in with absolutely no knowledge, this will tell you some things about the trilogy.]

I’ve recently read the first two books of Timothy Zahn’s new Thrawn trilogy and it’s given me some thoughts about writing competent characters.

The unique thing about Thrawn as a character is that he comes off as genuinely competent. The first book of the new Thrawn trilogy makes this particularly vivid because, instead of having to pair Thrawn with characters with canonical non-competent personalities, Zahn was free to invent the equally competent Eli Vanto, Arindha Pryce, and Nightswan.

In most books, characters who are supposed to be geniuses go one of three ways. First, they’re chessmasters with incredibly tangled Rube Goldberg Machine plans: “of course, I knew all along that overhearing that conversation would cause the protagonist’s wife to conclude she’s cheating on her and divorce her, motivating the protagonist to seek out the Stone of Plot, thus moving my plans one step closer to fruition!” Second, they’re scientific or magical geniuses with fake science or magic: “of course! If you reverse the polarity of the subspace photon launcher it will babble the techno and that will lower the shields, letting us in!” Third, they’re very good improvisers, which is a genuine kind of intelligence, but also one that is kind of easy for writers. The character doesn’t have to have clever plans, they just have to react cleverly to what’s happening—and the audience will cut them some slack if their solution isn’t amazing. After all, they did think of it in a few seconds.

Thrawn takes a fourth approach.

In his first scene, Thrawn is fighting some Imperial troops which are camped in the forest. He figures out that their shield system must let through small forest animals, or they’d be constantly dealing with false alarms. So he tapes bombs to squirrels and blows up the Imperial camp.

Now, I have no idea if this is a legitimately clever military tactic, or if it makes sense that Thrawn is the first person to think of it. I’m not a tactician. Maybe the Imperial troops should already have had a way of dealing with squirrel bombs. But I can understand Thrawn’s plan. There are no hidden steps of “oh, he’s just a genius, if I were as smart as Thrawn it would make perfect sense why this plan works as well as it does, it only looks like a leap of logic because I’m stupid.” And it’s legitimately clever. I wouldn’t have thought of that! I don’t read a lot of military science fiction, but I don’t recall seeing people using a similar tactic before.

Thrawn also comes off as competent because he does the sort of thing that, in the real world, makes military generals competent. Indeed, it’s almost as if his true secret weapon is that he studied with the secret masters of the long-lost art of the Masters of Business Administration.

Despite living in the Star Wars universe, where multiple people manage their underlings by murdering anyone who gives them bad news, Thrawn repeatedly refuses to punish people when they fail at a task, as long as they did their best or there were unforeseen circumstances. In the original Thrawn trilogy, he once asked why a subordinate let the heroes escape, discovered that the subordinate didn’t know how to use a particular piece of technology, and then punished the man’s trainer.

In the new Thrawn trilogy, Thrawn sometimes looks at spreadsheets. I have never before read a military science fiction novel, much less a Star Wars tie-in novel, in which spreadsheets play such an important role. But sometimes Thrawn and Eli Vanto need to track down smugglers, and then Eli looks at spr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Published on October 14, 2021

[Spoiler note: I have used only examples from relatively early on in the new Thrawn Trilogy and ones which I believe won’t spoil your enjoyment of the story; however, if you’d like to go in with absolutely no knowledge, this will tell you some things about the trilogy.]

I’ve recently read the first two books of Timothy Zahn’s new Thrawn trilogy and it’s given me some thoughts about writing competent characters.

The unique thing about Thrawn as a character is that he comes off as genuinely competent. The first book of the new Thrawn trilogy makes this particularly vivid because, instead of having to pair Thrawn with characters with canonical non-competent personalities, Zahn was free to invent the equally competent Eli Vanto, Arindha Pryce, and Nightswan.

In most books, characters who are supposed to be geniuses go one of three ways. First, they’re chessmasters with incredibly tangled Rube Goldberg Machine plans: “of course, I knew all along that overhearing that conversation would cause the protagonist’s wife to conclude she’s cheating on her and divorce her, motivating the protagonist to seek out the Stone of Plot, thus moving my plans one step closer to fruition!” Second, they’re scientific or magical geniuses with fake science or magic: “of course! If you reverse the polarity of the subspace photon launcher it will babble the techno and that will lower the shields, letting us in!” Third, they’re very good improvisers, which is a genuine kind of intelligence, but also one that is kind of easy for writers. The character doesn’t have to have clever plans, they just have to react cleverly to what’s happening—and the audience will cut them some slack if their solution isn’t amazing. After all, they did think of it in a few seconds.

Thrawn takes a fourth approach.

In his first scene, Thrawn is fighting some Imperial troops which are camped in the forest. He figures out that their shield system must let through small forest animals, or they’d be constantly dealing with false alarms. So he tapes bombs to squirrels and blows up the Imperial camp.

Now, I have no idea if this is a legitimately clever military tactic, or if it makes sense that Thrawn is the first person to think of it. I’m not a tactician. Maybe the Imperial troops should already have had a way of dealing with squirrel bombs. But I can understand Thrawn’s plan. There are no hidden steps of “oh, he’s just a genius, if I were as smart as Thrawn it would make perfect sense why this plan works as well as it does, it only looks like a leap of logic because I’m stupid.” And it’s legitimately clever. I wouldn’t have thought of that! I don’t read a lot of military science fiction, but I don’t recall seeing people using a similar tactic before.

Thrawn also comes off as competent because he does the sort of thing that, in the real world, makes military generals competent. Indeed, it’s almost as if his true secret weapon is that he studied with the secret masters of the long-lost art of the Masters of Business Administration.

Despite living in the Star Wars universe, where multiple people manage their underlings by murdering anyone who gives them bad news, Thrawn repeatedly refuses to punish people when they fail at a task, as long as they did their best or there were unforeseen circumstances. In the original Thrawn trilogy, he once asked why a subordinate let the heroes escape, discovered that the subordinate didn’t know how to use a particular piece of technology, and then punished the man’s trainer.

In the new Thrawn trilogy, Thrawn sometimes looks at spreadsheets. I have never before read a military science fiction novel, much less a Star Wars tie-in novel, in which spreadsheets play such an important role. But sometimes Thrawn and Eli Vanto need to track down smugglers, and then Eli looks at spr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Published on October 14, 2021

I've been using the term symbol-grounding in a slightly different way to how people often use it. I've been using it to mean:

"This agent's symbol is reliable enough, across a wide variety of environments, for me to use it as evidence for the referent of that symbol."

What do I mean by "referent of that symbol"? I mean the actual object or concept that the symbol is referring to[1] ("referring", hence referent).

Whereas classical symbol grounding is closer to:

"This agent understands what this symbol means and refers to."

My definition is empirical, and implies the classical definition (though the reverse is not true): if an entity uses a symbol reliably enough for me to know that it tracks the real thing, then I can generally assume they understand what they're talking about.

Here I'll look at the connection between the definition I've been using and the classical one, and see how we could get empirical evidence for the classical one as well - what would it take to convince us that this agent actually understands what a symbol means?

Sufficient, not necessary

Let's consider a knowledgeable meteorologist, with a full suite of equipment, motivated to figure out the temperature of a room. According to my definition, if we could detect the variable corresponding to "temperature" in their head, this variable would be well grounded to the actual temperature[2]; the same would be true for the classical definition: the meteorologist understands what temperature is.

We can contrast that with a digital thermometer, lying in the room. The readout of that thermometer is not grounded, according to either definition.

Let's put this into a causal graph:

We have the true temperature, the thermometer readout, and the meteorologist's verdict. These three nodes are highly correlated under standard situations. Why then do I feel that the meteorologist's verdict was well grounded while the thermometer's readout was not?

Because the meteorologist could adapt to interfering variables in a way that the thermometer could not. Suppose the thermometer was put in the sun or in a fridge. Or if its readout lost power and started flashing 99.99. Then the thermometer could not adapt or correct its estimate. But the meteorologist could; if they found their own thermometer was corrupted, they could move it around, fix it, mentally correct it, or just ignore it, and get a more accurate estimate some other way.

And in doing that, they would demonstrate an understanding of the causal structure of the world and how it worked. The thermometer in the sun is wrong because it continues to absorb radiation and shows a continuously increasing temperature. The thermometer in the fridge is obviously placed in a special location within the room. And so on.

Thus, if the meteorologist can keep their estimate correlated with the true temperature, even in complicated and interfering situations, it demonstrates that they understand[3] how the variable in question works and how it connects with other aspects of the world. Thus my definition (correlated variables across many environments) implies the standard definition (understanding of what the variable is or means).

Lazy, but understanding

The next day, our highly motivated meteorologist is lazy or indifferent - they're no longer on the clock. Now, they have no real interest in the room's true temperature, but they are willing to state a vague impression:

In this case, my definition of grounding fails: the meteorologist's symbol is poorly correlated with the temperature, and is relatively easy to throw off.

Nevertheless, we should still say that the meteorologist's understanding of temperature is well grounded - after all, it's the same person! The classical definition agrees. But is it something we can define empirically?

It seems we can; there ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Published on October 14, 2021

I've been using the term symbol-grounding in a slightly different way to how people often use it. I've been using it to mean:

"This agent's symbol is reliable enough, across a wide variety of environments, for me to use it as evidence for the referent of that symbol."

What do I mean by "referent of that symbol"? I mean the actual object or concept that the symbol is referring to[1] ("referring", hence referent).

Whereas classical symbol grounding is closer to:

"This agent understands what this symbol means and refers to."

My definition is empirical, and implies the classical definition (though the reverse is not true): if an entity uses a symbol reliably enough for me to know that it tracks the real thing, then I can generally assume they understand what they're talking about.

Here I'll look at the connection between the definition I've been using and the classical one, and see how we could get empirical evidence for the classical one as well - what would it take to convince us that this agent actually understands what a symbol means?

Sufficient, not necessary

Let's consider a knowledgeable meteorologist, with a full suite of equipment, motivated to figure out the temperature of a room. According to my definition, if we could detect the variable corresponding to "temperature" in their head, this variable would be well grounded to the actual temperature[2]; the same would be true for the classical definition: the meteorologist understands what temperature is.

We can contrast that with a digital thermometer, lying in the room. The readout of that thermometer is not grounded, according to either definition.

Let's put this into a causal graph:

We have the true temperature, the thermometer readout, and the meteorologist's verdict. These three nodes are highly correlated under standard situations. Why then do I feel that the meteorologist's verdict was well grounded while the thermometer's readout was not?

Because the meteorologist could adapt to interfering variables in a way that the thermometer could not. Suppose the thermometer was put in the sun or in a fridge. Or if its readout lost power and started flashing 99.99. Then the thermometer could not adapt or correct its estimate. But the meteorologist could; if they found their own thermometer was corrupted, they could move it around, fix it, mentally correct it, or just ignore it, and get a more accurate estimate some other way.

And in doing that, they would demonstrate an understanding of the causal structure of the world and how it worked. The thermometer in the sun is wrong because it continues to absorb radiation and shows a continuously increasing temperature. The thermometer in the fridge is obviously placed in a special location within the room. And so on.

Thus, if the meteorologist can keep their estimate correlated with the true temperature, even in complicated and interfering situations, it demonstrates that they understand[3] how the variable in question works and how it connects with other aspects of the world. Thus my definition (correlated variables across many environments) implies the standard definition (understanding of what the variable is or means).

Lazy, but understanding

The next day, our highly motivated meteorologist is lazy or indifferent - they're no longer on the clock. Now, they have no real interest in the room's true temperature, but they are willing to state a vague impression:

In this case, my definition of grounding fails: the meteorologist's symbol is poorly correlated with the temperature, and is relatively easy to throw off.

Nevertheless, we should still say that the meteorologist's understanding of temperature is well grounded - after all, it's the same person! The classical definition agrees. But is it something we can define empirically?

It seems we can; there ...

Published on October 13, 2021

Wherever you are in the world, you can still be part of EA Global: London (October 29-31).

We’re no longer accepting applications for in-person attendees, but we’re excited to announce our free virtual attendee package, which includes:

A profile on the Swapcard event app, giving you access to:

Hundreds of profiles of other virtual attendees

Messaging and meeting scheduling with other virtual attendees

50+ organisation profiles (including their latest research and jobs)

24 virtual speed meeting sessions covering a range of cause areas and affinity groups

A virtual opening reception and after-party in Gather

The opportunity to interact with the live stream of our keynote sessions by:

Submitting and upvoting questions for speakers

Live text chat with other attendees

To attend the conference as a virtual attendee, you’ll need to submit an application by 8 am UTC on 28 October.

Apply now

EA Global conferences are for people who have made a career plan, volunteered, or donated a significant proportion of their income on the basis of effective altruist principles, either currently or in the past. Applicants should be quite familiar with the ideas of effective altruism and will need to provide an example of how they’ve applied these ideas to their career, study, or donation plans.

If you’re relatively new to effective altruism, you’re welcome to join us as a virtual attendee. The virtual environment is a good place to explore and practice the key ideas of effective altruism before attending an in-person conference.

If you have any questions, please contact us at [email protected].

Published on October 13, 2021

Wherever you are in the world, you can still be part of EA Global: London (October 29-31).

We’re no longer accepting applications for in-person attendees, but we’re excited to announce our free virtual attendee package, which includes:

A profile on the Swapcard event app, giving you access to:

Hundreds of profiles of other virtual attendees

Messaging and meeting scheduling with other virtual attendees

50+ organisation profiles (including their latest research and jobs)

24 virtual speed meeting sessions covering a range of cause areas and affinity groups

A virtual opening reception and after-party in Gather

The opportunity to interact with the live stream of our keynote sessions by:

Submitting and upvoting questions for speakers

Live text chat with other attendees

To attend the conference as a virtual attendee, you’ll need to submit an application by 8 am UTC on 28 October.

Apply now

EA Global conferences are for people who have made a career plan, volunteered, or donated a significant proportion of their income on the basis of effective altruist principles, either currently or in the past. Applicants should be quite familiar with the ideas of effective altruism and will need to provide an example of how they’ve applied these ideas to their career, study, or donation plans.

If you’re relatively new to effective altruism, you’re welcome to join us as a virtual attendee. The virtual environment is a good place to explore and practice the key ideas of effective altruism before attending an in-person conference.

If you have any questions, please contact us at [email protected].

Published on October 13, 2021

Audience: I expect this to be helpful for anyone thinking of running a similar survey, and probably not interesting for anyone else.

We ran a survey of prominent AI safety and governance researchers which aimed to identify which AI existential risk scenarios those researchers find most likely. We learned several lessons that I expect to generalise to surveys with a similar audience or aims (e.g. surveys aiming to understand researchers' views in some field, or to test hypotheses about EAs' beliefs on certain issues, etc.). I've had several conversations where I've repeated these same lessons, so I figured I'd write them up quickly.

"Walkthroughs" are a good way to improve the questions

Before launching the survey, we iterated on the questions via a series of about 10 "walkthroughs". Each "walkthrough" involved being on a call with someone (a "tester") who would complete the survey, while verbalising their thought process and any thoughts that came to mind on what was confusing/could be improved, and so on. We did this until the survey design started to stabilise, i.e. the testers were basically happy.

Compared to asking testers to complete the survey in their own time and email us with feedback, this generated much richer and more helpful information. It was also lower effort for the testers (no need to write up their thoughts afterward). It’s also likely that people forget some of their feedback by the end, so it’s better to collect it as they go along.

The most helpful testers were those who we would want in the survey population, which (for our small population) posed a limitation on the number of tests we could do, so using tests wisely is important.

Testers are good at identifying flaws, but bad at proposing improvements

During these walkthroughs, testers reported things they didn't like about the current design, and sometimes suggested improvements to fix those flaws. Whilst the flaws that testers identified robustly pointed to areas that could be improved, more often than not, testers' suggestions on how to address the flaws weren't that helpful. In particular, several times we had the experience of making a change that one tester suggested, only for the next tester to suggest reverting to the previous design. (If you're interested: whether to make the AI risk scenarios mutually exclusive.) So, we found that suggested improvements should be taken as evidence that something was bad about the current design, and less as evidence that the particular change they suggest will actually make the survey better.

I'm told that this mirrors common wisdom in UI/UX design: that beta testers are good at spotting areas for improvement, but bad (or overconfident) at suggesting concrete changes.

Include relatively more "free-form" questions, or do interviews instead of a survey

Regarding the type of survey questions, there is a spectrum from "free-form" to "specific". Here are examples of questions on either end of the spectrum:

Very free-form: "Assume an existential catastrophe due to AI has occurred. What do you expect to have been the major causes and why?"

Very specific: "What is your estimate of the probability that an AI takeover scenario will occur?"

Specific questions ask for a response to some precise question, whereas free-form questions just prompt the respondent for their reactions to some vaguer question.

In our survey, most questions were very specific and asked for probability estimates. We expected this would make the analysis easier, because we could simply report summary statistics of these estimates, rather than having to synthesise and draw conclusions from a range of qualitative answers (which tend to be confusing because survey participants always write quickly!).

However, in hindsight this wasn’t clearly the best choice, because asking specific questions requires a level of conceptual precision that wasn't available for many of our questions.

By conceptual precision, I mean that th...