Published on October 13, 2021
Audience: I expect this to be helpful for anyone thinking of running a similar survey, and probably not interesting for anyone else.
We ran a survey of prominent AI safety and governance researchers which aimed to identify which AI existential risk scenarios those researchers find most likely. We learned several lessons that I expect to generalise to surveys with a similar audience or aims (e.g. surveys aiming to understand researchers' views in some field, or to test hypotheses about EAs' beliefs on certain issues, etc.). I've had several conversations where I've repeated these same lessons, so I figured I'd write them up quickly.
"Walkthroughs" are a good way to improve the questions
Before launching the survey, we iterated on the questions via a series of about 10 "walkthroughs". Each "walkthrough" involved being on a call with someone (a "tester") who would complete the survey, while verbalising their thought process and any thoughts that came to mind on what was confusing/could be improved, and so on. We did this until the survey design started to stabilise, i.e. the testers were basically happy.
Compared to asking testers to complete the survey in their own time and email us with feedback, this generated much richer and more helpful information. It was also lower effort for the testers (no need to write up their thoughts afterward). It’s also likely that people forget some of their feedback by the end, so it’s better to collect it as they go along.
The most helpful testers were those who we would want in the survey population, which (for our small population) posed a limitation on the number of tests we could do, so using tests wisely is important.
Testers are good at identifying flaws, but bad at proposing improvements
During these walkthroughs, testers reported things they didn't like about the current design, and sometimes suggested improvements to fix those flaws. Whilst the flaws that testers identified robustly pointed to areas that could be improved, more often than not, testers' suggestions on how to address the flaws weren't that helpful. In particular, several times we had the experience of making a change that one tester suggested, only for the next tester to suggest reverting to the previous design. (If you're interested: whether to make the AI risk scenarios mutually exclusive.) So, we found that suggested improvements should be taken as evidence that something was bad about the current design, and less as evidence that the particular change they suggest will actually make the survey better.
I'm told that this mirrors common wisdom in UI/UX design: that beta testers are good at spotting areas for improvement, but bad (or overconfident) at suggesting concrete changes.
Include relatively more "free-form" questions, or do interviews instead of a survey
Regarding the type of survey questions, there is a spectrum from "free-form" to "specific". Here are examples of questions on either end of the spectrum:
Very free-form: "Assume an existential catastrophe due to AI has occurred. What do you expect to have been the major causes and why?"
Very specific: "What is your estimate of the probability that an AI takeover scenario will occur?"
Specific questions ask for a response to some precise question, whereas free-form questions just prompt the respondent for their reactions to some vaguer question.
In our survey, most questions were very specific and asked for probability estimates. We expected this would make the analysis easier, because we could simply report summary statistics of these estimates, rather than having to synthesise and draw conclusions from a range of qualitative answers (which tend to be confusing because survey participants always write quickly!).
However, in hindsight this wasn’t clearly the best choice, because asking specific questions requires a level of conceptual precision that wasn't available for many of our questions.
By conceptual precision, I mean that th...