Note: this episode was recorded in August of 2022.

In the latest Elucidation, Matt talks to Witold Więcek about the difficulties that come up for researchers who would like to draw upon statistics. 

Lots of academic fields need to draw heavily on statistics, whether it’s economics, psychology, sociologym, linguistics, computer science, or data science. This means that a lot of people coming from different backgrounds often need to learn basic statistics in order to investigate whatever question they’re investigating. But as we’ve discussed on this podcast, statistical reasoning is easy for beginners to mess up, and it’s also easy for bad faith parties to tamper with in undetectable ways. They can straight up fabricate data, they can cherry pick it, they can keep changing the hypothesis they are testing until they find one that is supported by a trend in the data they have. So what should we do? We can’t give up on statistics; it is simply too useful a tool.

Witold Więcek argues that researchers have to be mindful of “p-hacking”. Statistical significance, the golden standard of academic publishing, can easily be guaranteed by unscrupulous research or motivated reasoning: statistically speaking, even noise can look like signal if we keep asking more and more questions of our data. Modern statistical workflows require us to either adjust the results for number of hypotheses tested or to follow principles of Bayesian inference. As a broader strategy, Więcek recommends that every research project making significant use of statistical arguments bring in in an external consultant, who can productively stress test those arguments in an adversarial way, given that they aren’t part of the main team.

It was a great conversation! I hope you enjoy it.

Matt Teichman

Hosted on Acast. See acast.com/privacy for more information.