Aaron Brown is an author and risk management professional, formerly the Chief Risk Officer at the hedge fund AQR. Aaron’s recent works are titled Wrong Number: How to Extract Truth From a Blizzard of Quantitative Disinformation and The Poker Face of Wall Street.

Greg and Aaron discuss why quantitatively flawed studies still persist today. Aaron argues that the central problem is not just incompetence or conspiracy but a macro phenomenon he calls tribalism, combined with the diffusion of responsibility across authors, reviewers, journals, and journalists. He discusses examples, including an NTSB “Chinatown bus” study, a USAID mortality claim, a Chunnel fire-risk study, an observational marijuana/heart-attack paper, and a study claiming 40% of COVID deaths were caused by evictions and later cited in courts and legislation. They contrast academia’s weak incentives with finance and gambling, where betting forces accountability, and Aaron describes the empirical Bayesian approach he prefers using base rates and evidence.

*unSILOed Podcast is produced by University FM.*

There are consequences to publicizing bad research.

[34:45] I think most researchers are careful not to let the university press office get ahold of their bad study. They're careful not to go out and give interviews on it. The ones who forget that, they're the ones who cause the problems and get caught. I mean, not many people do get caught, and the consequences of getting caught are pretty low, but it can happen. You lose professional credibility, and that's extremely important, you know? That's really the be all and end all for most researchers I know, is what their peer researchers think of them. And that's where you get hurt. In fact, you get hurt even for getting publicity for your good work, you know? There still is a real feeling in a lot of sciences that the guy in the headline is not a real scientist.

Why are people so easily misled by quantitative information?

[05:32]  It's been documented over and over in lots of different ways, that most published research findings are false, and yet nobody seems to care. 

Aaron discusses the promise and pitfalls of Bayesian reasoning. 

[57:28] You don't have to go all the way to Bayesian to know that what they're doing in the journals is wrong. The journal, the frequentist, the Fisher classical hypothesis testing, the gold standard, double-blind control trials—those things are just wrong. And you don't have to go all the way to Bayesianism. You can just say, "Okay, we can just show mathematically that those don't work."

Recommended Resources:

Guest Profile:

Guest Work:

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.