Judge the Judges: With Guests Daniel Kahneman, James Hutchinson & G.M. Pucilowski

Many episodes of this podcast deal with cognitive biases that can hinder our decision-making abilities. In this episode of Choiceology with Katy Milkman, we look at a different kind of error: how completely irrelevant information can negatively influence our judgments, making them varied and unpredictable.

This variability of human judgment—or noise—is the topic of an upcoming book by Nobel Prize-winning economist Daniel Kahneman, with Cass Sunstein and Olivier Sibony. You’ll hear an interview with Kahneman later in the episode where he explains his preoccupation with the substantial and expensive effects of noise. He proposes ways to reduce the problem of noise for industries, businesses, and individuals who need to make more objective judgements.

But first, we’ll dive into the world of wine judging. G.M. “Pooch” Pucilowski will take you on a guided tour of the California State Fair Commercial Wine Competition. You’ll hear about the criteria for judging different varietals—and the accompanying challenges that wine judges face as they swirl, sniff, and sip through dozens and sometimes hundreds of different wines.

After years of coordinating and observing the judges, Pooch noticed a large amount of variability in the results. This may not be surprising, since taste is subjective. But after some tweaks to the process, he even began to notice that judges were inconsistent with themselves!

Enter vintner and retired oceanographer Robert Hodgson. Pooch teamed up with Hodgson to devise a way to study and improve the consistency of wine judging and push for a more objective competition. The results were promising, but not without controversy.

G.M. “Pooch” Pucilowski is a speaker, writer, wine judge, and educator. He runs wine appreciation classes through his University of Wine.

You’ll also hear about the potential role of chemical analysis and artificial intelligence in improving the results of wine judging from James Hutchinson, formerly of the Royal Society of Chemistry and currently CEO of KiwiNet.

And finally, Katy explores the potential of leveraging noise to produce better decisions by employing the wisdom of crowds—and even “the crowd within.”