https://astralcodexten.substack.com/p/mantic-monday-scoring-rule-controversy

 

Metaculus scoring rule controversy

Zvi considered using some Metaculus markets for his weekly coronavirus roundup, but was turned off by the scoring rules.

Ross Rheingans-Yoo writes about the issue here. Everyone agrees Metaculus’ scoring rule is “proper”, a technical term meaning that it correctly incentivizes you to choose the probability you think is true. Zvi and Ross’s objection is that it doesn’t correctly incentivize you about whether to bet at all, or how much effort to put into betting.

For example, on many questions, you can make guaranteed-positive bets - you’ll gain points on the prediction even if you were maximally wrong. If you were trying to maximize your Metaculus points, you would bet on all of these questions. If you were trying to maximize your Metaculus points in a limited amount of time, you might even bet on them without investigating at all. The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.