Posted by Jay Livingston
If this is Superbowl weekend, then the Socioblog’s fancy must be turning to thoughts of the wisdom of crowds vs. the smart money. It’s a question I have returned to several times since the first year this blog was on the field. (See. for example, this post about the 2010 Superbowl.)
The “wisdom of crowds” is like the Ask-the-Audience option in “Who Wants to Be a Millionaire.” The “smart money” is like Phone-a-Friend — a friend who knows a lot about the subject.
The trouble with the wisdom of crowds is that sometimes the crowd is wrong, as it was in the 2007 NFC championship game between the Bears and the Saints that I blogged about at the time (here.)
Now, a trio of academics — John McCoy (marketing), Dražen Prelec (management), and H. Sebastian Seung (neuroscience) — has a variation that allows you to derive the right answer from the crowd even when the crowd is wrong. You might call it the first derivative of crowd wisdom.
Is Philadelphia the capital of Pennsylvania?
Suppose you don’t know, and you ask the crowd.
| The correct answer is no. The capital is Harrisburg. But many people think it is, because Philadelphia is a large, populous city. Most people know about Philadelphia. When you ask that question to a crowd of people, as we did with MIT students, only about a third of the crowd gets the correct answer.* |
Yes is the popular answer. The crowd, by two-to-one, says Yes, Philadelphia is the capital. The crowd is wrong. The capital of Pennsylvania is Harrisburg. So much for the wisdom of crowds.
Wait, not so fast, say McCoy and his colleagues. Let’s also ask another question: “What percent of people do you think will answer No to this question?” The average estimate is 23%. But in fact, 33% answer No. This makes No a “surprisingly popular” answer, surprising in that more people than expected say No. It’s as though you are taking the first derivative of crowd wisdom rather than the wisdom function itself.
If you went with the popular answer, you’d say Yes and be wrong. But if you go with the derivative — the “surprisingly popular” answer — you’ll get it right.
McCoy sees applications of this to all kinds of forecasts — the market for some product, the price of gold, voting, He doesn’t mention the Superbowl. Right now, about 25% of bettors think that the Rams will win or that they will lose by 2 points or less. But suppose we asked all bettors, “What fraction of people do you think are betting the Rams?” If they guessed that only 10% of them are backing the Rams, then the Rams would be the “surprisingly popular” choice, and you would be a fool not to put down a grand to win $1250. Alas, I know of no such surveys. Besides, I don’t trust Belichick.
Two other thoughts:
First, McCoy’s makes the concept harder to understand by choosing an example where No is right. “Is No the correct answer?” “Yes, No is right.”
Second, I was stunned that two-thirds of MIT students did not know the capital of fifth most populous state in the country. Look, people, we’re not asking about Pierre or Carson City. This is not rocket science. And now I get the feeling that at MIT a question about rocket science might have gotten a higher proportion of correct answers.
-----------------------------
*From an interview with McCoy on a Wharton School podcast. An article by Prelec, Seung, and McCoy in Nature is here behind a paywall.
Perhaps the high proportion of incorrect MIT student responses is less surprising if we also expect MIT students to contain a large proportion of international students who were not made to memorize the United States state capitals in grade school.
ReplyDeleteYes, I thought of that too. That's the trouble with all these immigrants and foreigners -- a veritable caravan of Asians invading Kendall Square.
ReplyDelete