Whose Bad Guess is More Bad? Difficult Comparisons

October 29, 2014
Jay Livingston

How to compare percentages that are very different? 

A recent Guardian/Ipsos poll asked people in fourteen wealthy nations to estimate certain demographics. What percent of the population of your country are immigrants? Muslim? Christian?

People overestimated the number of immigrants and Muslims, and underestimated the number of Christians. But the size of the error varied.  Here is the chart on immigration that the Guardian published (here).

Italy, the US, Belgium, and France are way off. The average guess was 18-23 percentage points higher than the true percentage.  People in Japan, South Korea, Sweden, and Australia were off by only 7-8 percentage points.

But is that a fair comparison? The underlying question is this: which country is better at estimating these demographics? Japan and South Korea have only 2% immigrants. People estimated that it was 10%, a difference of eight percentage points. But looked at another way, their estimate was five times the actual number. The US estimate was only 2½ times higher than the true number.

The Guardian ranks Hungary, Poland, and Canada together since they all have errors of 14 points. But I would say that Canada’s 35% vs. 21% is a better estimate than Hungary’s 16% vs. 2%.  Yet I do not know a statistic or statistical technique that factors in this difference and allows us to compare countries with very few immigrants and those with far more immigrants.* 

My brother suggested that the Guardian’s readers could get a better picture of the differences if the chart ordered the countries by the immigrant percentage rather than by the percentage-point gap.

This makes clearer that the 7-point overestimate in Sweden and Australia is a much different sort of error than the 8-point overestimate in South Korea and Japan. But I’m still uncertain as to the best way to make these comparisons.

* Saying that I know of no such statistic is not saying much. Perhaps others who are more familiar with statistics will know how to solve this problem.

No comments: