Do You Hear What I Hear? Maybe Not.

December 18, 2011
Posted by Jay Livingston

As I’ve said before (here), the question the researcher asks is not always the question people hear.   That’s especially true when the question is about probabilities.

Here, for example, is the ending of a fictional vignette from a recent study in the Journal of Personality and Social Psychology.

Richard found a wallet on the sidewalk. Nobody was looking, so he took all of the money out of the wallet. He then threw the wallet in a trash can.

Is it more probable that Richard is
    a. a teacher
    b. a teacher and a rapist

Since the category “a teacher” necessarily includes teacher/rapists as well, the correct answer is “a.” But many people choose “b.”  The study used this “conjunction fallacy”* to probe for prejudices by switching out the rapist for various other categories.  Some subjects were asked about atheist/teachers, others about Muslim/teachers, and so on.  The finding:

A description of a criminally untrustworthy individual was seen as comparably representative of atheists and rapists but not representative of Christians, Muslims, Jewish people, feminists, or homosexuals.

Andrew Gelman, a usually mild-mannered reporter on things methodological, had a post on this with the subject line, “This one is so dumb it makes me want to barf.”

What’s really disturbing about the study is that many people thought it was “more probable” that the dude is a rapist than that he is a Christian! Talk about the base-rate fallacy.

Maybe it would settle Andrew’s stomach to remember that the question the researchers asked was almost certainly not the question people heard.   What the researchers pretend to be asking is this:

Of all thieves, which are there more of – teachers or rapist/teachers? 

After all, that is indeed the literal meaning.  But it’s pretty obvious that the question people are answering is something different:

Which group has a higher proportion of thieves among them – all teachers or the subset rapist/teachers?

The researchers say they weren’t at all interested in demonstrating the conjunction fallacy.  They were just using it to uncover the distrust people feel towards atheists.  What they found was that when it comes to dishonesty, people (specifically, 75 female and 30 male undergrads at the University of British Columbia) rank atheists at about the same level as rapists.

But why resort to such roundabout tricks?  Why not ask the question directly?**

Who is more likely to steal a wallet when nobody is looking?
    a.  an atheist
    b. a rapist
    c.  neither; they are equally larcenous

Or:

On a seven-point scale, rank each of the following on how likely they would be to steal a wallet when nobody is looking:

•     an atheist: 1   2   3   4   5   6   7
•     a Christian: 1   2   3   4   5   6   7
•     a rapist: 1   2   3   4   5   6   7
•     etc. 

Instead, they asked questions that they knew would confuse nearly anyone not fluent in the language of statistics and probability.  I wonder what would happen if in their “who do you distrust” study they had included a category for experimental social psychologists.***

---------------
*  Daniel Kahneman and Amos Tversky pretty much invented the conjunction fallacy thirty years ago with their “Linda problem,” and Kahneman discusses it in his recent book Thinking Fast and Slow.  To get the right answer, you have to ignore intuition and make your thinking very, very slow.  Even then, people with no background in statistics and logic may still get it wrong.

** The authors’ presentation of their results is also designed to frustrate the ordinary reader. Each condition (rapist/teacher, atheist/teacher, homosexual/teacher, etc.) had 26 (or in one case 27) subjects.  The payoff was the number of errors in each group.  But the authors don’t say what that number was.  They give the chi-square, the odds ratios, the p’s and the b’s.  But they don’t tell us how many of the 26 subjects thought that the wallet snatcher was more likely to be an atheist/teacher or a Christian/teacher than to be merely a teacher.

*** The JPSP is one of the most respected journals in the field, maybe the most respected, influential, and frequently cited, as I pointed out here.