Overcoming Social Desirability Bias – He’s Got a Little List

April 19, 2011
Posted by Jay Livingston

As some day it may happen that a survey must be done, you need a little list, a quick five-item list – for sex or race or crime or things quite non-PC but fun, where pollsters all have missed, despite what they insist. There’s the guy who says he’d vote for blacks if they are qualified; he’d vote for women too, but are we sure he hasn’t lied? “How many partners have you had?” Or “Did you ever stray?” With things like this you can’t always believe what people say. You tell them it’s anonymous, but still their doubts persist, and so your methodology can use this little twist.

It’s called the List Experiment (also the Unmatched Count Technique). It’s been around for a few years, though I confess I wasn’t aware of it until I came across this recent Monkey Cage post by John Sides that linked to another post from the presidential year of 2008. Most surveys then were finding that fewer than 10% of the electorate were unwilling to vote for a woman (Hillary was not mentioned by name). But skeptical researchers (Matthew Streb et al., here gated), instead of asking the question directly, split the sample in half. They asked one half

How many of the following things make you angry or upset?
  • The way gasoline prices keep going up.
  • Professional athletes getting million dollar-plus salaries.
  • Requiring seat belts to be used when driving.
  • Large corporations polluting the environment.
Respondents were told not to say which ones pissed them off, merely how many. Researchers calculated the average number of items people found irritating. The second half got the same list but with one addition:
  • A woman serving as president.
If the other surveys are correct, adding this one item should increase the mean by no more than 10%. As it turned out, 26% of the electorate would be upset or angry about a woman president, considerably more than the 6% in the GSS sample who said they wouldn’t vote for a woman.

The technique reminds me of a mentalist act: “Look at this list, sir, and while my back is turned tell me how many of those things you have done. Don’t tell me which ones, just the total number. Now I want you to concentrate very hard . . . .” But I can certainly see its usefulness as a way to check for social desirability bias.

2 comments:

Robin said...

This seems problematic. Isn't it possible that the inclusion of the woman president comment affects how people see the other comments? Maybe people are thinking, "Women leaders are so not annoying... in fact athletes, who I normally wouldn't find annoying, seem annoying in comparison."

Imagine you have four men and you are asked to name how many are tall. Say they're 5'8", 6'0", 6'2" and 6'8." You might say only one is tall. If you trade out the 6'0" for another 5'8"er, it might cause you to see the 6'2" person as tall also. So your answer will go up - but it's not the new person - it's just the new person affecting your perception of what's there.

Jay Livingston said...

Woman president as the low-end anchor. Is it possible? Yes. But you could make the same argument for absolutely any item anybody might come up with. Do I think it's likely that this is what's going on? No. Do I have any way of determining which explanation is correct? No.