Men Are From Mars, Survey Respondents Are From Neptune

November 22, 2015
Posted by Jay Livingston

Survey researchers have long been aware that people don’t always answer honestly. In face-to-face interviews especially, people may mask their true opinion with the socially desirable response. Anonymous questionnaires have the same problem, though perhaps to a lesser degree. But self-administered surveys, especially the online variety, have the additional problem of people who either don’t take it seriously or treat it with outright contempt. Worse, as Shane Frederick (Yale, management) discovered, the proportion of “the random and the perverse” varies from item to item.

On open-ended questions, when someone answers “asdf” or “your mama,” as they did on an online survey Frederick conducted, it’s pretty clear that they are making what my professor in my methods class called “the ‘fuck you’ response.”

But what about multiple-choice items.

Is 8+4 less than 3? YES / NO
11% Yes.

Maybe 11% of Americans online really can’t do the math.  Or maybe all 11% were blowing off the survey. But then what about this item?

Were you born on the planet Neptune? YES / NO
17% Yes     

Now the ranks of the perverse have grown by at least six percentage points, probably more. Non-responders, the IRB, and now the random and the perverse – I tell ya, survey researchers don’t get no respect.

----------------
Big hat tip to Andrew Gelman. I took everything in this post from his blog (here), where commenters try seriously to deal with the problem created by these kinds of responses.