Surveys — Questions and Answers

December 10, 2012
Posted by Jay Livingston

Neil Caren at Scatterplot  lifts up the rock that is the New Family Structure Study (NFSS) – the basis of Mark Regnerus’s controversial research on children of gay parents – and discovers some strange creatures wriggling about underneath: 

. . .   85 people reported living at least four months with their “mother’s girlfriend/partner.” However—and this is where it gets tricky—a different question (S8) asked, “Did you ever live with your mother while she was in a romantic relationship with another woman?” Eight people who reported in the calendar that they lived with their mother’s girlfriend answered no to this question.

So ten percent of the people who said they lived with the mother’s girlfriend also said on a different question that they did not live with the mother’s girlfriend.
                   
We all rely on surveys – pollsters, social scientists, market researchers, government agencies, businesses. We try to make our questions straightforward.  But the question we ask is not always the question people answer.  And people’s answers – about what they think and what they did – are influenced by external factors we might not have considered.  Especially if the survey is a one-off (unlike the GSS and other surveys with frequently asked questions),  we have to be cautious about taking the results at face value.

(Previous posts on this problem are here and here.)