Conclusions First, Then the Research

July 15, 2007
Posted by Jay Livingston

Sometimes in class, I’ll ask students how they might find get information on some sociological question. “Do a survey,” is the frequent answer. Students seem to think that a survey, any survey, has magical powers to reveal the truth.

“A survey is just asking a bunch of people a bunch of questions,” I usually say. “Who are you going to ask, and what are you going to ask them?” In other words, the validity of a survey depends on the quality of the sample and the quality of the questions.

Michael Schwartz, when he was a grad student TA for the methods course, would give students an assignment intended to tarnish the objectivity-mystique of surveys: : “Design a questionnaire to show that . . .” Conclusion first, then the research. Mike’s point was that the questions – the order in which they are asked, the phrasing, etc. – can bias the results. I think part two of the assignment was to design a second questionnaire that would yield a result directly opposite to the first.

It was all in good, educational fun, I thought. Real researchers would never deliberately do something like that. But today, Mark Kleiman at The Reality-Based Community gives some examples of result-oriented questions from Zogby polls. Zogby is a pretty good source on political questions, especially their surveys of international populations. So it was sobering to see these polls custom-tailored to produce the results wanted by the people who were paying for the survey.

Here’s an example for a poll Zogby did for Judicial Watch, a conservative watchdog group that gets a lot of its money from the Clinton-hating Scaife foundation.

Some people believe that the Bill Clinton administration was corrupt. Whether or not you believe the Clinton administration was corrupt, how concerned are you that there will be high levels of corruption in the White House if Hillary Clinton is elected President in 2008?

I think that the Sociology department at Montclair will rewrite some of the items in our teacher evaluation forms.

Some students in this professor’s other courses have said that this instructor is one of the best in the university. Regardless of those other courses, how would you rate the professor’s performance in this course?