The Uses and Abuses of Surveys

May 10, 2010
Posted by Jay Livingston

Ask a silly question, you get a silly answer. Ask a politically loaded question, you get a political answer – even if the literal meaning of your question seems to be asking about matters of fact and not opinion..

Here are eight questions from a Zogby poll. Respondents were given a Likert scale from Strongly Agree to Strongly Disagree, but the authors treat answers as either correct or incorrect according to basic economic principles.

1. Restrictions on housing development make housing less affordable.
2. Mandatory licensing of professional services increases the prices of those services.
3. Overall, the standard of living is higher today than it was 30 years ago.
4. Rent control leads to housing shortages.
5. A company with the largest market share is a monopoly.
6. Third-world workers working for American companies overseas are being exploited.
7. Free trade leads to unemployment.
8. Minimum wage laws raise unemployment.

Respondents were also asked to classify themselves on a political spectrum – Progressive, Liberal, Moderate, Conservative, Very Conservative, Libertarian.

This survey wasn’t designed to discover what people think. It was designed to prove a political point: “The Further Left You Are the Less You Know About Economics.” That’s the title of a post about it at Volokh Conspiracy. A paper by Zeljka Buturovic and Dan Klein, who designed the survey, gives the results.

(Click on the image for a view large enough to actually read)

The results were similar for the other questions.

To be sure, the liberals view of economic cause-effect relationships reflects the way they would like the world to be rather than the way the world actually is. But the bias of the poll is obvious. As monkeyesq says in his comment at Volokh,

1. Pick 8 liberal positions that have a questionable economic basis;
2. Ask people whether they “agree” or “disagree” with the statements;
3. Find that liberals are more likely to support liberal positions;
4. Claim that liberals don’t understand economics.

There’s an even larger problem here – a problem that affects not just polls that have an obvious ax to grind,* but a basic problem of all survey research: the question the survey asks may not be the question the respondent hears or answers.

These eight questions have a literal meaning. As Todd Zywicki, who wrote the Volokh post, says, “Note that the questions here are not whether the benefits of these policies might outweigh the costs, but the basic economic effects of these policies.”

True, the questions do not ask about costs and benefits, although I don’t think that the survey included an explicit caveat like the one Zywicki adds after the fact. Still, we have to wonder about how people really heard these questions.

“Mandatory licensing of professional services increases the prices of those services” – Agree or Disagree? Maybe some people hear a different question, a question about policy implications: “Would you like cheaper, but unlicensed, doctors.”

“A company with the largest market share is a monopoly.” Maybe the what the person hears is: “Can companies with large market share – though less than the share required for it to be a monopoly (100%?) – still exercise monopolistic powers?”

As for the “exploitation” of third-world workers, the word may have a precise economic definition (e.g., it’s exploitation only if the worker has no choice) – I don’t know. But even if such an economic definition exists, to most people the word evokes moral judgment, not economics.

The other items also have flaws, as some of the comments at Volokh (now 200 and counting) point out. (I confess that I’m still puzzled by the responses to Standard of Living. Nearly a third of all the respondents think that the standard of living today is no better than it was 30 years ago – 55% on the left, 12% on the right 21% of libertarians.)

The survey may tell us that “epistemic closure” is a disease that can infect the left as well the right. But it also tells us to be cautious about interpreting survey questions literally. Even innocuous questions may mean different things to survey respondents. Until a question has been tested several times, we can’t be sure what respondents hear when they are asked that question.

*A Kos poll that set out to show that quite a few Republicans were extremist nuts suffers from a similar problem. I blogged it here.

How Genetics Works (borrowed post)

May 8, 2010
Posted by Jay Livingston

Flaneuse at Graphic Sociology reposted this from some place. I couldn't resist reposting it as well.

(Click on the image for a slightly larger view.)

I know nothing about the photo. It reminds me of Elliot Erwitt, though I’m sure it’s not.
It looks too old to have been photoshopped.

Lies, Damn Lies, and Statistics About Lies

May 7, 2010
Posted by Jay Livingston

We all know that Barack Obama is a Muslim who was born in Kenya and that his campaign was funded by Hugo Chavez.

Yes, I know, presidents going back to Washington have been the subject of rumors. But it seems that with the arrival of Obama, presidential rumors have become something of a growth industry (maybe our only growth industry these days.)

J.L. Bell blogs mostly about comic books and fantasy literature at his Oz and Ends. But two weeks ago he posted some numbers about presidential rumors on the Internet.* For his data, he went where most of us would go – Snopes.com. Here’s what he found.

(Click on the graph for a larger view.)

In less than two years, Obama rumor-mongers have had nearly twice the output that their Bush counterparts managed in eight years – 87 to 47. And while the Bush rumors split almost evenly true-false, false Obama rumors dwarfed the true ones.** The false rumors about Obama outnumbered the total number of rumors about Bush. And while the lies about Obama are almost all negative, some of the false rumors about Bush are quite flattering, along the lines of the George Washington cheery tree rumor – like the rumor that had Bush paying for the funeral of a boy who had drowned near the Crawford ranch.

Is there really a right-left difference? If the “epistemic closure” hypothesis is accurate – if conservatives, even the chattering intellectuals, live and write in a bubble that keeps out any realities that might conflict with their ideology – then conservatives of all sorts might also welcome into the bubble even the most preposterous and unfounded rumors.

Surely, there must be a sociology of rumor. What are the demographic correlates, if any? What are the conditions under which rumors are more likely to arise and spread? I would imagine that lack of trust is important. The less we trust others who are outside our relatively small circle, and the less we interact with them, the more likely we will be to rely on rumor.

Trust, at least trust in government, has been decreasing generally, but conservatives, when they are not in control of the government, are especially mistrusting. Under any circumstances, false beliefs are frustratingly resistant to facts. It probably doesn’t improve conservative’s grasp of reality when they have a major TV network giving airtime to these rumors and when their leaders tell them about death panels.


* His post was picked up by Salon.com and then by Brendan Nyhan, who has had some interesting posts on the epistemic closure discussion.

** The other categories were “mixed,” “undetermined,” and “unclassifiable.” In the graph, I collapsed the latter two categories into “other.”

Messenger NAEP - or Charles Murray Channelling the Left

May 5, 2010
Posted by Jay Livingston

How fitting in these weeks of final exams to be reminded that tests are irrelevant – at least if you don’t like the results. They are the messenger who brings bad news. But will killing the messenger solve the problem?

I remember the good old days when academics railed at standardized tests like the SAT. If the tests showed group differences – between black and white, male and female – that just showed that the tests were biased and should not be used.

Now, Charles Murray, the man on the right that everyone on the left loves to hate,* has joined the anti-test chorus and, on the Times op-ed page today, he’s singing lead. Murray, like most conservatives, is a supporter of charter schools. He’s badmouthing standardized tests because they show that charter schools do no better than traditional public schools at educating students, especially the kinds of kids who are most in need of effective schools.

The volume of studies on school “choice” – comparing charter schools and voucher programs to traditional public schools – is now large. Most commonly, these find no difference in the progress of students in charters and those in publics. A few charters do better; a greater number underperform their public counterparts.

Rather than dismissing the research as merely wrong or turning up the volume on the few studies favorable to charters and ignoring the rest, as do some conservatives I know, Murray goes radical. What these studies really show, he says, is that tests don’t matter.

Why not instead finally acknowledge that standardized test scores are a terrible way to decide whether one school is better than another?

I’m not going to search through Murray’s oeuvre to see how many times he has cited test scores as meaningful or where he stood on No Child Left Behind. But I am going to guess that if the results had been different, if research showed that charters consistently outperformed publics on standardized tests, Murray would be putting up billboards praising NAEP and the rest.

But Murray goes way beyond the idea that tests are irrelevant. He says that when it comes to teaching kids to read and do math, the schools themselves are irrelevant.

Cognitive ability, personality and motivation come mostly from home. What happens in the classroom can have some effect, but smart and motivated children will tend to learn to read and do math even with poor instruction, while not-so-smart or unmotivated children will often have trouble with those subjects despite excellent instruction. If test scores in reading and math are the measure, a good school just doesn’t have that much room to prove it is better than a lesser school.

Murray is deliberately ignoring one inconvenient fact: that some teachers and some schools consistently do a better job of teaching hard-to-teach populations. But instead of wanting more effort to figure out just what those teachers and schools do so that others can also do more of it, Murray denies that they have any meaningful effect.

What Murray mostly wants is not good education, though he would probably not oppose such improvement. What he wants is charter schools. Since test scores that measure learning don’t support charters, Murray goes back to another song from the old left fake book – School as Ideology. The left used to complain – maybe it still does – that what schools really did was not so much teach subjects but indoctrinate kids into the dominant capitalist ideology so as to turn out a compliant labor force.

Right on, says Murray. But the dominant ideology, in his view, is now liberal and therefore to be avoided. Charter schools “would give parents a choice radically different from the progressive curriculum used in the county’s** other public schools.” Charters are a way for conservative parents to keep their kids out of the hands of the liberals – a sort of home-school away from home.

He has a point. If I lived in Texas, I might want the option of sending my kid to a state-funded charter school that included in the curriculum some of the arcane figures from America’s history that will now be excluded – people like Thomas Jefferson. Such choice would be welcome to middle-class parents like me and Murray, who are not much worried about our kids learning to read. But a choice of ideology is probably not high on the list of concerns of the parents whose kids are in schools where most of the students are years behind in reading and math.


* The mention of Murray’s name has been, for at least fifteen years now, a Pavlovian ringing of the bell curve, guaranteed to set lefties to frothing at the mouth. I actually used to admire Murray, at least for his writing. When a Marxist colleague asked me to review a manuscript she had written, a rather tendentious book on violence, I told her that she should try to write more like Murray – to present radical and probably offensive ideas in language that makes them seem calm and reasonable.

** Murray is referring to a specific charter proposal in the Maryland county where he lives, but for the argument he is making, he might just as easily have said country instead of county.