Surveys and Confirmation Bias

November 10, 2011
Posted by Jay Livingston

When he taught research methods as a grad student, Michael Schwartz gave his students this assignment: “Create a survey to show . . .” and he would tell them the conclusion he wanted the survey to support.  The next week, he’d give them the same assignment but with the desired conclusion the opposite of the first one.

A year and a half ago, I criticized (here) a much publicized study by Dan Klein and Zeljka Buturovic:  “This survey, I said, “wasn’t designed to discover what people think. It was designed to prove a political point,” and that point was that liberal ideology blinds people to economic facts. 

I was reminded of Mike’s assignment when I read Klein’s recent article at The Atlantic.  In a bit of academic fairness that’s probably all too rare, Klein went on to create a survey designed to see if conservative ideology has a similar effect.

Klein hoped that his conservative and libertarian allies would not so readily agree with politically friendly economic ideas that were nevertheless unsound. But conservatives in the new survey were “equally stupid” as the liberals in the earlier survey.

Klein also expected some nasty nyah-nyahing from his liberal critics.  But no, “The reaction to the new paper was quieter than I expected.”   In fact, one of those liberal critics, Matt Yglesias, offered an observation that Klein used as his takeaway from the two surveys: “there’s a lot of confirmation bias out there.” 

Yes, but confirmation bias is not just something that affects people who respond to surveys.  As Mike’s assignment makes clear, we also need to be wary of confirmation bias on the part of those who create the surveys. There is the further problem I mentioned in my earlier post:  a one-shot survey is inherently ambiguous. We can’t be sure just what the respondents really hear when they are asked the question. 

My own takeaway, besides admiration for Klein’s honesty, is that when you design your research as a statement (proving some point), you don’t learn nearly as much as when you design it as a genuine question.

Defining Deviance — Up and Down

November 9, 2011
Posted by Jay Livingston

Most of the increase in economic inequality over the past two or three decades comes from the enormous growth in money going to the very rich.  David Brooks attributes that growth in part to a change in culture.*
You see a shift in social norms. Up until 1970 or so, a chief executive would have been embarrassed to take home more than $20 million. But now there is no shame, and top compensation zooms upward.
It’s what Daniel Patrick Moynihan nearly two decades ago (1993) called “defining deviancy down.”  Things which had once been a matter of shame have become acceptable. Norms change.

But they don’t change all by themselves – as though they were part of some “low-pressure system” or “cold air mass” moving into the region.  And the change is not always towards looser standards.  Moral entrepreneurs campaign to define deviancy up, and sometimes they succeed.  If we were back in the pre-feminist, “Mad Men” world of the 1950s, Herman Cain wouldn’t be having his current problems.**  But women – those pesky feminists – in just a few decades, have changed the dominant public judgment about men using a position of power to get laid.  Once accepted, maybe even admired and envied, it’s now something a guy doesn’t want other people, even his friends, to know about. 

“Mad Men” shows us some other examples of deviance defined up.  Look in on an ad agency today and you won’t see anyone smoking. A few souls may go out to the street for a cigarette break, but we see their smoking not as pleasure but as addiction, something to be pitied. Nor will you see anyone coming back drunk from a three-martini lunch or pouring himself a tumbler of Canadian Club in his office. Score one for the anti-tobacco and anti-drunkenness forces.

Besides moral entrepreneurship, norms can change as a matter of invidious social comparison – when those lower down the social scale take cues from those above them.  Fashions in clothes or names filter down through the class structure. So do ideas of unacceptable behavior. In the 18th century, new canons of manners started with the court, then the aspiring gentry, and eventually even commoners were embarrassed by the audible belch or fart. In the 21st century,  it’s not that we suddenly realized that drinking at work or smoking is harmful. Rather, as my British friend once said, “it isn’t done” – meaning that it isn’t done by people of our social position. In fact, moral entrepreneurs might be more successful if instead of trying to convince people that something is wrong, they tried to convince them it is low class.

As Oscar Wilde said,
As long as war is regarded as wicked, it will always have its fascination. When it is looked upon as vulgar, it will cease to be popular. (HT: Mark Kleiman)
-----------

*  Brooks is a conservative columnist for the New York Times. But his liberal counterpart on the Times op-ed page, Paul Krugman, years ago offered a similar explanation for the skyrocketing pay of those at the top.

** For those who might not remember: in 2011, when I originally posted this, Herman Cain, a business executive, was a candidate for the Republican nomination for President, and some of his female former employees had accused him of sexual harassment.

Patriotism Goes to the Movies

November 7, 2011
Posted by Jay Livingston

“Patriotism,” says Paul Krugman. “is about making sacrifices for the national good, not serving your personal motives or interests.”  Krugman (in his blog, here ) was citing Michael Lind’s Slate article about “The Patriot,” the 2000 film starring Mel Gibson.  Lind complains that the Patriot of the title, “sits out the American Revolution, until a sadistic . . . British commander kills one of his sons. whereupon he spends the next two days – oops, I mean two hours – avenging himself.”

That’s not patriotism, harrumphs Lind, it’s “amoral familism”*
It appears that today's audiences can't imagine any cause that could justify political violence other than injury to a child or wife.
This movie is deeply subversive of patriotism. Indeed, patriotism is a concept that neither the screenwriter . .  nor the director . . . seems to understand.
Maybe so.  But the writer and director do understand something that Lind apparently does not:  movies are not real life.  If they are, then  “Singin’ in the Rain” is deeply subversive of rational reactions to meteorological events. 

Patriotism may be the last refuge of a scoundrel, but it’s no refuge at all for a filmmaker. Real-life Americans are patriotic, sometimes to an extent others find offensive. But that kind of patriotism doesn’t make for good movies.  In the American movies that I know, good guys never do their good deeds out of abstract idealism. Their motives are always personal. (Even better than a non-ideoogical hero is the character who has an ideology but abandons it in order to kill bad guys – e.g., Grace Kelly in “High Noon”).  

America movie-heroes often take up arms against bad guys, but we would mistrust a hero whose actions are purely ideological and not rooted in personal revenge. Our heroes, even when they are fighting for Good, have the decency to deny any ideological motive. Here’s one familiar (I hope) example:


I’m afraid Michael Lind would be disappointed in Rick, and in Grace Kelly shooting the bad guy to protect her husband.  “[In] the Zeitgeist in the United States in A.D. 2000 . . . American national patriotism is giving way . . .to the perennial rival of patriotism at all levels: amoral familism.”

If that’s the Zeitgeist, it’s a Geist that goes back a long Zeit.  “Casablanca” was made in 1942, “High Noon” in 1952.  Or take another classic from the early 1950s, “On the Waterfront.”  Marlon Brando winds up doing the right thing in ratting out the racketeer union boss (the right thing according to the film’s construction of morality).  But it’s not until he has a personal reason – the union boss has his brother Charley killed – that he takes action.  
You gave it to Joey, you gave it to Dugan, and you gave it to Charley who was one of your own.. . and I’m glad what I done to you!
The film’s attempts to make stevedores spout lofty motives ring embarrassingly false, as when one of the longshoremen urges Brando to defy the boss in order to
give us back our union, so we can run it on the up and up.
(I still cringe when I hear that line.)

Is this bias towards the personal and against the political an aspect of American culture?  Or is it the medium?  Maybe political ideals – socialist realism, capitalist realism, patriot realism, etc. – don’t make for compelling movies.  Movies are first about characters, not ideas. The medium washes out the message.  

---------------
* The term comes from Edward Banfield’s 1958 book The Moral Basis of a Backward Society.  Amoral familism, according to Banfield, was that basis.

Lying With Statistics, and Really Lying With Statistics

November 4, 2011
Posted by Jay Livingston

“The #1 way to lie with statistics is . . . to just lie!” says Andrew Gelman, who a) knows much about statistics and b) is very good at spotting statistical dishonesty.

But maybe there’s a difference between lying with statistics and just plain making stuff up.

I’ve commented before about social psychologists’ affinity for Candid-Camera deception, but this Dutch practitioner goes way beyond that.  [The Telegraph has the story .] 


The committee set up to investigate Prof Stapel said after its preliminary investigation it had found "several dozen publications in which use was made of fictitious data" . . .
[Stapel’s] paper that linked thoughts of eating meat eating with anti-social behaviour was met with scorn and disbelief when it was publicised in August, it took several doctoral candidates Stapel was mentoring to unmask him. . . .

the three graduate students grew suspicious of the data Prof Stapel had supplied them without allowing them to participate in the actual research. When they ran statistical tests on it themselves they found it too perfect to be true and went to the university's dean with their suspicions.
What’s truly unsettling is to think that maybe he’s not the only one.