Counting the Un-askable and Un-tellable

This summary is not available. Please click here to view the post.

The Ecological Fallacy

May 10, 2010
Posted by Jay Livingston

The ecological fallacy is alive and well. Ross Douthat, the New York Times’s other conservative (the one that isn’t David Brooks), breathes life into it in his op-ed today on Red Families v. Blue Families, the new book by Naomi Cahn and June Carbone.

First Douthat gives props to the “blue family” model:

couples with college and (especially) graduate degrees tend to cohabit early and marry late, delaying childbirth and raising smaller families than their parents, while enjoying low divorce rates and bearing relatively few children out of wedlock.

Then there’s the “red family” for whom the stable, two-parent family is more a hope than a reality:

early marriages coexist with frequent divorces, and the out-of-wedlock birth rate keeps inching upward.

Blue looks good – good for the couples, good for the kids, good for society. But Douthat finds a moral thorn among the blue roses – abortion.

The teen pregnancy rate in blue Connecticut, for instance, is roughly identical to the teen pregnancy rate in red Montana. But in Connecticut, those pregnancies are half as likely to be carried to term.

So it isn’t just contraception that delays childbearing in liberal states, and it isn’t just a foolish devotion to abstinence education that leads to teen births and hasty marriages in conservative America. It’s also a matter of how plausible an option abortion seems, both morally and practically, depending on who and where you are.

Douthat is channeling Balzac: Behind every great fortune lies a great crime. Behind every more modest fortune – say, enough to live in Danbury if not Greenwich – is a more modest crime, i.e., an abortion or two.

But here’s the fallacy: Douthat makes it appear that the Connecticut residents who are getting those abortions are the same “couples with college and (especially) graduate degrees” we met in the paragraph on blue families. The illogic goes like this:

Blue states with higher levels of income and education also have higher levels of abortion than do Red states.
Therefore more Blue chip people have more abortions than do Red necks.

No, no, no (I hear myself repeating to my students). You cannot assume that a correlation at the state level also exists at the individual level. Just because wealthier states have higher rates of abortion, you cannot assume that wealthier individuals have higher rates of abortion. To make that assumption is to commit the ecological fallacy.

In fact, the Connecticut women who are getting abortions may also be relatively poor and uneducated. The difference is that abortion may give them access to further education or employment – not a graduate degree and a 6-figure job, but something better than what they could expect were they in Alabama. Or Montana.

The Uses and Abuses of Surveys

May 10, 2010
Posted by Jay Livingston

Ask a silly question, you get a silly answer. Ask a politically loaded question, you get a political answer – even if the literal meaning of your question seems to be asking about matters of fact and not opinion..

Here are eight questions from a Zogby poll. Respondents were given a Likert scale from Strongly Agree to Strongly Disagree, but the authors treat answers as either correct or incorrect according to basic economic principles.

1. Restrictions on housing development make housing less affordable.
2. Mandatory licensing of professional services increases the prices of those services.
3. Overall, the standard of living is higher today than it was 30 years ago.
4. Rent control leads to housing shortages.
5. A company with the largest market share is a monopoly.
6. Third-world workers working for American companies overseas are being exploited.
7. Free trade leads to unemployment.
8. Minimum wage laws raise unemployment.

Respondents were also asked to classify themselves on a political spectrum – Progressive, Liberal, Moderate, Conservative, Very Conservative, Libertarian.

This survey wasn’t designed to discover what people think. It was designed to prove a political point: “The Further Left You Are the Less You Know About Economics.” That’s the title of a post about it at Volokh Conspiracy. A paper by Zeljka Buturovic and Dan Klein, who designed the survey, gives the results.

(Click on the image for a view large enough to actually read)

The results were similar for the other questions.

To be sure, the liberals view of economic cause-effect relationships reflects the way they would like the world to be rather than the way the world actually is. But the bias of the poll is obvious. As monkeyesq says in his comment at Volokh,

1. Pick 8 liberal positions that have a questionable economic basis;
2. Ask people whether they “agree” or “disagree” with the statements;
3. Find that liberals are more likely to support liberal positions;
4. Claim that liberals don’t understand economics.

There’s an even larger problem here – a problem that affects not just polls that have an obvious ax to grind,* but a basic problem of all survey research: the question the survey asks may not be the question the respondent hears or answers.

These eight questions have a literal meaning. As Todd Zywicki, who wrote the Volokh post, says, “Note that the questions here are not whether the benefits of these policies might outweigh the costs, but the basic economic effects of these policies.”

True, the questions do not ask about costs and benefits, although I don’t think that the survey included an explicit caveat like the one Zywicki adds after the fact. Still, we have to wonder about how people really heard these questions.

“Mandatory licensing of professional services increases the prices of those services” – Agree or Disagree? Maybe some people hear a different question, a question about policy implications: “Would you like cheaper, but unlicensed, doctors.”

“A company with the largest market share is a monopoly.” Maybe the what the person hears is: “Can companies with large market share – though less than the share required for it to be a monopoly (100%?) – still exercise monopolistic powers?”

As for the “exploitation” of third-world workers, the word may have a precise economic definition (e.g., it’s exploitation only if the worker has no choice) – I don’t know. But even if such an economic definition exists, to most people the word evokes moral judgment, not economics.

The other items also have flaws, as some of the comments at Volokh (now 200 and counting) point out. (I confess that I’m still puzzled by the responses to Standard of Living. Nearly a third of all the respondents think that the standard of living today is no better than it was 30 years ago – 55% on the left, 12% on the right 21% of libertarians.)

The survey may tell us that “epistemic closure” is a disease that can infect the left as well the right. But it also tells us to be cautious about interpreting survey questions literally. Even innocuous questions may mean different things to survey respondents. Until a question has been tested several times, we can’t be sure what respondents hear when they are asked that question.

*A Kos poll that set out to show that quite a few Republicans were extremist nuts suffers from a similar problem. I blogged it here.

Meanness and Means

April 2, 2010
Posted by Jay Livingston

On March 27, the Times ran an op-ed by David Elkind, “Playtime is Over,” about the causes of bullying:

it seems clear that there is a link among the rise of television and computer games, the decline in peer-to-peer socialization and the increase of bullying in our schools.

I was skeptical. Had there really been an increase in bullying? Elkind offered no evidence. He cited numbers for current years (school absences attributable to bullying), but he had no comparable data for the pre-computer or pre-TV eras. Maybe he was giving a persuasive explanation for something that didn’t exist.

I sent the Times a letter expressing my doubts. They didn’t publish it. Elkind is, after all, a distinguished psychologist, author many books on child development. As if to prove the point, three days later, the big bullying story broke. An Irish girl in South Hadley, Massachusetts committed suicide after having been bullied by several other girls in her high school. The nastiness had included Facebook postings and text messages.

I guess Elkind was right, and I was wrong. Bullying has really exploded out of control in the electronic age.

But today the op-ed page features “The Myth of Mean Girls,” by Mike Males and Meda-Chesney Lind. They look at all the available systematic evidence on nastiness by teenagers – crime data (arrests and victimizations), surveys on school safety, the Monitoring the Future survey, and the CDC’s Youth Risk Behavior Surveillance. They all show the same trend:

This mythical wave of girls’ violence and meanness is, in the end, contradicted by reams of evidence from almost every available and reliable source.

Worse, say the authors, the myth has had unfortunate consequences:

. . . more punitive treatment of girls, including arrests and incarceration for lesser offenses like minor assaults that were treated informally in the past, as well as alarmist calls for restrictions on their Internet use.*

This is not to say that bullying is O.K. and nothing to worry about. Mean girls exist. It’s just that the current generation has fewer of them than did their parents’ generation. Should we focus on the mean or on the average? On average, the kids are not just all right; they’re nicer. Funny that nobody is offering explanations of how the Internet and cell phones might have contributed to this decline in meanness.

*For a recent example, see my post about criminal charges brought against young teenage girls for “sexting,” even though the pictures showed no naughty bits.


UPDATE: At Salon.com, Sady Doyle argues that Lind and Males looked at the wrong data.

Unfortunately, cruelty between girls can't really be measured with the hard crime statistics on which Males and Lind's argument relies. . . . Bullying between teenage girls expresses itself as physical fighting less often than it does as relational aggression, a soft and social warfare often conducted between girls who seem to be friends. You can't measure rumors, passive-aggressive remarks, alienation and shaming with statistics.

She has a point. While most of the evidence Males and Lind cite is not “hard crime statistics,” it does focus on overt violence. But Doyle is wrong that you can’t measure “relational aggression.” If something exists, you can measure it. The problem is that your measure might not be valid enough to be of use.

If Doyle is right, if nonphysical bullying hasn’t been measured, that doesn’t mean that Males and Lind are wrong and that bullying has in fact increased. It means that we just don’t know. We do know that physical violence has decreased. So here are the possibilities.

1. Physical and nonphysical aggression are inversely related. Girls have substituted nonphysical aggression for physical aggression – social bullying has increased.
   
2. Less serious forms of aggression usually track with more serious forms (nationwide, the change in assault rates runs parallel to the change in murder rates). So we can use rates of physical aggression as a proxy for rates of bullying – social bullying has decreased.
   
3. Physical and nonphysical aggression are completely unrelated, caused by different factors and in found in different places – the change in social bullying is anybody’s guess.

How Much is Three Percent?

March 11, 2010
Posted by Jay Livingston

The Freakonomics blog today assures us that emergency room overutilization is a “myth.” All that talk about the uninsured doing what George W. Bush suggested and using the emergency rooms as primary care, that’s just baseless scare tactics. Citing a Slate article, they give the data

E.R. care represents less than 3 percent of healthcare spending, only 12 percent of E.R. visits are non-urgent, and the majority of E.R. patients are insured U.S. citizens, not uninsured, illegal immigrants.

That “majority” might be 99.9% or it might be 50.1%. It turns out that the uninsured account for about 20% of E.R. visits.

My trouble is that I never know if those percents are a lot or a little. Take that 3% of spending. I’m not an economist, and although I haven't done the math, I figure that 3% of $2.3 trillion might still be a significant chunk of change. So just to make sure that 3% was in fact a pittance, a part of the “emergency room myth,” I looked for other Freakonomics articles with a similar number.

• foreclosure rates began a steady rise from 1.7 percent in 2005 to 2.8 percent in 2007. [Three percent of healthcare spending is a little; 2.8% of mortgages is a lot.]
  
• I was surprised at how high the fees were. . . . Even on big-ticket items like airline tickets, the credit-card company collects nearly 3 percent. [Three percent of healthcare spending is a little; 3% of an airline ticket is a lot.]
  
• The homeownership rate in the U.S. increased by 3 percentage points over the past decade — a clear break from the two previous decades of stagnation. [Three percent of healthcare spending is a little; 3% of homeownership is a lot.]

You get the idea. Maybe whether 3% is a lot or a little depends on its political use. I don’t follow the Freaknomics political views closely, but I’m guessing that they don’t like Hugo Chavez down in Venezuela.

opposition voters [those who opposed Chavez] experienced a 5 percent drop in earnings and a 1.5 percent drop in employment rates after their names were released. The authors also conclude that the retaliatory measures may have cost Venezuela up to 3 percent of G.D.P. due to misallocation of workers across jobs.

Chavez “may have” cost his country a whopping 3% of GDP, i.e, $9.4 billion (or possibly less -- note that “up to”). E.R. visits cost the US only a negligible 3% of healthcare spending. And the uninsured are only one-fifth of that, a mere $14 billion.

Whether 3% is a lot or a little seems to depend on your politics and what the issue is.

Unions too are bad, at least for business.

a successful unionization vote significantly decreases the market value of the company even absent changes in organizational performance. Lee and Mas run a policy simulation and conclude that, “ … a policy-induced doubling of unionization would lead to a 4.3 percent decrease in the equity value of all firms at risk of unionization.”

For a paltry increase of 100% in the number of workers getting the benefits of unionization compaines would suffer on overwhelming decrease of 4.3% decrease in equity.

Now about those 20 people in front of you in line at the emergency room. Only four of them (20%) are there because they don’t have insurance. They are part of what Freakonomics calls a “rosier picture.” I wonder if Freakonomics maybe has one or two posts where 20% is pretty big amount, something to worry about, instead of being the equivalent of a bunch of roses in the hospital.

Cooking the Books - A Second Look

February 19, 2010
Posted by Jay Livingston

Do the police undercount crime?

The graph I cribbed from Rick Rosenfeld in yesterday’s post showed a remarkable similarity between victimization surveys and official crime statistics. In 2000, for example the rate of reported burglaries according to the NCVS was nearly identical to the UCR rate. Both were about 4.4 per 1,000.

Yet in the recent Eterno-Silverman study, police commanders, responding anonymously, said that crime statistics were suppressed. And Josh in his comment yesterday refers to Peter Moskos’s “let me count the ways” description of how the police keep crimes off the books. (See Moskos’s own take on the study at his website.)

The problem is that the graph I presented was somewhat misleading The NCVS and UCR rates of burglary do not measure exactly the same thing. It’s not exactly oranges and apples; more like oranges and tangerines.

1. The NCVS data are for the New York metro area, so we have to use similar UCR data even though the rap about fudging the stats is only about the NYPD. No way to get around that problem

2. More crucially, the NCVS counts only residential burglaries; the UCR number includes both commercial and residential burglaries. Nationwide, about 2/3 of all UCR burglaries are residential. Using that figure for the New York area we get a UCR rate for Residential burglaries of only 3.0 per 1,000 population, about one-third less than we would expect from the estimate of the number of residential burglaries that victims say they reported. Here’s an amended graph. I’ve added a line for residential burglaries that uses the simple 2/3 formula.

(Click on the graph for a larger view.)

The rate of residential burglaries that victims say that they report is usually one-and-a-half to two times greater than the rate of residential burglaries officially “known to the police.” For the year 2000, the NCVS rate of 4.4 per 1,000 population works out to 40,000 reported residential burglaries. If 2/3 of burglaries are residential, only 27,500 of those made it onto the police books.

Does that mean that the police canned 12,5000 reported burglaries? Probably not. There may be other explanations for the some of the discrepancy. But the data do provide some support for those who are skeptical of the precision of the police numbers.

Cooking the Crime Books?

February 18, 2010
Posted by Jay Livingston

“Crimes known to the police” is the official count of Crime in the United States – the annual report published by the FBI, which compiles data from local police departments. It’s also known as the Uniform Crime Reports (UCR).

Many years ago, a friend of mine found that his car had been broken into and wanted to report the crime to the police. He went to the local precinct, and when the desk sergeant finally acknowledged him, he said, “Someone broke into my car and stole my stuff.”

“So what do you want me to do?” said the sergeant.

That was one larceny that never became “known to the police,” at least not on the books of the 20th precinct.

The problem of uncounted crime has been around a long time. In the late 1940s, New York’s burglary rate grew by 1300% in a single year, a huge increase but entirely attributable to changes in bookkeeping. Word had gone out that burglaries should no longer be routinely assigned to “Detective Can.”

In the 1980s, Chicago’s robbery rate rose after the FBI threatened the city that it wouldn’t include their data because the numbers were so suspect. Atlanta kept its numbers artificially low prior to the Olympics. This week, the Dallas police chief is under attack for the way his department reports crimes.

Now two criminologists, John Eterno and Eli Silverman, are claiming that New York’s crime data have been fudged consistently for the last 15 years, and they point to CompStat as the culprit (NY Times article here.) CompStat is the system that William Bratton brought to New York when he became police commissioner in 1994. It required commanders to report every week on statistics and patterns of crime in their areas.

Eterno and Silverman gave anonymous surveys to retired precinct commanders, Under pressure to appear effective in the war on crime, precinct commanders might stretch the facts. The value of a theft might be creatively investigated to keep the total under the $1000 threshold a misdemeanor and the felony known as “grand larceny.” Felonies look worse on your statistical report.

A purse snatch might get recorded as a theft instead of a robbery because robberies fall into the broader category of “violent” crimes. Or victims, like my friend in the old days, might be persuaded not to bother reporting the crime.

In an op-ed in the Times yesterday, Bratton vigorously defended the NYPD numbers. He provided no data, but he could have.

Since 1973, the US has had an alternate count of crime, the National Crime Victimization Survey. Most of the data are for the US, but Rick Rosenfeld and Janet Lauritsen were able to get three-year averages for New York City, and they have looked at the data for burglary.

(Click on the graph for a larger view.)

The graph shows the rate (percents) of

• people who told the NCVS they had been victims of a burglary
• people who say they reported the burglary to the police
• the official rate of burglaries “known to the police”

The numbers are not precisely comparable (the NCVS rate may be based on households rather than population, and the UCR rate includes commercial burglaries as well as residential). But the data in the graph do not support the idea that CompStat increased the fudging of burglary statistics If it had, then starting in 1994, we should see a widening gap between the NCVS line and the UCR line, with the UCR line moving downward much more. But if anything, it’s the Victimization line that descends more steeply.

In the decade following CompStat, both sources of data show a 68% decrease in burglary. So if commanders were cooking the books, they weren't including burglary in the recipe.

What Was the Question?

February 5, 2010
Posted by Jay Livingston

Survey questions may seem straightforward, but especially if the poll is a one-off, with questions that haven’t been used in other polls, you can’t always be sure how the respondents interpret them.

The Kos/Research 2000 poll of Republicans has been getting some notice, and no wonder. At first glance, it seems to show that one of our two major political parties is home to quite a few people who are not fully in touch with reality, especially when Obama is in view.

Do you believe Barack Obama is a racist who hates White people?
Yes 31
No 36
Not Sure 33


Do you believe Barack Obama wants the terrorists to win?
Yes 24
No 43
Not Sure 33


Should Barack Obama be impeached, or not?
Yes 39
No 32
Not Sure 29

I’m not sure what the results mean. Self-identified Republicans are about 25% of the electorate.* If one-third of them hold views that are “ludicrous” (Kos’s term), that’s still only 8% of the voters.

But what about non-ludicrous Republicans. Suppose you were a mainstream conservative and Research 2000 phoned you. To find out, I put some of the questions to a Republican I know – non-ludicrous (he reads the Wall Street Journal, he doesn’t watch Glenn Beck.)

Do you believe Sarah Palin is more qualified to be President than Barack Obama? (In the survey, 53% said, “yes.”)

Such a loaded question! I think she's nuts and he's sane – but in principle, she's right and he's wrong about most issues.

Do you believe Barack Obama wants the terrorists to win?

They don't WANT terrorists to win – no – but they don't care as much about the battle as most Americans do.

He might have said Yes to the interviewer just because he thought a Yes was more in line with the spirit of the question than with its actual wording. Or he would have refused to answer (and possibly have been put in the “Not sure” category?)

So the questions are more ambiguous than they seem, even on close reading.

Should public school students be taught that the book of Genesis in the Bible explains how God created the world?
Seventy-seven per cent of the sample said, “Yes.” And Kos, who commissioned the poll in connection with his book – to be called American Taliban – will see that result as rabid pro-creationism and anti-science. But re-read the actual question. Here’s what my sane Republican had to say:

This one's easy:
Absolutely yes. “public school students should be taught” a lot of important facts about our culture and civilization – that the Greeks invaded Ilium and destroyed Troy, that Confucius was the inspiration for a great religion, that Thomas A. Edison invented the electric light bulb, that Darwin in his Origin of the Species explained how animals change according to the process of natural selection, and “that the book of Genesis in the Bible explains how God created the world.” Why the hell not teach that fact? Who could say no to that?

Who indeed? Not me.

-----------------------
* The poll may have oversampled the fringe (see Emily Swanson at Pollster ), but those folks at the fringe are more likely to be active at the local level, so it’s possible they’ll swing some weight at the national level too. Their preferred candidate is, of course, Sarah Palin. So while political scientists think the poll may be exaggerating the far right (see Joshua Tucker’s excellent critique at The Monkey Cage), the Palinstas are hailing the poll as spot on.

Correlation and Cause - Feeding and Breeding

January 25, 2010
Posted by Jay Livingston

Andre Bauer’s idea that poor people are like stray animals is what will get most of the attention, as I suppose it should. Bauer* is running for governor of the enlightened state of South Carolina, where Appalachian Trail hiker Mark Sanford is still in that office.** Bauer is Lt. Gov., and here’s what he said à propos programs for free and reduced-price lunches in the public schools.

My grandmother was not a highly educated woman, but she told me as a small child to quit feeding stray animals. You know why? Because they breed. You're facilitating the problem if you give an animal or a person ample food supply. They will reproduce, especially ones that don't think too much further than that. And so what you've got to do is you've got to curtail that type of behavior. They don't know any better,

Bauer stands by his analogy and says he was quoted out of context. Right.

Obviously, Bauer did not take Sociology of Poverty. Of less importance politically is that he also skipped the methods course. Apparently, he has some data – a bar graph – but he mistakes correlation for cause.

I can show you a bar graph where free and reduced lunch has the worst test scores in the state of South Carolina. You show me the school that has the highest free and reduced lunch, and I'll show you the worst test scores, folks. It's there, period.

I suppose that it is somehow possible that providing food for impoverished kids makes them dumb. Maybe electing people to office in the Palmetto State has a similar effect.

*Lt. Gov. Andre Bauer is not to be confused with Andre Braugher, the excellent actor who plated Detective Pembleton on “Homicide” (the forerunner to “The Wire”) and is currently in “Men of a Certain Age.” Pictures below. You figure out which Andre is which.


** What’s up with The Palmetto State and its public servants? Lt. Gov. Bauer is incautious not just in his campaign speeches. He also tends to get stopped for speeding, and he once crash-landed a small plane. (CSM article here.) Then, besides Sanford and Bauer, there’s the former chair of the SC Board of Education, who home schooled her kids, believes that “intelligent design” and “abstinence only” should be taught in the schools, and resigned only when it was revealed that she also publishes online porn (oops, I mean erotic fiction.) The story and links to her very NSFW prose are here. I guess she just wanted to put the palm back in palmetto.

Sexting and Percentaging - The Wrong Way

December 23, 2009
Posted by Jay Livingston

The Pew survey on sexting – it came out over a week ago. I don’t know how I missed it. Must be the holiday blahs. And where was the media hysteria? Most news outlets ignored it, probably because the results weren’t all that alarming.

For the entire sample of eight hundred 12-17 year olds, the estimated proportion who sent “sexually suggestive nude or nearly nude images of themselves” was 4%. Given the margin of error, that means that the actual percentage, as Dan Ryan at Sociology of Information notes, is somewhere between 0% and 8%.

Of course, we’re not going to see a headline like “Sexting Teens May be 0%.” Not when you can goose up the numbers to 30%. Here’s the headline that ran in The Washington Post:

Sexting hasn't reached most young teens, poll finds;
30% of 17-year-olds report getting nude photos on their cells

That subhead manages to get the highest percentage by

• using only the oldest subgroup in the sample
• measuring receiving rather than sending

Dan has some other methodological criticisms, including this one. First the Pew summary paragraph:

One parental intervention that may relate to a lower likelihood of sending of sexually suggestive images was parental restriction of text messaging. Teens who sent sexually suggestive nude or nearly nude images were less likely to have parents who reported limiting the number of texts or other messages the teen could send. Just 9% of teens who sent sexy images by text had parents who restricted the number of texts or other messages they could send; 28% of teens who didn’t send these texts had parents who limited their child’s texting.

I spent the last two weeks of the semester trying to get students to percentage tables correctly. “Percentage on the independent variable,” I repeated and repeated. And now Amanda Lenhart at the Pew Foundation undermines all my good work. As Dan says,

It is unlikely that the authors are thinking that sexting causes parental restrictions – the sense is just the opposite – and so the percentaging should be within the categories of parental behavior and comparison across these.

Dan even does the math and finds:

• Children of restrictive parents who ever sent a sext: 1.4% (3 of 218)
• Children of non-restrictive parents who ever sent a sext: 5% (29 of 572)

Read Dan’s entire critique. Or for the truly absurd and probably counter-effectual, see the anti-sexting videos featuring (I am not making this up) James Lipton.

March Madness

March 18, 2009
Posted by Jay Livingston

“When Losing Leads to Winning.” That’s the title of the paper by Jonah Berger and Devin Pope. In the New York Times recently, they put it like this:

Surprisingly, the data show that trailing by a little can actually be a good thing.
Take games in which one team is ahead by a point at the half. . . . The team trailing by a point actually wins more often and, relative to expectation, being slightly behind increases a team’s chance of winning by 5 percent to 7 percent.

They had data on over 6500 NCAA games in four seasons. Here’s the key graph.

(Click on the chart for a larger view.)

The surprise they refer to is in the red circle I drew. The dot one point to the left of the tie-game point is higher than the dot one point to the right. Teams behind by one point at the half won 51.3% of the games; teams leading by a point won only 48.7.*

Justin Wolfers** at Freakonomics reprints the graph and adds that Berger and Pope are “two of the brightest young behavioral economists around.”

I’m not a bright behavioral economist, I’m not young, I’m not a methodologist or a statistician, and truth be told, I’m not much of an NCAA fan. But here’s what I see. First of all, the right half of the graph is just the mirror image of the left. If teams down by one win 51.3%, teams ahead by one have to lose 51.3%, and similarly for every other dot on the chart.

Second, this is not the only discontinuity in the graph. I’ve put yellow squares around the others.


Teams down by 7 points at the half have a slightly higher win percentage than do teams down by 6. By the same graph-reading logic, it’s better to be down by 4 points than by only 3. And the percentage difference for these points is greater than the one-point/tie-game difference.

Then, what about that statement that being down by one point at the half “ increases a team’s chance of winning by 5 percent to 7 percent”? Remember, those teams won 51.3% of the games. How did 1.3 percentage points above 50-50 become a 5-7% increase? You have to read the fine print: “relative to expectation.” That expectation is based on a straight-line equation presumably derived from the ten data points (all the score differentials from 10 points to one – no sense in including games tied at the half). That model predicts that teams down by one at the half will win only 46% of the time. Instead, they won 51.3%.

Berger and Pope’s explanation of their finding is basically the Avis factor. The teams that are behind try harder. Maybe so, but that doesn’t explain the other discontinuities in the graph. Using this logic, we would conclude that teams behind by seven try harder than teams behind by six. But teams behind by 5 don’t try harder than teams behind by four. And so on. Why do only some point deficits produce the Avis effect?


* Their results are significant at the .05 level. With 6500 games in the sample, I’d bet that any difference will turn out to be statistically significant, though the authors don’t say how many of those 6500 games had 1-point halftime differences.

**Wolfers himself is the author of another economics journal article on basketball, a study purporting to reveal unwitting racism among NBA referees. In that article as well, I thought there might be less there than meets the economist’s eye.

Muslims and Methodology

May 26, 2007
Posted by Jay Livingston

The Pew Research Center this week published a poll that asked Muslims in the US and other countries their views on several political issues. News stories here focused on the US results, but whether those results were cause for relief or alarm depends on who was reading the poll.

The mainstream press (aka “the mainstream liberal press”), ran headlines like these:
In many ways, US Muslims are in mainstream America (Christian Science Monitor)
Muslim Americans in line with US values (Financial Times)
Survey: U.S. Muslims Assimilated, Opposed to Extremism (The Washington Post)

The right-wing press picked up on one number in the poll:
TIME BOMBS IN OUR MIDST - 26% OF YOUNG U.S. MUSLIMS BACK KILLINGS (The New York Post)

Or as the Washington Times put it in an op-ed piece by Diana West, “According to Pew's data, one-quarter of younger American Muslims approve of the presence of skin-ripping, skull-crushing, organ-piercing violence in civilian life as a religious imperative —‘in defense of Islam.’”

For some on the right, 26% was a lowball estimate. Here’s Rush Limbaugh:

“Two percent of them say it can often be justified, 13% say sometimes, and 11% say rarely.” So let’s add it up, 26 and 2 is 28, so 31% think to one degree or another, suicide bombing is justified. If you add to that the 5% that don't know or refuse to answer, it's even worse. So almost a third of young American Muslims who support in one way or another homicide bombings according to the Pew poll.

(If Limbaugh had taken a basic statistics course, he could have inflated his estimate even more. There were only about 300 young Muslims in the sample, so the margin of error means that the true proportion might have been several percentage points higher.)

When a result can be open such different interpretation, maybe there’s something wrong with the survey methodology. Let’s take a look at the actual question:

Some people think that suicide bombing and other forms of violence against civilian targets are justified in order to defend Islam from its enemies. Other people believe that, no matter what the reason, this kind of violence is never justified. Do you personally feel that this kind of violence is often justified to defend Islam, sometimes justified, rarely justified, or never justified?

The conclusion of Limbaugh and others is that unless you say that killing is never ever justified, you’re a menace to society.

But what would happen if we substituted “Christianity” for “Islam” and polled Christians in the US? How many Christians would absolutely reject all violence in defense of Christianity? And how many might say that violence, even violence against civilians, is sometimes justified to defend Christianity from its enemies? I wouldn’t be surprised if the number were higher than 26%.

Consider the war in Iraq, which Limbaugh and the others support. Hundreds of thousands of civilians have been killed, several thousand of them as “collateral damage” in US operations. The “shock and awe” bombing in the original invasion certainly included “skin-ripping, skull-crushing, organ-piercing violence” upon civilians. But at the time, a majority of Americans supported it, and Bush’s position still remains that we went to war to defend our freedom against its enemies.

The survey has many more interesting findings, and it’s especially useful for comparing US Muslims with those of other countries. But of course those matters were not newsworthy.

Asking About Housework

October 20, 2006

Posted by Jay Livingston

The working mother — how does she find the time? Did being a worker mean that she would spend less time being a mom? A new study by Suzanne Bianchi finds that contrary to expectations some years back, work did not reduce the time mothers spent with their kids. In fact, parents — both moms and dads— are spending more time with their kids than parents did in previous generations. What’s been cut back is housework. (NPR did a longish (16 minute) report on the study — an interview with one of the authors, calls from listeners— which you can find here.)

There’s much to be said and blogged about Bianchi’s findings, but I want to make one small methodological observation, something I’ve mentioned to students. Some questions have a built-in “social desirability” bias. Suppose you want to know about reading habits. It’s socially desirable to have read more books (at least I hope it still is), so if you ask “How many books do you read a year?” or “How many books did you read last year?” you’re likely to get something higher than the actual number. Instead, you ask, “Did you read a book last week?” A person who never reads a book might be reluctant to say that he hadn’t read a single book last year. But there’s no social stigma attached to not having read a book last week.

The same thing goes for housework and parenting. Ask me how many hours I spend on housework and childcare each week, even though as a good friend of social research I’d try to be accurate, I’d probably try to be accurate on the good side. So as the Times reports, “Using a standard set of questions, professional interviewers asked parents to chronicle all their activities on the day before the interview.” (The study notes that we dads are doing more of both than fathers of only a few years ago.)

(More later. Right now, I have to put the wash in the drier, start making dinner, and help my son with his homework.)

Negative Results

September 20, 2006

Posted by Jay Livingston

A man gets thrown into a jail cell with a long-term occupant and then begins a series of attempts to escape, each by some different method. He fails every time, getting captured and thrown back in the cell. The older prisoner looks at him silently after each failure. Finally, after six or seven attempts, the man loses his patience with the old prisoner and says “Well, couldn’t you help me a little?” “Oh,” says the old guy, “I’ve tried all the ways you thought of—they don’t work.” “Well why the hell didn’t you tell me?!” shouts the man. “Who reports negative results?” says the old prisoner.

thanks to sociologist and blogger Kieran Healy (http://www.kieranhealy.org/blog/).

I hadn’t heard the joke before, but I’ve certainly heard of the bias towards positive results. Back when I was in graduate school, one of my professors, an experimental social psychologist, proposed that journals evaluate papers solely on the basis of research design. Researchers would submit all the preliminary stuff including the design but not the results. Then, if they accepted the article, it would be published regardless of whether the results showed the intended effects.

Healy used the joke in connection with an article on biases in political science journals. Articles which just make the p <.05 level are much more likely to be published than are those that just miss it. I’m not sure if political science and other disciplines (like sociology) that rely on survey data could use the same strategy on deciding on publication before they see the data. That strategy may be more applicable to experiments rather than surveys.

I find it interesting, even ironic, that my social psych professor who proposed this soon became very well known for his own experimental study, whose results were widely discussed even outside academia. But statisticians reviewing the data claimed that he had used the wrong statistical analyses in order to make his results look significant. His idea might be right, the critics said —in fact they probably hoped it was right — but the numbers in the study didn’t prove it. The professor and others claimed that the numbers did support the idea and defended their analyses. Clearly, it was a case that needed replication studies, lots of them. But I’m not sure what attempts to replicate the study have been done, nor do I know what the results have been. But I am fairly certain that researchers who attempted to replicate and got negative results had a harder time getting published than did those who got positive results.

This professor also had our class do a replication of one of his experiments. It didn’t work. In fact, the strongest correlation was with a variable that by design was randomized. There were two versions of some test, A and B, distributed randomly to seats in a room. We wanted to see if the two versions of the test produced different results. People came in, sat down, and did the test. But it turned out that the strongest correlation was between sex and test version. That is, the A version wound up being taken mostly by girls, the B version by boys, just by accident of where they chose to sit. No other difference between the two versions was nearly so strong. It made me a bit skeptical about the whole enterprise.