Sexting and Percentaging - The Wrong Way

December 23, 2009
Posted by Jay Livingston

The Pew survey on sexting – it came out over a week ago. I don’t know how I missed it. Must be the holiday blahs. And where was the media hysteria? Most news outlets ignored it, probably because the results weren’t all that alarming.

For the entire sample of eight hundred 12-17 year olds, the estimated proportion who sent “sexually suggestive nude or nearly nude images of themselves” was 4%. Given the margin of error, that means that the actual percentage, as Dan Ryan at Sociology of Information notes, is somewhere between 0% and 8%.

Of course, we’re not going to see a headline like “Sexting Teens May be 0%.” Not when you can goose up the numbers to 30%. Here’s the headline that ran in The Washington Post:

Sexting hasn't reached most young teens, poll finds;
30% of 17-year-olds report getting nude photos on their cells

That subhead manages to get the highest percentage by

• using only the oldest subgroup in the sample
• measuring receiving rather than sending

Dan has some other methodological criticisms, including this one. First the Pew summary paragraph:

One parental intervention that may relate to a lower likelihood of sending of sexually suggestive images was parental restriction of text messaging. Teens who sent sexually suggestive nude or nearly nude images were less likely to have parents who reported limiting the number of texts or other messages the teen could send. Just 9% of teens who sent sexy images by text had parents who restricted the number of texts or other messages they could send; 28% of teens who didn’t send these texts had parents who limited their child’s texting.

I spent the last two weeks of the semester trying to get students to percentage tables correctly. “Percentage on the independent variable,” I repeated and repeated. And now Amanda Lenhart at the Pew Foundation undermines all my good work. As Dan says,

It is unlikely that the authors are thinking that sexting causes parental restrictions – the sense is just the opposite – and so the percentaging should be within the categories of parental behavior and comparison across these.

Dan even does the math and finds:

• Children of restrictive parents who ever sent a sext: 1.4% (3 of 218)
• Children of non-restrictive parents who ever sent a sext: 5% (29 of 572)

Read Dan’s entire critique. Or for the truly absurd and probably counter-effectual, see the anti-sexting videos featuring (I am not making this up) James Lipton.

March Madness

March 18, 2009
Posted by Jay Livingston

“When Losing Leads to Winning.” That’s the title of the paper by Jonah Berger and Devin Pope. In the New York Times recently, they put it like this:

Surprisingly, the data show that trailing by a little can actually be a good thing.
Take games in which one team is ahead by a point at the half. . . . The team trailing by a point actually wins more often and, relative to expectation, being slightly behind increases a team’s chance of winning by 5 percent to 7 percent.

They had data on over 6500 NCAA games in four seasons. Here’s the key graph.

(Click on the chart for a larger view.)

The surprise they refer to is in the red circle I drew. The dot one point to the left of the tie-game point is higher than the dot one point to the right. Teams behind by one point at the half won 51.3% of the games; teams leading by a point won only 48.7.*

Justin Wolfers** at Freakonomics reprints the graph and adds that Berger and Pope are “two of the brightest young behavioral economists around.”

I’m not a bright behavioral economist, I’m not young, I’m not a methodologist or a statistician, and truth be told, I’m not much of an NCAA fan. But here’s what I see. First of all, the right half of the graph is just the mirror image of the left. If teams down by one win 51.3%, teams ahead by one have to lose 51.3%, and similarly for every other dot on the chart.

Second, this is not the only discontinuity in the graph. I’ve put yellow squares around the others.


Teams down by 7 points at the half have a slightly higher win percentage than do teams down by 6. By the same graph-reading logic, it’s better to be down by 4 points than by only 3. And the percentage difference for these points is greater than the one-point/tie-game difference.

Then, what about that statement that being down by one point at the half “ increases a team’s chance of winning by 5 percent to 7 percent”? Remember, those teams won 51.3% of the games. How did 1.3 percentage points above 50-50 become a 5-7% increase? You have to read the fine print: “relative to expectation.” That expectation is based on a straight-line equation presumably derived from the ten data points (all the score differentials from 10 points to one – no sense in including games tied at the half). That model predicts that teams down by one at the half will win only 46% of the time. Instead, they won 51.3%.

Berger and Pope’s explanation of their finding is basically the Avis factor. The teams that are behind try harder. Maybe so, but that doesn’t explain the other discontinuities in the graph. Using this logic, we would conclude that teams behind by seven try harder than teams behind by six. But teams behind by 5 don’t try harder than teams behind by four. And so on. Why do only some point deficits produce the Avis effect?


* Their results are significant at the .05 level. With 6500 games in the sample, I’d bet that any difference will turn out to be statistically significant, though the authors don’t say how many of those 6500 games had 1-point halftime differences.

**Wolfers himself is the author of another economics journal article on basketball, a study purporting to reveal unwitting racism among NBA referees. In that article as well, I thought there might be less there than meets the economist’s eye.

Muslims and Methodology

May 26, 2007
Posted by Jay Livingston

The Pew Research Center this week published a poll that asked Muslims in the US and other countries their views on several political issues. News stories here focused on the US results, but whether those results were cause for relief or alarm depends on who was reading the poll.

The mainstream press (aka “the mainstream liberal press”), ran headlines like these:
In many ways, US Muslims are in mainstream America (Christian Science Monitor)
Muslim Americans in line with US values (Financial Times)
Survey: U.S. Muslims Assimilated, Opposed to Extremism (The Washington Post)

The right-wing press picked up on one number in the poll:
TIME BOMBS IN OUR MIDST - 26% OF YOUNG U.S. MUSLIMS BACK KILLINGS (The New York Post)

Or as the Washington Times put it in an op-ed piece by Diana West, “According to Pew's data, one-quarter of younger American Muslims approve of the presence of skin-ripping, skull-crushing, organ-piercing violence in civilian life as a religious imperative —‘in defense of Islam.’”

For some on the right, 26% was a lowball estimate. Here’s Rush Limbaugh:

“Two percent of them say it can often be justified, 13% say sometimes, and 11% say rarely.” So let’s add it up, 26 and 2 is 28, so 31% think to one degree or another, suicide bombing is justified. If you add to that the 5% that don't know or refuse to answer, it's even worse. So almost a third of young American Muslims who support in one way or another homicide bombings according to the Pew poll.

(If Limbaugh had taken a basic statistics course, he could have inflated his estimate even more. There were only about 300 young Muslims in the sample, so the margin of error means that the true proportion might have been several percentage points higher.)

When a result can be open such different interpretation, maybe there’s something wrong with the survey methodology. Let’s take a look at the actual question:

Some people think that suicide bombing and other forms of violence against civilian targets are justified in order to defend Islam from its enemies. Other people believe that, no matter what the reason, this kind of violence is never justified. Do you personally feel that this kind of violence is often justified to defend Islam, sometimes justified, rarely justified, or never justified?

The conclusion of Limbaugh and others is that unless you say that killing is never ever justified, you’re a menace to society.

But what would happen if we substituted “Christianity” for “Islam” and polled Christians in the US? How many Christians would absolutely reject all violence in defense of Christianity? And how many might say that violence, even violence against civilians, is sometimes justified to defend Christianity from its enemies? I wouldn’t be surprised if the number were higher than 26%.

Consider the war in Iraq, which Limbaugh and the others support. Hundreds of thousands of civilians have been killed, several thousand of them as “collateral damage” in US operations. The “shock and awe” bombing in the original invasion certainly included “skin-ripping, skull-crushing, organ-piercing violence” upon civilians. But at the time, a majority of Americans supported it, and Bush’s position still remains that we went to war to defend our freedom against its enemies.

The survey has many more interesting findings, and it’s especially useful for comparing US Muslims with those of other countries. But of course those matters were not newsworthy.

Asking About Housework

October 20, 2006

Posted by Jay Livingston

The working mother — how does she find the time? Did being a worker mean that she would spend less time being a mom? A new study by Suzanne Bianchi finds that contrary to expectations some years back, work did not reduce the time mothers spent with their kids. In fact, parents — both moms and dads— are spending more time with their kids than parents did in previous generations. What’s been cut back is housework. (NPR did a longish (16 minute) report on the study — an interview with one of the authors, calls from listeners— which you can find here.)

There’s much to be said and blogged about Bianchi’s findings, but I want to make one small methodological observation, something I’ve mentioned to students. Some questions have a built-in “social desirability” bias. Suppose you want to know about reading habits. It’s socially desirable to have read more books (at least I hope it still is), so if you ask “How many books do you read a year?” or “How many books did you read last year?” you’re likely to get something higher than the actual number. Instead, you ask, “Did you read a book last week?” A person who never reads a book might be reluctant to say that he hadn’t read a single book last year. But there’s no social stigma attached to not having read a book last week.

The same thing goes for housework and parenting. Ask me how many hours I spend on housework and childcare each week, even though as a good friend of social research I’d try to be accurate, I’d probably try to be accurate on the good side. So as the Times reports, “Using a standard set of questions, professional interviewers asked parents to chronicle all their activities on the day before the interview.” (The study notes that we dads are doing more of both than fathers of only a few years ago.)

(More later. Right now, I have to put the wash in the drier, start making dinner, and help my son with his homework.)

Negative Results

September 20, 2006

Posted by Jay Livingston

A man gets thrown into a jail cell with a long-term occupant and then begins a series of attempts to escape, each by some different method. He fails every time, getting captured and thrown back in the cell. The older prisoner looks at him silently after each failure. Finally, after six or seven attempts, the man loses his patience with the old prisoner and says “Well, couldn’t you help me a little?” “Oh,” says the old guy, “I’ve tried all the ways you thought of—they don’t work.” “Well why the hell didn’t you tell me?!” shouts the man. “Who reports negative results?” says the old prisoner.

thanks to sociologist and blogger Kieran Healy (http://www.kieranhealy.org/blog/).

I hadn’t heard the joke before, but I’ve certainly heard of the bias towards positive results. Back when I was in graduate school, one of my professors, an experimental social psychologist, proposed that journals evaluate papers solely on the basis of research design. Researchers would submit all the preliminary stuff including the design but not the results. Then, if they accepted the article, it would be published regardless of whether the results showed the intended effects.

Healy used the joke in connection with an article on biases in political science journals. Articles which just make the p <.05 level are much more likely to be published than are those that just miss it. I’m not sure if political science and other disciplines (like sociology) that rely on survey data could use the same strategy on deciding on publication before they see the data. That strategy may be more applicable to experiments rather than surveys.

I find it interesting, even ironic, that my social psych professor who proposed this soon became very well known for his own experimental study, whose results were widely discussed even outside academia. But statisticians reviewing the data claimed that he had used the wrong statistical analyses in order to make his results look significant. His idea might be right, the critics said —in fact they probably hoped it was right — but the numbers in the study didn’t prove it. The professor and others claimed that the numbers did support the idea and defended their analyses. Clearly, it was a case that needed replication studies, lots of them. But I’m not sure what attempts to replicate the study have been done, nor do I know what the results have been. But I am fairly certain that researchers who attempted to replicate and got negative results had a harder time getting published than did those who got positive results.

This professor also had our class do a replication of one of his experiments. It didn’t work. In fact, the strongest correlation was with a variable that by design was randomized. There were two versions of some test, A and B, distributed randomly to seats in a room. We wanted to see if the two versions of the test produced different results. People came in, sat down, and did the test. But it turned out that the strongest correlation was between sex and test version. That is, the A version wound up being taken mostly by girls, the B version by boys, just by accident of where they chose to sit. No other difference between the two versions was nearly so strong. It made me a bit skeptical about the whole enterprise.