How to Misread a Graph (It’s Not Easy, but The Heritage Foundation Finds a Way)

September 20, 2013
Posted by Jay Livingston

My post of a few days ago (here) showed The Heritage Foundation presenting a graph and deliberately drawing a conclusion that the graph clearly showed to be wrong.  Apparently, that’s something of a specialty at The Heritage Foundation.

Here’s their graphic purporting to show that preschool programs don’t work. (The original is here.)

The problem in the Oklahoma graph is the lag time between cause and effect.  For example, the baby boom began in 1947, but we would not look for its effects on healthcare and Social Security costs until much, much later.

Most people know this, but  Heritage seems to be lagging behind. “Fourth grade reading achievement scores in Oklahoma have actually declined.” True, they are lower now than in 1998, when universal preschool started. But is that the year should we use for a starting point for data on fourth grade reading scores?

Pre-school kids are three or four years old.  They don’t take the fourth-grade reading test until six or seven years later – in Oklahoma, that would be 2005 for the first cohort.  Amazingly (amazing to Heritage, I guess), that was the year reading scores began to increase, and despite a slight dip last year, they are still above that level.

As for the Georgia graph, anyone glancing at it (anyone except for people at The Heritage Foundation) would see this: reading scores in Georgia began increasing in 1995, two years after universal preschool began, and continued to rise when the first preschoolers reached fourth grade; scores have continued to rise faster than the national average.  Georgia was behind, now it’s ahead. Something good has been happening.

Heritage, however, manages not to see this and instead complains about how long it took Georgia to reach that point. (“Georgia’s program was in place for 13 years before scores caught up to the U.S. average.”)

A simple graph of scores is not really an adequate assessment of universal preschool. Those assessments, which include many other relevant variables,* have been done, and they generally conclude that the programs do work.  But that’s not the point.  The point is that Heritage is again misreading its own graph. So again I repeat, “Who you gonna believe, the Heritage Foundation or your lyin’ eyes?”

HT: Philip Cohen, who apparently thinks the Heritage deliberate obtuseness is so obvious as to be unworthy of elaboration.

-----------

* These include the usual demographics, especially to see if preschool effects are different for different groups. But there’s also the problem of post-preschool education. A state might have great preschools, but if it also has lousy primary schools, the benefits of preschool will be eroded away by the time the kids are in fourth grade.

Anecdotal Evidence – One More Time

June 14, 2013
Posted by Jay Livingston

Anecdotal evidence seems more convincing, I tell my students in Week One, but if you want to find out general truths, you need systematic evidence.  The New York Times today provides my example for next semester.

The Times had run an op-ed  last week about only children. The author, Lauren Sandler, referred to results from “hundreds of studies” showing that only children are generally no different from those with siblings on variables like “leadership, maturity, extroversion, social participation, popularity, generosity, cooperativeness, flexibility, emotional stability, contentment.” Nor were they more self-involved or lonelier.  And they score higher on measures of intelligence and achievement.   

Today, the Times printed a letter challenging these conclusions.  

 Another problem with these studies is that they put families in boxes: the only-child box, the divorced-parent box, the single-mother box — all of which I am in. They oversimplify family situations. I have seen the offspring of single divorced mothers grow up happy and successful, and I have seen children of two-parent families turn out disastrously.

Regarding the precocity of only children, my granddaughter at 2, like Ms. Sandler's daughter, could tell the difference between the crayon colors magenta and pink, and she is not an only child. So much for boxes.

Or as a student will usually ask, “But doesn’t it depend on the individual?”

Yes, I say.  But scientific generalizations do not apply 100% to everyone in that box.  Are men taller than women?  Are smokers less healthy than non-smokers?   Of course. Yes, there’s Maria Sharapova and the WNBA, and there are no doubt thousand of pack-a-day octogenarians.  Does that mean we should throw categories (i.e., boxes)  like Sex and Smoking in the trash?

As the letter writer says, categories simplify. They overlook differences. But categories are inevitable. Pineapple is a category. We know that not all pineapples are alike, and yet we talk about pineapples.  And men.  And smokers. And divorced mothers and only children.

I’m not surprised that my students – 18-year old freshmen or transfers from the community colleges – need this brief reminder. But the New York Times?

In any case, the concern over the problems of only children seems to be fading, though I'm not sure how to interpret that.  The Google n-grams graph of the phrase in books looks like this: 

The first decline in the phrase only children runs parallel to the baby boom (though it starts a few years earlier) and the burgeoning of multi-child families.  But the second decline comes in a period when multi-child families are decreasing.  Perhaps there is less concern because single-child families have become frequent rather than freakish. 

Wanted – Bad Research

April 22, 2013
Posted by Jay Livingston

I’m not a research director.  But if I were, I hope I wouldn’t write questions that are obviously designed to bias the results.*  And if I did ask such questions, I wouldn’t boast about it in the newspaper, especially if my stacking of the deck got barely a majority to give the answer I wanted. 

But then, I’m not Michael Saltsman, research director for the Employment Policies Institute, whose letter to the Record (formerly known as The Bergen Record) was published today.

Regarding "Most favor minimum wage hike" (Page L-7, April 18):

The recent Rutgers-Eagleton poll finding that 76 percent of New Jerseyans support a minimum wage increase only proves that incomplete poll questions yield misleading results.

My organization commissioned ORC International to conduct a similar poll regarding an increase in the minimum wage. When respondents were informed of the unintended consequences of minimum wage hikes — particularly how such hikes make it more difficult for the least-skilled to find work— 70 percent support flipped to 56 percent opposition. [emphasis added]

This consequence isn't a hypothetical: Fully 85 percent of the most credible economic studies from the past two decades indicate a loss of job opportunities following a wage hike.

Michael Saltsman
Washington, D.C. , April 18

As for the facts on the effects of an increase in the minimum wage, Saltsman’s literature review is on a par with his questionnaire construction.  Apparently he missed John Schmitt’s CEPR article from two months ago (here).    The title pretty much sums it up:

Why Does the Minimum Wage Have No Discernible Effect on Employment?

Schmitt includes this graph of minimum-wage effects from a meta-analysis.

Hristos Doucouliagos and T. D. Stanley (2009) conducted a meta-study of 64 minimum-wage studies published between 1972 and 2007 measuring the impact of minimum wages on teenage employment in the United States. When they graphed every employment estimate contained in these studies (over 1,000 in total), weighing each estimate by its statistical precision, they found that the most precise estimates were heavily clustered at or near zero employment effects.

Schmitt offers several guesses as to why employers don’t cut jobs when the minimum wage rises – maybe they raise prices, or accept a lower profit margin, or reduce the wages of better-paid employees; or maybe the increased minimum wage brings more customers, and so on.**

But regardless of the findings on minimum wage, Saltsman’s letter carries a more important if depressing message.  We try to teach our students to design good research.  We tell them that good research skills might help them get jobs.  Yet here is an example of a research-director job that depends on designing bad surveys and doing bad research. 
                                                           
------------------------------
*In his methods course, my colleague Chris Donoghue uses a made-up abortion item for teaching items that introduce bias:
“Every year in the US, over a million babies are killed by abortion. Do you agree that laws should make it more difficult to get an abortion?”

** Brad Plumer at WaPo’s WonkBlog has more on this, including a fuller discussion of Schmitt’s paper (here).

What Would You Do?

December 27, 2012
Posted by Jay Livingston

When you ask a “what if” question, can you take people’s responses at face value?

A student sent me a link to a study that asked whether Americans or Turks were more likely to act on principles of universalism as opposed to particularism.

I had talked in class about universalism (apply general rules to everyone) and particularism (decide based on the needs, desires, abilities, etc. of the actual people in some real situation).  My five-cent definition was this: With particularism, if the rules don’t fit the people, too bad for the rules.  With universalism, if the rules don’t fit the people, too bad for the people. 

One of the examples I used to illustrate the difference was shopping.  For most items, we prefer universalism – a fixed price.  Everyone pays the amount marked on the price tag. You have only two options: buy it or leave it.  In Mediterranean cultures, buyers and sellers are much more likely to haggle, arriving at a price based on the unique utility curves and bargaining skills of the buyer and seller.  This winds up with different people paying different prices for the same item.

The researchers asked American and Turkish students about a “hypothetical situation”:

You are a professional journalist who writes a restaurant review column for a major newspaper. A close friend of yours has invested all her savings in her new restaurant. You have dined there and think the restaurant is not much good. Does your friend have some right to expect you to “hedge” your review or does your friend have no right to expect this at all?

I assumed that the study would find Americans to be more universalistic.  But I was wrong, at least according to this study.

	Turkish	American	Total
Particularistic	8 (19%)	85 (65%)	93
Universalistic	34 (81%)	45 (35%)	79
Total	42	130	172

Four out of five Turkish students said they would write their review according to universalistic principles.  Two-thirds of the Americans said they’d give their friend a break even if that meant departing from the standards of restaurant reviewing.

I was surprised.  So was my Yasemin Besen-Cassino.  Not only is she Turkish (though very global cosmopolitan), but she sometimes teaches a section of our methods course.  She added, “I am not a fan of hypotheticals on surveys.”

And oh boy, is this hypothetical.

• IF you were a reviewer for a major paper and
• IF the restaurant were bad and
• IF the owner were your friend and
• IF she had invested all her money in the place
  what kind of review would you write?

The more hypothetical the situation, the more I question people’s ability to know what they would do.   “IF the election were held today, who would you vote for?” probably works.  The situation – voting – is a familiar one, and there’s not all that much difference between saying the name of a candidate to an interviewer and choosing that name on a ballot.   But how many of us have experience writing reviews of friends’ restaurants? 

Nearly all my students say that if they were in the Milgram experiment, they’d have no trouble telling the experimenter to take a hike.  And all those concealed-carrying NRA members are sure that when a mass murderer in a crowd started firing his AR-15, they would coolly identify the killer and bring him down.  But for novel and unusual situations, we’re not very good at predicting what we would do. 

When I present the Milgram set-up and ask, “What would you do?”  sometimes a student will say, “I don’t know.”  That’s the right answer.

Surveys — Questions and Answers

December 10, 2012
Posted by Jay Livingston

Neil Caren at Scatterplot  lifts up the rock that is the New Family Structure Study (NFSS) – the basis of Mark Regnerus’s controversial research on children of gay parents – and discovers some strange creatures wriggling about underneath: 

. . .   85 people reported living at least four months with their “mother’s girlfriend/partner.” However—and this is where it gets tricky—a different question (S8) asked, “Did you ever live with your mother while she was in a romantic relationship with another woman?” Eight people who reported in the calendar that they lived with their mother’s girlfriend answered no to this question.

So ten percent of the people who said they lived with the mother’s girlfriend also said on a different question that they did not live with the mother’s girlfriend.
                   
We all rely on surveys – pollsters, social scientists, market researchers, government agencies, businesses. We try to make our questions straightforward.  But the question we ask is not always the question people answer.  And people’s answers – about what they think and what they did – are influenced by external factors we might not have considered.  Especially if the survey is a one-off (unlike the GSS and other surveys with frequently asked questions),  we have to be cautious about taking the results at face value.

(Previous posts on this problem are here and here.)

Prediction Methodology - Not Too Swift

November 6, 2012
Posted by Jay Livingston

One of the first things I try to get students to understand is the difference between systematic evidence and anecdotal and impressionistic evidence.  Or no evidence, which usually takes the form of “We don’t need studies to know that . . . .” or “Common sense tells us . . . .”

So in one corner we have Nate Silver (known in some circles as Nate the Great at Five Three Eight), systematically weighing the data from polls and other sources.  He sees Obama as the likely winner.

And then there’s Peggy Noonan at The Wall Street Journal.

Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us?

In front of her eyes is victory for Romney.  Here are some more excerpts that show the evidence she uses as the basis for her prediction.

Among the wisest words spoken this cycle were by John Dickerson of CBS News and Slate, who said, in a conversation the night before the last presidential debate, that he thought maybe the American people were quietly cooking something up, something we don’t know about.

I think they are and I think it’s this: a Romney win.

There is no denying the Republicans have the passion now, the enthusiasm.
it feels like a lot of Republicans have gone from anti-Obama to pro-Romney.
And there is Obama, out there seeming tired and wan, showing up through sheer self discipline.

All the vibrations are right. A person who is helping him who is not a longtime Romneyite told me, yesterday: “I joined because I was anti Obama—I’m a patriot, I’ll join up But now I am pro-Romney.”

And there’s the thing about the yard signs. In Florida a few weeks ago I saw Romney signs, not Obama ones. From Ohio I hear the same. From tony Northwest Washington, D.C., I hear the same.

I imagine going to the World Series.  The guy at the hot dog stand says he thinks the Tigers are about to make a move.  I see Detroit players’ faces, full of passion and enthusiasm; the Giants look tired and wan. The Tigers are getting hits.  They even had a home run.  Their pitchers are tall and strong.  And then there’s the thing about caps – all those Detroit caps with the old English D.  I see them everywhere.

It all points to a big win by the Tigers. Clearly, the Giants are toast.

And then some nerd – “a man of very small stature, a thin and effeminate man with a soft-sounding voice, a poster child for the New Castrati”* – taps me on the arm and points to the scoreboard which posts the number of runs that each team has actually scored and the number of games that they have won.

Yes, Romney could win.  But remember Damon Runyon’s riff on Ecclesiastes: “The race is not always to the swift, nor the battle to the strong, but that's how the smart money bets.”

And they’re betting Obama.  At In-trade, a $100 bet would bring Ms. Noonan $300, and somewhat more if she bet in the UK.  My own hunch is that betting a bundle on Romney right now is not too swift.

UPDATE:  Another Republican speechwriter-turned-columnist, Michael Gerson, is yapping at Nate Silver.  John Sides at The Monkey Cage offers an excellent critique of Gerson and a defense of data- based social science.  (It is kind of depressing – Gerson and Noonan and the rest are intelligent people, and yet they force you to defend the radical idea of using systematic evidence.  But then again, their party is the standard-bearer for people who think that global warming is a myth and that earth is only 7000 years old.)

------------------------
 * Yes, this is what someone at the right-wing examiner.com actually wrote about Nate Silver.  I am not making this up.

Surveys and Sequence

July 8, 2012
Posted by Jay Livingston

Push polls are an extreme example of the problems inherent in surveys, even surveys that are not apparently tendentious.  You ask a seemingly straightforward questions, but respondents may not be answering the question you think you asked.  That’s why I tend to distrust one-shot surveys with questions that have never been used before.  (Earlier posts on this are here and here).

Good surveys also vary the sequence of questions since Question #1 may set the framework a person then uses to think about Question #2. 

“Yes, Prime Minister” offers a useful example – exaggerated, but useful in the research methods course nevertheless.

HT: Keith Humphrey

When the Going Gets Tough – Lipstick and Evolution

June 28, 2012
Posted by Jay Livingston

L’Oreal did not lose sales during the current recession.  And psychologist Sarah Hill says that this increase is part of a more general trend – “the lipstick effect.”  In a recession, women cut back on other stuff, but not cosmetics.

What makes L’Oreal worth it, even when times get tough, according to Hill, is evolutionary psychology.  (Hill’s new JPSP article  is here. She also has a shorter, more general write-up at Scientific American.)  It’s all about “reproductive strategy” – how to get your genes strewn about as much as possible. 

Human ancestors regularly went through cycles of abundance and famine, each of which favors different reproductive strategies. While periods of abundance favor strategies associated with postponing reproduction in favor of one’s own development (e.g., by pursuing an education), periods of scarcity favor more immediate reproduction. The latter strategy is more successful during times of resource scarcity because it decreases the likelihood that one will perish before having the chance to reproduce.

Got it? In good times, our human ancestors would try to get an education.  In hard times, they would try to get laid. 

Hill elaborates on the special problems for women.

For women, periods of scarcity also decrease the availability of quality mates, as women’s mate preferences reliably prioritize resource access.

“Reliably prioritize resource access” is from the SciAm blogpost, presumably the venue that’s reader-friendly for the general public.  What the sentence means, I think, is this:  A recession reduces the number of guys with enough money to take care of a family.

Those well-off guys, I mean males, thanks to evolution, are “men who seek in mates qualities related to fertility, such as youth and physical attractiveness.”  So a girl has to go even further in dolling herself up in order to snag one of them. 

It all makes sense, but it ignores one important factor – the economic  inequality between men and women.  The evol-psych explanation takes as a given that women must rely on men for “resource access” (which I think is roughly what you and I call “money.”)  What if women knew that their chances of getting a decent job were as good as a man’s, or better?  Would hard times still send them to the cosmetics counter?

Hill did include a measure of resource access, and found that it was not significantly related to the lipstick-effect, at least not in the lab experiments.  Here was the set-up: Subjects read an article that was either about the recession (“Worst Economic Crisis Since ’30s With No End in Sight”) or about “current architecture.” Then they were asked which products they preferred.  Women who read about the recession were more likely to go for (in the words of evolutionary psychologist C. Berry) “tight dresses and lipstick.”*  The “resource access” measure did not significantly alter that effect.  Rich girls and poor girls alike switched their preference to L'Oreal.

As for the guys, reading about the recession did not affect them in this way.  Their desire for “attractiveness products” was unchanged.

I never know what to make of psychology experiments.  Their elaborate contrivance gives them enviable control over the variables, but it also raises questions about their link to the real world.  In Hill’s experiments, as is typical, the subjects were “unmarried female university students” – what we used to call “college girls” (plus, in one of the experiments, college boys).  It would be interesting to see if actual recessions lead to lipstick-buying across the socio-economic landscape.  Evol-psych would predict that the effect should be most visible in places where the recession hits hardest.

It’s also worth noting that L’Oreal might have been the exception this time around.  Sales in the industry as a whole suffered in the recession and did not reach pre-recession levels till 2010, and much of the increase came from bargain hunters.  (An industry report is here.) That contradicts Hill’s lab experiment results showing that “the lipstick effect applies specifically to products that enhance beauty, even when those products are more expensive.”   The larger increase in cosmetics sales came in 2011, especially for nail products (up 59%, go figure).

The experiment’s “priming” with newspaper stories is also a problem.  I’m puzzled about the use of that “current architecture” article as a control.  Why not an article that was upsetting but had nothing to do with economics – something like “How Hackers Easily Get Your Phone Messages”?  Maybe any disturbing article would have the same lipstick effect, even though cell phone privacy has nothing to do with a woman’s ability to pass along her genes.  As the t-shirt says, “When the going gets tough, the tough go shopping.” Maybe it doesn’t matter whether the tough-going is economic or something else.

Finally, I wonder about those guys.  If recessions make women but not men worry about their genes, asking college guys about face cream and tight polo shirts might not be the best way to operationalize the variable.  Why not ask about things that most guys think make them more attractive to women – probably consumer goods that signal cultural and economic capital?  Maybe college boys who read the recession article would shift their preference from video games to dress shirts and ties; or maybe the change would go the other way.  Whatever the outcome, I'm sure evol-psych would have an explanation. 

-----------------------
* I am not making this up: “The three attractiveness-enhancing products were (a) form-fitting jeans, (b) form-fitting black dress (women) / form-fitting polo shirt (men), and (c) lipstick (women) / men’s facial cream (men).”  And, as noted above, these were college girls, not all that much older than sweet little sixteen.

Free Samples

June 23, 2012
Posted by Jay Livingston

Google has nGrams for quick content analysis of words and phrases in “lots of books.”  Google also has Correlate which allows you to trace search strings across time and place and to discover correlations between search strings. 

Facebook too makes information on their users available, though their motive is not so selfless as Google’s.  The do it so that advertisers can narrow their target.  Planet Money had a story recently about a pizza joint in New Orleans that used FB’s data to select the target audience for its ads.

Their first idea was to target the friends of people who already liked Pizza Delicious on Facebook. But that wound up targeting 74 percent of people in New Orleans on Facebook — 224,000 people. They needed something narrower.

The Pizza Delicious guys really wanted to find people jonesing for real New York pizza. So they tried to target people who had other New York likes — the Jets, the Knicks, Notorious B.I.G. Making the New York connection cut the reach of the ad down to 15,000.

Seemed perfect. But 12 hours later, Michael called us. “It was all zeroes across the board,”  he said. Facebook doesn't make money till people click on the ad. If nobody clicks, Facebook turns the ad off. They'd struck out.

So they changed the target to New Orleans fans of Italian food: mozzarella, gnocchi, espresso. This time they were targeting 30,000 people.

Those ads went viral. They got twice the usual number of click-throughs, on average. The ad showed up more than 700,000 times. Basically, everyone in New Orleans on Facebook saw it. Twice.

To get the access to the data, you don’t really have to be an advertiser; you just have to play one on Facebook.  Neal Caren at UNC tells you how.  He used Facebook to compare rates of same-sex and hetero preferences across age groups and states.  His instructional post is here.

(HT: Philip Cohen)

Whose Kids Are All Right?

June 22, 2012
Posted by Jay Livingston

Miscellaneous thoughts on the Regnerus study.

1.    Oranges and apples.  This study is not about the effects of gay marriage.  Opponents of gay marriage trying to cram it into that cubbyhole apparently have not read the title: “How different are the adult children of parents who have same-sex relationships? Findings from the New Family Structures Study.” [emphasis added]

Who are these “parents who have same-sex relationships”?  They are not gay couples (there were only two of those in the sample, both female).  The image I get is the closeted homosexual trying to do the right thing, maybe even “cure” himself, by getting married.  The cure doesn’t work and he is now in an unhappy, unfulfilling marriage, but he stays because of the kids.  Eventually, he gives in to his desires, has a “same-sex relationship,” and maybe leaves his family.  

Is this scenario common in Regnerus’s sample?  I don’t know.  But to make gay parent vs. straight parent comparisons on the basis of the sample with only two gay couples is to compare these unhappily married oranges with Ozzie-and-Harriet apples.  As Regnerus’s defenders delicately put it, “This is not an ideal comparison.”

2.    Secondary deviance.  Edwin Lemert coined this term to refer to deviance that arises as a reaction to the social or legal stigma that comes with the primary deviance.   The crime is primary, the coverup is secondary.  The coverup occurs only because the original act is criminal.  The same applies to non-criminal forms of deviance and to social sanctions rather than legal ones.

Again, the Regnerus defense team: “This instability may well be an artifact of the social stigma and marginalization that often faced gay and lesbian couples during the time (extending back to the 1970s, in some cases) that many of these young adults came of age.” 

3.    Rights and Research.  As Ilana Yurkiewicz at Scientific American says, even if good, relevant research on the topic of gay marriage (which the Regnerus study is not) showed that kids from gay marriages do worse than kids from straight marriages, that’s no reason to deny people the right to marry.

Research has already found such differences between other categories of people – poor vs rich, for example.  Should we deny poor people the right to marry because their kids are less likely to do well in school or more likely to have run-ins with the law?  I would not be surprised if back in the mid-20th century, research would have shown (or perhaps did show) that the children of interracial marriages did not do as well on several variables as did Ozzie-and-Harriet or Cosby-show offspring.  Would that have been a valid reason to uphold laws banning interracial marriage?

4.    Etc.  Philip Cohen is much more qualified than I am to offer criticisms and comments on the study.  You should read his as yet unpublished op-ed.

They Work Hard for a Ton of Money

May 21, 2012
Posted by Jay Livingston

I’m not very good at looking at a scatterplot and estimating the correlation. 

This morning’s Wall Street Journal had a front-page story  about CEO pay.  Here’s the lede:

Chief executives increasingly are being paid based on their companies' financial results and share prices, according to a Wall Street Journal analysis.

The WSJ even had an outside source check their calculations and conclusions.

Pay was “highly correlated with performance,” says Steven Kaplan, a professor of finance at the University of Chicago's Booth School of Business who reviewed the Journal calculations.

Here’s the scatterplot showing the 300 largest companies:

(Click on the chart for a larger view.  Those wedge-shaped lines point to
very large photographs of individual CEOs, which I cropped out.)

I guess “highly correlated” is a term of art.  Unfortunately, the WSJ does not provide a regression line or correlation coefficient, but apparently the slope is +0.6.

On average, for every additional 1% a company returned to shareholders between 2009 and 2011, the CEO was paid 0.6% more last year, the analysis found. For every 1% decline in shareholder return, the CEO was paid 0.6% less.

I like that idea of considering the profitable CEOs separately from the CEOs whose firms lost money.  Here is the same scatterplot split down the middle. 

If you divide the Pay axis at $20 million, the relation becomes clear.  For every $20M+ CEO in a losing company, there are three in profitable companies. 

But here’s where my inability to look at the dots and estimate correlations messes me up.  To me, it looks as though among the losing firms, there’s no relation between CEO pay and how well the company did (i.e., how small its losses).  Same thing on the profit side, especially if you ignore the three $60M+ outliers.  (Timothy Cook of Apple, at $378M, lies out so far he’s not even on the chart.)  

I’m not sure to who to believe – the Wall Street Journal or my lyin’ eyes. 
The WSJ site has a chart listing the compensation of all 300 – from Apple down to Whole Foods, whose CEO didn’t even snag $1 million.

The story also heralds 2011 as showing huge improvement over the previous year in rationality, or at least the proportionality of pay to profits,

In 2010, there was no correlation; for every 1% decrease in shareholder return, the average CEO was paid 0.02% more.

Yes, you read that correctly.  The correlation was negative  – the smaller the profit (or larger the loss), the higher the CEO pay. 

Methodology in the News

April 20, 2012
Posted by Jay Livingston

1. “Survey Research Can Save Your Life,” says Joshua Tucker at the Monkey Cage. He links to this NBC news story about a woman who went into diabetic shock while on the phone with a student pollster working for Marist.  He sensed something was wrong and told his supervisor.  She spoke to the woman and then called 911.  (The news story does not identify the student working the phone survey, only the supervisor.  Nor does it say whether the woman approved or disapproved of Mayor Bloomberg.)

2.  The New York Times this week reported on a RAND study that found no relation between obesity and “food deserts.”  The study used a large national sample; it’s undoubtedly comprehensive.  The problem is that if you are using a national sample of schools or supermarkets or stores or whatever,  two units that fall into the same category on your coding sheet might look vastly different if you went there and looked at them from close range. 

Peter Moskos at Cop in the Hood took a closer look at the RAND study, reported in the Times, RAND relied on a pre-existing classification of businesses. The prefix code 445 indicates  a grocery store. Peter, an ethnographer at heart, has his doubts:

New York is filled with bodega “grocery stores” (probably coded 445120) that don't sell groceries. You think this matters? It does. And the study even acknowledges as much, before simply plowing on like it doesn't. A cigarette and lottery seller behind bullet-proof glass is not a purveyor of fine foodstuffs, and if your data doesn't make that distinction, you need to do more than list it as a “limitation.” You need to stop and start over.

3.  NPR’s “Morning Edition” had a story (here) on death penalty research, specifically on the question of deterrence.  A National Research Council panel headed by Daniel Nagin of Carnegie Mellon University reviewed all the studies and concluded that they were inconclusive, mostly for methodological reasons.  For example, most deterrence studies looked at the death penalty in isolation rather than comparing it with other specified punishments. 

Another methodological problem not mentioned in the brief NPR story is that the number of executions may be too small to provide meaningful findings.  For that we’d need a much larger number of cases.  So this is one time when, at least if you are pro-life, an inadequate sample size isn’t all bad.

The Wall Street Journal Or Your Lying Eyes

March 13, 2012
Posted by Jay Livingston

This graph tracks the share of income going to the top 1% in seven countries.  It’s from a paper by two Swedish economists, Jesper Roine and Daniel Waldenström (pdf here).

(Click on the graph for a larger view.)

The trend was towards greater equality up to 1980 – the share of the 1% was shrinking.    Since then, the 1% have increased their share of the income pie in all seven countries.  But the graph seems to show important differences, especially in recent decades.  Here is a  cropped version of the graph showing the 1980-2004 years.  I have added straight lines connecting those two points for Sweden and for the US.

Both changes are increases, but are they the same or are they different?  The answer is crucial.  The US and Sweden have different economic policies.  If the changes are no different between countries, then inequality is just one of those inevitable things that’s happening no matter what governments do.  But if the growth of inequality in the US is much greater than in Sweden, maybe government policy can in fact mitigate the trend towards inequality.

The Swedish 1% share went from a little under 5% to about 7.5%.  In the US, the 1% share increased from about 7% to 16%.* You might see those increases as very similar.

In fact, Allan Meltzer in the Wall Street Journal takes precisely that view.  He stretches out the graph to de-emphasize the vertical differences, and adds a title implying that all countries are “together” in this shift of income to the top 1%.

He adds this explanation:

As the . . . chart . . . shows, the share of income for the top 1% in these seven countries generally follows the same trend line. That means domestic policy can’t be the principal reason for the current spread between high earners and others. Since the 1980s, that spread has increased in nearly all seven countries. The U.S. and Sweden, countries with very different systems of redistribution, along with the U.K. and Canada show the largest increase in the share of income for the top 1%. [emphasis added]

If your pay went from $5 an hour to $7.50 an hour while your co-worker’s went from $7 to $16, you might think that your co-worker had gotten a substantially heftier raise.  But if so, that’s because you’re not the Wall Street Journal.  

Meltzer’s main point in the article is that we should not raise taxes on the very wealthy.  However, as Bruce Barlett points out (here), if the rich are getting just as rich in high-tax countries like Sweden and the Netherlands as they are in low-tax countries like the US, we may as well raise taxes on them. They’ll be doing just as well, like their Swedish and Dutch counterparts, and the nation will have more revenue to put towards Medicare, education, deficit-reduction, etc. 

But Meltzer is wrong.  Sweden and the Netherlands are very different from the US.  As the graph shows, the income share of the 1% in the US is twice that of the 1% in Sweden and 3 times that of the 1% in the Netherlands.  And it has risen more rapidly.  Yet Meltzer claims that inequality trends are similar everywhere. 

So who are you going to believe - the Wall Street Journal or your lying eyes?

 -------------- 
* Big hat tip to Andrew Perrin at Scatterplot.  Several economics blogs have also looked at the Meltzer article. 

UPDATE March 16: Gwen Sharp at Sociological Images posted this link to a database of income data from various countries.  You can to create your own graphs of income shares.

Deep Change in the Deep South?

March 12, 2012
Posted by Jay Livingston

The polling news today is that very few Republicans in Alabama and Mississippi (14% and 12%, respectively) think that President Obama is a Christian.  Three times as many think he’s a Muslim. (A pdf of the entire survey is here.)

The poll also finds that only about one in four Republicans in those states believe in evolution.  Five times that many flatly reject evolution, with about 10% “not sure.” 

The results I found most curious were the opinions on interracial marriage.  Alabama 21% thought it should be illegal, 67% thought it should be legal; in Mississippi,  29% illegal, 54% legal.  None of the news stories I looked on this noted that when the same pollsters (Public Policy Polling) asked the same question of Mississippi Republicans less than a year ago, the results were very different.  A plurality thought it should be illegal.  (My post on that poll is here.)

The margin of error is 4% (N = 600), so the 15-point swing supposedly reflects a real change.  But I’m skeptical.  What could account for such a large change if not sampling variation?  Did the GOP organize mass screenings of “The Help” and shame some of their number into allowing that maybe Loving v. Virginia wasn’t a mistake after all? Did the Heidi Klum - Seal breakup make it OK?   I can’t come up with even a dubiously speculative explanation.

Psychology (!!!) or Sociology (zzz)

February 8, 2012
Posted by Jay Livingston

News media have to come up with provocative headlines and ledes, even when they’re reporting on academic papers.  And even when the reasonable reaction would be “Well, duh,” rather than a gasp in 72-point caps.  But if that’s the route you want to go, it usually helps to think psychologically rather than sociologically.

Here’s a headline from Forbes

Facebook More Addictive Than Cigarettes, 

Study Says

And the Annenberg School Website started their story with this.

Cigarettes and alcohol may not be the most addicting drugs on the market, according to a recent study.
A team from the University of Chicago's business school has suggested everyone's suspicion: social networking is addictive. So addictive that constantly using sites like Facebook and Twitter may be a harder vice to kick than smoking and drinking.  [emphasis added]

The study in question is “Getting Beeped With the Hand In The Cookie Jar: Sampling Desire, Conflict, and Self-Control in Everyday Life” by Wilhelm Hofmann, Kathleen D. Vohs, and Roy F. Baumeister, presented at a recent Society for Personality and Social Psychology conference.  They had subjects (N=205) wear beepers and report on their desires. 
I found out about it in a Society Pages research round-up (here).

A study of 205 adults found that their desires for sleep and sex were the strongest, but the desire for media and work were the hardest to resist. Surprisingly, participants expressed relatively weak levels of desire for tobacco and alcohol. This implies that it is more difficult to resist checking Facebook or e-mail than smoking a cigarette, taking a nap, or satiating sexual desires.

Of course it’s more difficult.   But the difficulty has almost nothing to do with the power of the internal desire and everything to do with the external situation, as The Society Pages (a sociology front organization) should well know.  In a classroom, a restaurant, a church, on the street, in an elevator – just about anywhere – you can quietly glance down at your smartphone and check your e-mail or Facebook page.  But to indulge in smoking, sleeping, and “satiating sexual desires,” you have to be willing to violate some serious norms and even laws.

It’s not about which desires are difficult to resist.  It’s about which desires are easy to indulge.  The study tells us not about the strength of psychological desires but the strength of social norms.  You can whip out your Blackberry, and nobody blinks.  But people might react more strongly if you whipped out, you know, your Marlboros. 

The more accurate headline might be

Checking Twitter at Starbucks OK, Having Sex There, Not So Much, Study Finds

But that headline is not going to get nearly as much attention.

Doing the Math

February 7, 2012
Posted by Jay Livingston

My students sometimes have trouble with math, even what I think is simple math.  Percentage differences, for example.  I blame it on the local schools.  Once I explain it, I think most of them catch on. 

Stephen Moore is not from New Jersey.  His high school diploma is from the highly regarded New Trier, he has an economics masters degree from George Mason, and he writes frequently about economics.  A couple of days ago he wrote in the Wall Street Journal (here) about how much better it was to work for the government than for private employers.* 

Federal workers on balance still receive much better benefits and pay packages than comparable private sector workers, the Congressional Budget Office reports. The report says that on average the compensation paid to federal workers is nearly 50% higher than in the private sector, though even that figure understates the premium paid to federal bureaucrats.

CBO found that federal salaries were slightly higher (2%) on average, while benefits -- including health insurance, retirement and paid vacation -- are much more generous (48% higher) than what same-skilled private sector workers get.

It’s not clear how Moore arrived at that 50% number.  Maybe he added the 2% and the 48%. 

Let’s assume that the ratio of salary to benefits is 3 - 1.  A worker in the private sector who makes $100,000 in salary would get $33,000 worth of benefits. The government worker would get 2% more in salary and 48% more in benefits. 

	Private	Gov't.
Salary	100,000	102,000
Benefits	33,000	49,500
Total	133,000	151,500

If total compensation for private-sector workers is $133,000, and if government workers were getting 50% more than that, their total compensation would be $200,000. But the percentage difference between the $150K and the $133K is nowhere near 50%.  The government worker pay package is 14% higher. 

I think I could explain this so my students would understand it.  But then again, they don’t write columns for the Wall Street Journal.

----------------------------
* The WSJ gives the article the title “Still Club Fed.” The more accurate title would be “Government Jobs Are Good Jobs.” Of course, the latter takes the perspective of people looking for work, a viewpoint that doesn’t get much consideration at the WSJ.

Applied Probability

 February 6, 2012
Posted by Jay Livingston

Long-odds prop bets are sucker bets.  The odds that bookmakers offer are nowhere near the true probability.  But expected values matter only if you’re playing a large number of times, which is what the house is doing.  The bettor is betting just once, and 50-to-one odds sounds like a lot.

Take yesterday’s game. The odds that the first points of the game would be the Giants scoring a safety were 50-1.  That’s what the bookies offered.

But what is the true probability?  In the previous NFL season, there were 2077 scores, not counting point-after-touchdown.  Here is the breakdown (I found the data here).

• Touchdowns      1270
• Field Goals           794
• Safeties                    13

The probability of the first score being a safety by either team is 2064 to 13 or about 160 to 1.  The probability of the first score being a safety by a specified side is double that.  Even if that specified side is the Giants and their defense is twice as good as the Patriots defense, that still makes the probability at least 200 to 1.  The Las Vegas books were offering only 50 - 1, one-fourth of the correct odds.  So the expected return on a $1000 bet is $250 – a $750 loss.   What a ripoff.

Of course, not everyone feels duped.

Somewhere, someone is walking around with an I ♥ Brady t-shirt. 

HT: My colleague Faye Glass, though she tells me this picture is all over the Internet.

Do You Hear What I Hear? Maybe Not.

December 18, 2011
Posted by Jay Livingston

As I’ve said before (here), the question the researcher asks is not always the question people hear.   That’s especially true when the question is about probabilities.

Here, for example, is the ending of a fictional vignette from a recent study in the Journal of Personality and Social Psychology.

Richard found a wallet on the sidewalk. Nobody was looking, so he took all of the money out of the wallet. He then threw the wallet in a trash can.

Is it more probable that Richard is
    a. a teacher
    b. a teacher and a rapist

Since the category “a teacher” necessarily includes teacher/rapists as well, the correct answer is “a.” But many people choose “b.”  The study used this “conjunction fallacy”* to probe for prejudices by switching out the rapist for various other categories.  Some subjects were asked about atheist/teachers, others about Muslim/teachers, and so on.  The finding:

A description of a criminally untrustworthy individual was seen as comparably representative of atheists and rapists but not representative of Christians, Muslims, Jewish people, feminists, or homosexuals.

Andrew Gelman, a usually mild-mannered reporter on things methodological, had a post on this with the subject line, “This one is so dumb it makes me want to barf.”

What’s really disturbing about the study is that many people thought it was “more probable” that the dude is a rapist than that he is a Christian! Talk about the base-rate fallacy.

Maybe it would settle Andrew’s stomach to remember that the question the researchers asked was almost certainly not the question people heard.   What the researchers pretend to be asking is this:

Of all thieves, which are there more of – teachers or rapist/teachers? 

After all, that is indeed the literal meaning.  But it’s pretty obvious that the question people are answering is something different:

Which group has a higher proportion of thieves among them – all teachers or the subset rapist/teachers?

The researchers say they weren’t at all interested in demonstrating the conjunction fallacy.  They were just using it to uncover the distrust people feel towards atheists.  What they found was that when it comes to dishonesty, people (specifically, 75 female and 30 male undergrads at the University of British Columbia) rank atheists at about the same level as rapists.

But why resort to such roundabout tricks?  Why not ask the question directly?**

Who is more likely to steal a wallet when nobody is looking?
    a.  an atheist
    b. a rapist
    c.  neither; they are equally larcenous

Or:

On a seven-point scale, rank each of the following on how likely they would be to steal a wallet when nobody is looking:

•     an atheist: 1   2   3   4   5   6   7
•     a Christian: 1   2   3   4   5   6   7
•     a rapist: 1   2   3   4   5   6   7
•     etc. 

Instead, they asked questions that they knew would confuse nearly anyone not fluent in the language of statistics and probability.  I wonder what would happen if in their “who do you distrust” study they had included a category for experimental social psychologists.***

---------------
*  Daniel Kahneman and Amos Tversky pretty much invented the conjunction fallacy thirty years ago with their “Linda problem,” and Kahneman discusses it in his recent book Thinking Fast and Slow.  To get the right answer, you have to ignore intuition and make your thinking very, very slow.  Even then, people with no background in statistics and logic may still get it wrong.

** The authors’ presentation of their results is also designed to frustrate the ordinary reader. Each condition (rapist/teacher, atheist/teacher, homosexual/teacher, etc.) had 26 (or in one case 27) subjects.  The payoff was the number of errors in each group.  But the authors don’t say what that number was.  They give the chi-square, the odds ratios, the p’s and the b’s.  But they don’t tell us how many of the 26 subjects thought that the wallet snatcher was more likely to be an atheist/teacher or a Christian/teacher than to be merely a teacher.

*** The JPSP is one of the most respected journals in the field, maybe the most respected, influential, and frequently cited, as I pointed out here.

Surveys and Confirmation Bias

November 10, 2011
Posted by Jay Livingston

When he taught research methods as a grad student, Michael Schwartz gave his students this assignment: “Create a survey to show . . .” and he would tell them the conclusion he wanted the survey to support.  The next week, he’d give them the same assignment but with the desired conclusion the opposite of the first one.

A year and a half ago, I criticized (here) a much publicized study by Dan Klein and Zeljka Buturovic:  “This survey, I said, “wasn’t designed to discover what people think. It was designed to prove a political point,” and that point was that liberal ideology blinds people to economic facts. 

I was reminded of Mike’s assignment when I read Klein’s recent article at The Atlantic.  In a bit of academic fairness that’s probably all too rare, Klein went on to create a survey designed to see if conservative ideology has a similar effect.

Klein hoped that his conservative and libertarian allies would not so readily agree with politically friendly economic ideas that were nevertheless unsound. But conservatives in the new survey were “equally stupid” as the liberals in the earlier survey.

Klein also expected some nasty nyah-nyahing from his liberal critics.  But no, “The reaction to the new paper was quieter than I expected.”   In fact, one of those liberal critics, Matt Yglesias, offered an observation that Klein used as his takeaway from the two surveys: “there’s a lot of confirmation bias out there.” 

Yes, but confirmation bias is not just something that affects people who respond to surveys.  As Mike’s assignment makes clear, we also need to be wary of confirmation bias on the part of those who create the surveys. There is the further problem I mentioned in my earlier post:  a one-shot survey is inherently ambiguous. We can’t be sure just what the respondents really hear when they are asked the question. 

My own takeaway, besides admiration for Klein’s honesty, is that when you design your research as a statement (proving some point), you don’t learn nearly as much as when you design it as a genuine question.

Lying With Statistics, and Really Lying With Statistics

November 4, 2011

Posted by Jay Livingston

“The #1 way to lie with statistics is . . . to just lie!” says Andrew Gelman, who a) knows much about statistics and b) is very good at spotting statistical dishonesty.

But maybe there’s a difference between lying with statistics and just plain making stuff up.

I’ve commented before about social psychologists’ affinity for Candid-Camera deception, but this Dutch practitioner goes way beyond that.  [The Telegraph has the story .] 

The committee set up to investigate Prof Stapel said after its preliminary investigation it had found "several dozen publications in which use was made of fictitious data" . . .

[Stapel’s] paper that linked thoughts of eating meat eating with anti-social behaviour was met with scorn and disbelief when it was publicised in August, it took several doctoral candidates Stapel was mentoring to unmask him. . . .

the three graduate students grew suspicious of the data Prof Stapel had supplied them without allowing them to participate in the actual research. When they ran statistical tests on it themselves they found it too perfect to be true and went to the university's dean with their suspicions.

What’s truly unsettling is to think that maybe he’s not the only one.