A Cautionary Tale

July 30, 2013
Posted by Jay Livingston

Andrew Gelman (here) summarizes and links to a “killer story” that “ is so great that all quantitative political scientists (and sociologists, and economists, and public health researchers)” should take notice.  He’s right.

First, there’s the clever finding that political moderates literally see more shades of gray than do people whose views are more extreme, right or left.  Literally.  It’s a test of color perception.*

But the more important part of the story is Part II.  The authors (Brian Nosek, Jeffrey Spies, and Matt Motyl) could have gotten the study published, but they decided to do a replication first just to put the clincher on their findings.  The result: a p = .01 effect completely disappeared: p = .59.

I’ve commented before (here) on difficulties with replication and the more general problem of diminishing effects.  (See also Jonah Lehrer’s New Yorker article  “The Truth Wears Off.”) But this is as dramatic a turnaround as I know of.

In a comment on Andrew’s blogpost, Ashok Rao suggests that authors post the odds they would give on replication.  Making the authors bet on their results “seems like a pretty good way to discern papers where authors believe what they publish from, well, that where the ‘ample incentives’ dominate.”  (Rao also links to his own paper where he quotes Alex Tabarrok: “bets are a tax on bullshit.”)

------------------------------
* If you’re thinking “50 Shades of Gray,” Andrew already beat you to it.  That, in part, is the title of his blogpost.