Posted by Jay Livingston
Anecdotal evidence? Everyone knows to be suspicious, even journalists. A Lexis-Nexis search returns about 300 news articles just in this month where someone was careful to specify that claims were based on “anecdotal evidence” and not systematic research.
Everywhere else, the anecdotal-systematic scale of credibility is reversed. As Stalin said, “The death of a million Russian soldiers – that is a statistic. The death of one Russian soldier – that is a tragedy.” He didn’t bother to add the obvious corollary: a tragedy is far more compelling and persuasive than is a statistic.
Yet here is journalist Heather Havrilesky in the paper of record reviewing Presence, a new book by social scientist Amy Cuddy:
| This detailed rehashing of academic research . . . has the unintended effect of transforming her Ph.D. into something of a red flag. |
Yes, you read that correctly. Systematic research supporting an idea is a bright red warning sign.
Amy Cuddy, for those who are not among the millions who have seen her TED talk, is the social psychologist (Ph.D. Princeton) at the Harvard Business School who claims that standing in the Wonder Woman “power pose” for just two minutes a day will transform the self-doubting and timid into the confident, assertive, and powerful. Power posing even changes levels of hormones like cortisol and testosterone.
| While Cuddy’s research seems to back up her claims about the effects of power posing, even more convincing are the personal stories sent to the author by some of the 28 million people who have viewed her TED talk. Cuddy scatters their stories throughout the book. . . . |
Systematic research is OK for what it is, Havrilesky is saying, but the clincher is the anecdotal evidence. Either way, the results fall into the category of “Amazing But True.”
Havrilesky was unwittingly closer to the truth with that “seems” in the first clause. “Cuddy’s research seems to back up her claims . . . ” Perhaps, but research done by people other than Cuddy and her colleagues does not. As Andrew Gelman and Kaiser Fung detail in Slate, the power-pose studies have not had a Wonder Woman-like resilience in the lab. Other researchers trying to replicate Cuddy’s experiments could not get similarly positive results.
But outside the tiny world of replication studies, Cuddy’s findings have had a remarkable staying power considering how fragile* the power-pose effect was. The problem is not just that the Times reviewer takes anecdotal evidence as more valid. It’s that she is unaware that contradictory research was available. Nor is she unique in this ignorance. It pervades reporting even in serious places like the Times. “Gee whiz science,” as Gelman and Fung call it, has a seemingly irresistible attraction, much like anecdotal evidence. Journalists and the public want to believe it; scientists want to examine it further.
| Our point here is not to slam Cuddy and her collaborators. . . . And we are not really criticizing the New York Times or CBS News, either. . . . Rather, we want to highlight the yawning gap between the news media, science celebrities, and publicists on one side, and the general scientific community on the other. To one group, power posing is a scientifically established fact and an inspiring story to boot. To the other, it’s just one more amusing example of scientific overreach. |
I admire Gelman and Fung’s magnanimous view. But I do think that those in the popular press who report about science should do a little skeptical fact-checking when the results seem too good to be true, for too often these results are in fact too good to be true.
---------------------
* “Fragile” is the word used by Joe Simmons and Uri Simonsohn in their review and replication of Cuddy’s experiments (here).
No comments:
Post a Comment