These Truths, These Untruths, These Really Big Numbers

July 21, 2020
Posted by Jay Livingston

If a number just doesn’t sound right — it’s way too big or way too small — you’d better double-check. That’s the warning sociologist Joel Best has been giving us since 1990, if not before, when he looked at claims that the number of children abducted by strangers annually was 50,000. Way too many.

And now we have distinguished historian Jill Lepore, author of the recent best-selling history of the US These Truths. In a recent issue of The New Yorker, in her article “The Invention of the Police” (here) she says this:

I’m using a screenshot of The New Yorker’s online version only to show that as of this writing, more than a week after its initial publication, this passage remains unchanged. The text reads:

One study suggests that two-thirds of Americans between the ages of fifteen and thirty-four who were treated in emergency rooms suffered from injuries inflicted by police and security guards, about as many people as the number of pedestrians injured by motor vehicles.

This number, two-thirds, does not sound right. I have been in NYC emergency rooms — in the city’s high-crime years and in its low-crime years. Never did it look like two-thirds of the people there had been roughed up by the cops. The study Lepore cites (by Justin Feldman, here ) used data from 2001-2014. In that 14-year period,  683,000 young people (15-34) turned up in emergency rooms after being hit by the police or security guards, roughly 50,000 a year. That’s the numerator.

The denominator is the total number of emergency room visits by this age group. My estimate is about 30 million a year.* Obviously, 50,000 out of 30,000,000 is not two-thirds. It’s less than two-tenths of a percent.

What happened? Isn’t The New Yorker the publication that made “fact check” part of our everyday language?

Louise Perry (here) has an explanation of how Lepore and the fact checkers at The New Yorker misread the numbers and prose in Feldman’s article.

But it’s not clear where Lepore got the ‘two-thirds’ figure from. Possibly she misunderstood a line from from the paper itself, which includes the finding that 61.1% of people injured by police fell into the 15-34 age bracket. Or from the Harvard press release, which reports that: Sixty-four percent of the estimated 683,033 injuries logged between 2001-2014 among persons age 15-34 resulted from an officer hitting a civilian.

That’s the “how.” Perry also thinks she knows the “why”: “political bias.” Liberal, anti-police bias. Of course, it’s impossible to get evidence for this idea. The magazine and the writer have politically liberal views. Maybe the fact-checkers do as well. But it’s impossible to get evidence that their politics caused their misreading of the data.

So as long as we’re speculating without evidence, here’s my explanation (I’m not necessarily rejecting the politics explanation, just maybe adding to it): writing and reading about big numbers. It’s not easy to write up statistics in a way that is unmistakably clear. A reader not familiar with the territory can easily take a wrong turn, especially when that territory takes the shape of large numbers. If I told you that before the current pandemic, emergency rooms in the US saw about 3 million people a year, that might sound reasonable. I mean, three million seems like a really big number. But it’s only one- tenth of the actual number.

I know very little about dinosaurs. If you told me they went extinct 10 million years ago, I would think, Yeah, that’s a long time ago; it sounds about right. If you told me that they went extinct 200 million years ago, I’d have the same reaction. Sounds plausible. But a paleontologist would wonder how I could be taken in by such obvious untruths.

Yes, it’s possible that The New Yorker’s fact-checkers were so eager to stick it to the police that they let an “obvious” mistake slip by.  But it’s also possible that they just didn’t know how to parse these claims about the data, and because some of the numbers were large and others (ED visits) still larger and unknown, they just seemed reasonable.


* This a rough and quick estimate. The average 15-34 population over those years was in the range of 30-35 million. This site  shows rates of ED use by age group. Unfortunately, the age groups are Under 18 and 18-44. So I estimated on the low side.


Jon said...

There's a significant difference between being dazzled by big numbers and being connected enough with the real world to be able to roughly estimate shares. 2/3 is a ridiculous share to attribute to police violence--it's like claiming 3/4 of all murders are committed by redheads. A reasonable person would be skeptical.

Jay Livingston said...

My guess is that she read the Harvard press release, which was not clearly written, and just didn't bother to question it since it fit with her beliefs. That can happen, as I more or less say in the first paragraph. As a journalist writing with limited time and maybe even a deadline, Lepore is not as careful about sources as she is when she's writing as a historian.

Over on the right, they're celebrating this as though it were a major event. The National Review headline says, "New Yorker writer falsely claims . . ." and a Breitbartish site calls it, "a lie so massive it beggars belief." It's not a lie; it's a mistake.

Andrew Gelman said...


Do you think another source of the problem was a Harvard effect? Maybe the fact checker didn't even think of checking because the author is a Harvard professor, and Harvard professors know everything.

Jay Livingston said...

I don't know much about how fact-checking goes at the New Yorker, though I did just now find this short piece about Daniel Radcliffe trying his hand at it (take your wizard to work day, I guess). It's also possible that Lepore supplied a link to the Harvard press release, and the way it's written can be read as confirming what she said.  So a seemingly legit reference plus her personal reputation plus Harvard plus unfamiliarity with how to calculate this fact and find the appropriate numbers plus a lot of facts to check — put them all together they spell error.

(BTW, I’ve mentioned my personal problem with the Harvard “brand” before in this blog (here) and my tendency when asked where I got my degree to say, “HarvardButDon’tBeImpressed.”)