How to Misread a Graph (It’s Not Easy, but The Heritage Foundation Finds a Way)

September 20, 2013
Posted by Jay Livingston

My post of a few days ago (here) showed The Heritage Foundation presenting a graph and deliberately drawing a conclusion that the graph clearly showed to be wrong.  Apparently, that’s something of a specialty at The Heritage Foundation.

Here’s their graphic purporting to show that preschool programs don’t work. (The original is here.)

The problem in the Oklahoma graph is the lag time between cause and effect.  For example, the baby boom began in 1947, but we would not look for its effects on healthcare and Social Security costs until much, much later.

Most people know this, but  Heritage seems to be lagging behind. “Fourth grade reading achievement scores in Oklahoma have actually declined.” True, they are lower now than in 1998, when universal preschool started. But is that the year should we use for a starting point for data on fourth grade reading scores?

Pre-school kids are three or four years old.  They don’t take the fourth-grade reading test until six or seven years later – in Oklahoma, that would be 2005 for the first cohort.  Amazingly (amazing to Heritage, I guess), that was the year reading scores began to increase, and despite a slight dip last year, they are still above that level.

As for the Georgia graph, anyone glancing at it (anyone except for people at The Heritage Foundation) would see this: reading scores in Georgia began increasing in 1995, two years after universal preschool began, and continued to rise when the first preschoolers reached fourth grade; scores have continued to rise faster than the national average.  Georgia was behind, now it’s ahead. Something good has been happening.

Heritage, however, manages not to see this and instead complains about how long it took Georgia to reach that point. (“Georgia’s program was in place for 13 years before scores caught up to the U.S. average.”)

A simple graph of scores is not really an adequate assessment of universal preschool. Those assessments, which include many other relevant variables,* have been done, and they generally conclude that the programs do work.  But that’s not the point.  The point is that Heritage is again misreading its own graph. So again I repeat, “Who you gonna believe, the Heritage Foundation or your lyin’ eyes?”

HT: Philip Cohen, who apparently thinks the Heritage deliberate obtuseness is so obvious as to be unworthy of elaboration.

-----------

* These include the usual demographics, especially to see if preschool effects are different for different groups. But there’s also the problem of post-preschool education. A state might have great preschools, but if it also has lousy primary schools, the benefits of preschool will be eroded away by the time the kids are in fourth grade.