Gee Whiz

November 28, 2007
Posted by Jay Livingston

Some time ago, the comments on a post here brought up the topic of the “gee whiz graph.” Recently, thanks to a lead from Andrew Gelman , I’ve found another good example in a recent paper.

The authors, Leif Nelson and Joseph Simmons, have been looking at the influence of initials. Their ideas seem silly at first glance (batters whose names begin with K are more likely to strike out), like those other name studies that claim people named Dennis are more likely to become dentists while those named Lawrence or Laura are more likely to become lawyers

But Nelson and Simmons have the data. Here’s their graph showing that students whose last names begin with C and D get lower grades than do students whose names begin with A and B.

The graph shows an impressive difference, certainly one that warrants Nelson and Simmon’s explanation:

Despite the pervasive desire to achieve high grades, students with the initial C or D, presumably because of a fondness for these letters, were slightly less successful at achieving their conscious academic goals than were students with other initials.

Notice that “slightly.” To find out how slight, you have to take a second look at the numbers on the axis of that gee-whiz graph. The Nelson-Simmons paper doesn’t give the actual means, but from the graph it looks as though the A students’ mean is not quite 3.37. The D students average between 3.34 and 3.35, closer to the latter. But even if the means were, respectively, 3.37 and 3.34, that’s a difference of a whopping 0.03 GPA points.

When you put the numbers on a GPA axis that goes from 0 to 4.0, the differences look like this.
According to Nelson and Simmons, the AB / CD difference was significant (F = 4.55, p < .001). But as I remind students, in the language of statistics, a significant difference is not the same as a meaningful difference.