Gee Whiz

November 28, 2007
Posted by Jay Livingston

Some time ago, the comments on a post here brought up the topic of the “gee whiz graph.” Recently, thanks to a lead from Andrew Gelman , I’ve found another good example in a recent paper.

The authors, Leif Nelson and Joseph Simmons, have been looking at the influence of initials. Their ideas seem silly at first glance (batters whose names begin with K are more likely to strike out), like those other name studies that claim people named Dennis are more likely to become dentists while those named Lawrence or Laura are more likely to become lawyers

But Nelson and Simmons have the data. Here’s their graph showing that students whose last names begin with C and D get lower grades than do students whose names begin with A and B.

The graph shows an impressive difference, certainly one that warrants Nelson and Simmon’s explanation:
Despite the pervasive desire to achieve high grades, students with the initial C or D, presumably because of a fondness for these letters, were slightly less successful at achieving their conscious academic goals than were students with other initials.

Notice that “slightly.” To find out how slight, you have to take a second look at the numbers on the axis of that gee-whiz graph. The Nelson-Simmons paper doesn’t give the actual means, but from the graph it looks as though the A students’ mean is not quite 3.37. The D students average between 3.34 and 3.35, closer to the latter. But even if the means were, respectively, 3.37 and 3.34, that’s a difference of a whopping 0.03 GPA points.

When you put the numbers on a GPA axis that goes from 0 to 4.0, the differences look like this.
According to Nelson and Simmons, the AB / CD difference was significant (F = 4.55, p < .001). But as I remind students, in the language of statistics, a significant difference is not the same as a meaningful difference.


Bad Runner said...

Jay, did they report the N on this? I'd like to use that for something.

maxliving said...

Perhaps to them, the 3 hundredths of a point were meaningful. Were they Stuy students?

Jay Livingston said...

Dan, they don't give the N. They do say that the sample consisted of 15 years of students graduating from an MBA program. (Sorry, Max, that lets out Stuy.)

One of the authors is at the Yale management school, so my guess is that the otherwise unnamed "large private American univeristy" begins with a Y.

If Yale graduates roughly 50 MBA per year (just a guess), multiply by 15, and that's an N of 750.

Phil said...

Now I know why I changed my initials from PC to PBC after I got married ...