SAT, GPA, and Bias

July 8, 2011
Posted by Jay Livingston

(Cross-posted at Sociological Images)

Is the SAT biased? If so, against who is it biased?

It has long been part of the leftist creed that the SAT and other standardized tests are biased against the culturally disadvantaged - racial minorities, the poor, et. al. Those kids may be just as academically capable as more privileged kids, but the tests don’t show it.

But maybe SATs are biased against privileged kids. That’s the implication in a blog post by Greg Mankiw. Mankiw is not a liberal. In the Bush-Cheney first term, he was the head of the Council of Economic Advisors. He is also a Harvard professor and the author of a best-selling economics text book. Back in May he had a blog post called “A Regression I’d Like to See.” If tests are biased in the way liberals say they are, says Mankiw, let’s regress GPA on SAT scores and family income. The correlation with family income should be negative.

a lower-income student should do better in college, holding reported SAT score constant, because he managed to get that SAT score without all those extra benefits.

In fact, the regression had been done, and Mankiw added this update:

Todd Stinebrickner, an economist at The University of Western Ontario, emails me this comment: “Regardless, within the income groups we examine, students from higher income backgrounds have significantly higher grades throughout college conditional on college entrance exam . . . scores.” [Mankiw added the boldface for emphasis.]

What this means is that if you are a college admissions officer trying to identify the students who will do best in college, as measured by grades, you would give positive rather than negative weight on family income.

Not to give positive weight to income, therefore, is bias against those with higher incomes.

To see what Mankiw means, look at some made-up data on two groups. To keep things civil, I’m just going to call them Group One and Group Two. (You might imagine them as White and Black, Richer and Poorer, or whatever your preferred categories of injustice are. I’m sticking with One and Two.) Following Mankiw, we regress GPA on SAT scores. That is, we use SAT scores as our predictor and we measure how well they predict students’ performance in college (their GPA).

(Click on the image for a larger, clearer view)

In both groups, the higher the SAT, the higher the GPA. As the regression line shows, the test is a good predictor of performance. But you can also see that the Group One students are higher on both. If we put the two groups together we get this.

Just as Mankiw says, if you’re a college admissions director and you want the students who do best, at any level of SAT score, you should give preference to Group One. For example, look at all the students who scored 500 on the SAT (i.e., holding SAT constant at 500). The Group One kids got better grades than did the Group Two kids. So just using the SATs, without taking the Group factor (e..g., income ) into account, biases things against Group One. The Group One students can complain: “the SAT underestimates our abilities, so the SAT is biased against us.”

Case closed? Not yet. I hesitate to go up against an academic superstar like Mankiw, and I don’t want to insult him (I’ll leave that to Paul Krugman). But there are two ways to regress the data. So there’s another regression, maybe one that Mankiw does not want to see.

What happens if we take the same data and regress SAT scores on GPA? Now GPA is our predictor variable. In effect, we’re using it as an indicator of how smart the student really is, the same way we used the SAT in the first graph.
Let’s hold GPA constant at 3.0. The Group One students at that GPA have, on average, higher SAT scores. So the Group Two students can legitimately say, “We’re just as smart as the Group One kids; we have the same GPA. But the SAT gives the impression that we’re less smart. So the SAT is biased against us.”

So where are we?

• The test makers say that it’s a good test - it predicts who will do well in college.
• The Group One students say the test is biased against them.
• The Group Two students say the test is biased against them.

And they all are right.


Huge hat tip to my brother, S.A. Livingston. He told me of this idea (it dates back to a paper from the1970s by Nancy Cole) and provided the made-up data to illustrate it. He also suggested these lines from Gilbert and Sullivan: And you'll allow, as I expect
That they are right to so object
And I am right, and you are right
And everything is quite correct.

Records and Performances

July 6, 2011
Posted by Jay Livingston

(A personal rant. If you are not interested in language or horse racing, read no further.)

One of these things is not like the others.

1. “He’s a guy with a track record that's beyond reproach.” (Nationals GM Mike Rizzo speaking about Davey Johnson as manager. USA Today, June 27)
2. The decision to publish was made easier by Ritter’s proven track record as a songwriter. (Steven King reviewing a book by Josh Ritter, NYT July 3)
3. Members of the House don't have a very good track record in primary campaigns. (Nate Silver’s FiveThirtyEight NYT blog on Michelle Bachman, June 23, a post called “Handicapping the Republican Field..”)
4. That’s what the OLC is good at, and what it has a long track record of doing on war powers. (Ruth Marcus on the Office of Legal Counsel, WaPo, June 22)
5. He walked a 20-kilometer race in Finland in 1:23:49.39 . . . just nine seconds off the U.S. men's 20K track record, which, as it happens, is held by his coach. (An article on racewalking, USA Today, June 24)
6. That No. 1 finish extends Pixar’s perfect box-office track record - all 12 of its feature films have opened in first place. (NYT article on “Cars 2,” June 27)
7. G.M.’s track record for making cars people want has not exactly been inspiring. (Joe Nocera, NYT, June 26)

One of these things uses “track record” in its original and logical sense. Can you guess which one is not like the others by the time I finish this post?

In the previous post about history, I said that horseplayers use the past performance charts to handicap the horses – i.e., to assign each horse a probability of winning – and decide which horse to bet on. “Past performances” are the details of each horse’s performance in previous races. It’s a lot of information.

Here’s an excerpt of the past performances for the horses in the first three post positions in this year’s Belmont Stakes. (A tutorial on how to read past performances is here.)

(Click on the image for a slightly larger view. Or go here and click on this same image for the full thing at readable size.)

After the race, horseplayers become historians. They go back to the same past performance charts to find the information that explain why the winner won (and why their horse finished out of the money).

The term “track record” also comes from horse racing. It indicates the fastest time for a given distance at a given track. The track record allows you to compare the performances of horses that have been running at different tracks.

For some reason, beginning in the late 1960s, “track record” spread far beyond the track.


The results were disastrous. Now, just about everybody except horseplayers – i.e., just about everybody – says “track record” when they mean “past performances.”

In the examples above, only #5 uses “track record” as it should be used (pardon my prescriptivism). Nate Silver (#3) even calls his blog post “Handicapping the Republican Field.” For Godssakes Nate, if you’re going call what you do “handicapping,” use the language of handicapping correctly.

It’s possible that this use of “track record” differs slightly from “past performances” and that the speaker is referring to someone’s record in some particular bailiwick, some metaphoric “track.” But in nearly every case, there is no other “track” that it could possibly be confused with. The added word “track” is totally unnecessary. Go back and read the sentences above, mentally removing the “track.” Go ahead, I’ll wait.

A simple “record” instead of “track record” retains all the meaning. And to my ear, once you remove the distraction of “track,” the sentences are clearer.

Why do people need to misuse the beautiful and precise language of horseplayers? Maybe they think that using a racing term gives their persona a dash of risk and romance, the roguish and the raffish. But inevitably, they get it wrong.

History and Horse Races - Run Only Once

July 5, 2011
Posted by Jay Livingston

Here’s sentence from a New York Times op-ed by David Greenberg last week.

Modern Republican isolationism began with the 1919 battle over joining the League of Nations, when Senate Republicans, led by so-called Irreconcilables like William Borah of Idaho, killed the deal - even though without American guidance, European affairs were doomed to explode again.

I’ve read sentences like this hundreds of times – I’ve probably written some myself. But last week, I had been reading Everything is Obvious* (*Once You Know the Answer) by Duncan Watts, and I got stuck on doomed. “European affairs were doomed to explode again.”

Were they? Was the explosion inevitable? And was the absence of US guidance a necessary cause of the explosion?

It certainly seems that way. We know that Europe exploded in war twenty years after that 1919 decision by the US to stay out of the League. But we know that only in retrospect. At the time, there may have been several other equally plausible outcomes.

It’s like saying that the 2011 Belmont Stakes was “doomed” to be won by Ruler on Ice. True, he did win. But before the start of the race, there were many other plausible outcomes, most of them more likely. If the same race were held tomorrow – the same twelve horses running on a similarly sloppy track – would Ruler on Ice inevitably win? I doubt it. (If he were again 24-1, I might be tempted to put a small bet on his nose, but I wouldn’t be at all confident of collecting.)

The trouble with history is that, like the 2011 Belmont, it’s run only once. And after it’s run, historians and op-ed writers do the same thing that horseplayers do after a race: they go back to the Racing Form, the past performances, and pick out the information nuggets, often pebble-sized, that account for the results. Sometimes they even add their own speculation as fact. A horseplayer might say that Ruler on Ice would also have won the Kentucky Derby if only he’d been entered, a statement for which we have zero observations. An op-ed writer might say that the US presence in the League of Nations would have prevented World War II, a speculation based on a similar number of observations.

Watts suggests a different way of looking at history. Here, for example, is his take on the surge in Iraq. In the fall of 2007, the US upped its Iraq force by 30,000 troops. By the next summer, violence had substantially decreased. Conclusion: the surge worked. It reduced the violence.

Or did it?

Many other things happened between the fall of 2007 and the summer of 2008 as well. Sunni resistance fighters, seeing an even greater menace from hard-core terrorist organizations like Al Qaeda than from American soldiers, began to cooperate with their erstwhile occupiers. The Shiite militia – most importantly Moktada Sadr’s Mahdi Army – also began to experience a backlash from their grassroots, possibly leading them to moderate their behavior. And the Iraqi Army and police forces, finally displaying sufficient competence to take on the the militias, began to assert themselves, as did the Iraqi government. Any one of these factors might have been at least as responsible for the drop in violence as the surge. Or perhaps it was some combination. Or perhaps it was something else entirely.

We can’t know because history is run only once.

Ideally, we should be able to go back, examine our variables, look at the possible scenarios, and assign each outcome a probability. Maybe the outcome that did happen would have a lower probability than others, and we would wind up saying that what happened was a fluke or at least improbable. But that’s not what we do.

Rather than producing doubt, the absence of “counterfactual” versions of history tends to have the opposite effect – namely that we tend to perceive what actually happened as having been inevitable.

As I was reading this part of Watts’s book, I kept thinking of Hans J. Morgenthau, the great political scientist. This was many years ago, perhaps during Vietnam, and after Morgenthau’s analysis, typically thoughtful and insightful, someone in the audience asked him what we could expect as the outcome. In his elegant, German-accented English. Morgenthau said,

Well, the answer to that is that I am a professor, not a prophet. I cannot tell you what will happen. I can only tell you why what did happen was absolutely bound to happen.

Will Ruler on Ice beat those same horses again? Will another surge reduce violence? Maybe, but don’t bet on it.

Whose Declaration Is It Anyway?

July 4, 2011
Posted by Jay Livingston

Mark Kleiman complains that the old Fourth-of-July custom of reading the Declaration of Independence “seems to have virtually disappeared.”

In a post three years ago, I reported that the custom was still alive and well in liberal Lenox, Mass. (Here’s the photo, the full post is here.)


Patriotism has usually been a value waved about by conservatives, especially when they are in power. But this was 2008, and after seven years of Bush-GOP rule, the liberals were starting to play Capture the Flag, a game they had been mostly sitting out of for decades.

When they are out of power, conservatives put their dissent-equals-disloyalty rhetoric back in the closet. The Declaration is an anti-government document. So I expect that we will see it read at many gatherings of the Tea Partistas. They have been calling Obama a tyrant for a while now, so they will be whistling and stomping their feet at every mention of tyranny and taxation. They’ll especially like the line about the “multitude of new offices.”

I wonder if they will feel uncomfortable (as the Lenox liberals did) at the negative tone in the references to “domestic insurrections” (i.e., slave rebellions) and “merciless savages” (native Americans).