SAT, GPA, and Bias

July 8, 2011
Posted by Jay Livingston

(Cross-posted at Sociological Images)

Is the SAT biased? If so, against who is it biased?

It has long been part of the leftist creed that the SAT and other standardized tests are biased against the culturally disadvantaged - racial minorities, the poor, et. al. Those kids may be just as academically capable as more privileged kids, but the tests don’t show it.

But maybe SATs are biased against privileged kids. That’s the implication in a blog post by Greg Mankiw. Mankiw is not a liberal. In the Bush-Cheney first term, he was the head of the Council of Economic Advisors. He is also a Harvard professor and the author of a best-selling economics text book. Back in May he had a blog post called “A Regression I’d Like to See.” If tests are biased in the way liberals say they are, says Mankiw, let’s regress GPA on SAT scores and family income. The correlation with family income should be negative.
a lower-income student should do better in college, holding reported SAT score constant, because he managed to get that SAT score without all those extra benefits.
In fact, the regression had been done, and Mankiw added this update:
Todd Stinebrickner, an economist at The University of Western Ontario, emails me this comment: “Regardless, within the income groups we examine, students from higher income backgrounds have significantly higher grades throughout college conditional on college entrance exam . . . scores. [Mankiw added the boldface for emphasis.]

What this means is that if you are a college admissions officer trying to identify the students who will do best in college, as measured by grades, you would give positive rather than negative weight on family income.
Not to give positive weight to income, therefore, is bias against those with higher incomes.

To see what Mankiw means, look at some made-up data on two groups. To keep things civil, I’m just going to call them Group One and Group Two. (You might imagine them as White and Black, Richer and Poorer, or whatever your preferred categories of injustice are. I’m sticking with One and Two.) Following Mankiw, we regress GPA on SAT scores. That is, we use SAT scores as our predictor and we measure how well they predict students’ performance in college (their GPA).

(Click on the image for a larger, clearer view)

In both groups, the higher the SAT, the higher the GPA. As the regression line shows, the test is a good predictor of performance. But you can also see that the Group One students are higher on both. If we put the two groups together we get this.

Just as Mankiw says, if you’re a college admissions director and you want the students who do best, at any level of SAT score, you should give preference to Group One. For example, look at all the students who scored 500 on the SAT (i.e., holding SAT constant at 500). The Group One kids got better grades than did the Group Two kids. So just using the SATs, without taking the Group factor (e..g., income ) into account, biases things against Group One. The Group One students can complain: “the SAT underestimates our abilities, so the SAT is biased against us.”

Case closed? Not yet. I hesitate to go up against an academic superstar like Mankiw, and I don’t want to insult him (I’ll leave that to Paul Krugman). But there are two ways to regress the data. So there’s another regression, maybe one that Mankiw does not want to see.

What happens if we take the same data and regress SAT scores on GPA? Now GPA is our predictor variable. In effect, we’re using it as an indicator of how smart the student really is, the same way we used the SAT in the first graph.
Let’s hold GPA constant at 3.0. The Group One students at that GPA have, on average, higher SAT scores. So the Group Two students can legitimately say, “We’re just as smart as the Group One kids; we have the same GPA. But the SAT gives the impression that we’re less smart. So the SAT is biased against us.”

So where are we?
  • The test makers say that it’s a good test - it predicts who will do well in college.
  • The Group One students say the test is biased against them.
  • The Group Two students say the test is biased against them.
And they all are right.

Huge hat tip to my brother, S.A. Livingston. He told me of this idea (it dates back to a paper from the1970s by Nancy Cole) and provided the made-up data to illustrate it. He also suggested these lines from Gilbert and Sullivan:
And you'll allow, as I expect
That they are right to so object
And I am right, and you are right
And everything is quite correct.


PCM said...

Very interesting.

I'm very torn on the SAT... I'm sure it indicates something... I'm just not sure what (is obscure vocabulary really the best we can do?)

I propose the SAT gives many questions based on subway maps and train schedules. I predict this could reverse a very pro-suburban bias.

PCM said...

Racial bias or not--and I do believe the tests are racially biased---I'd feel a lot more confident if somebody could devise a test that isn't racially biased (see previous comment).

Could it not simply be that people with money, on average, get a better education because more money can buy a better education?

It's a shockingly simply and almost blindingly obvious hypothesis... And yet so vehemently denied by rich suburban Republicans who pay so much for their kids' education but don't want to pay for the education of poorer children whose parents don't have the money to live in better school districts. segregation.

brandsinger said...

" vehemently denied by rich suburban Republicans who pay so much for their kids' education but don't want to pay for the education of poorer children whose parents don't have the money to live in better school districts."

PCM - you're a piece of work.
Do you read the papers? follow the news at all? ... have any idea who favors funding charter schools and giving tax vouchers for poor families to have money for private schooling?... and who opposes it? Must be amazing to be so clueless. Guess you think the Democrats and their teachers unions are the good guys, right? Nice not having to think about the real world around you... like this report from early June:

"Black charter-school parents personally delivered a message to the head of the New York NAACP yesterday: Don't hurt our kids!

State NAACP President Hazel Dukes got an earful during an emotional, private 90-minute meeting at her Bryant Park office when a group of parents urged the civil-rights organization to back out of a lawsuit aimed at halting the co-location or expansion of 20 charter schools in public-school buildings.

"Why is the NAACP trying to block access to a better education for my child," Ny Whitaker, whose son, Taj, is a third-grader at Harlem Success Academy, said following the meeting with Dukes.

"Why does an organization that's fought for opportunities for communities of color file a lawsuit to prevent access to public charter schools?" Whitaker said.

PCM: Why indeed? Check out who is blocking education reform and who favors it.

Nah. Never mind. Stay asleep... "rich Republicans" blah blah blah..

PCM said...

The bully is back. What ever happened to "Sayonara"?

It's one thing to attack me without reading my books... it's another to attack me as if you've didn't read the post or my comments.

I never mentioned charter schools or school reform.

You, sir, have issues.

brandsinger said...

PCM - I'm no bully. You want to be free to share your personal political biases without suffering pain of correction. I know you claim to have written books, but that doesn't mean that the internet will automatically forgive your cavalier slanders.

You wrote this:" suburban Republicans who pay so much for their kids' education but don't want to pay for the education of poorer children."

That's just slander. Should we bow to it? Do you have students who are Republicans? Would they accept that without replying?

I pointed out that, for example, tax vouchers -- giving tax money back to poor parents to pay for private education -- is what many Republicans favor. That's one example of paying more for the education of poorer children... a much better education... which poor families tend to favor and most Democrats and teachers unions oppose.

So you were wrong... and instead of acknowledging it, you refer to books you have written. Quite sad. Perhaps you like a band-aid for having to read my response?

I have a suggestion: When you want to sling around your prejudices about "rich Republicans," prepare for a reply. Better advice: Think before you lapse into caricature of your political opponents.

PCM said...

I don't know whether it's possible to have a polite conversation with you about the facts that you don't turn personal. But I'll try just one more time.

Now here's my point, and it has nothing to do with what type of education is better for the little ones. (maybe it's charter, maybe it's not... I don't really care.)

Now I just did a bit of research to find a New Jersey school district that spends a lot of money per pupil. I don’t know NJ well (despite living nearby and having gone to college there), but Saddle River Boro kind of jumps out at me. And I've never heard of it.

78% of their K-6 school spending comes from the local tax base. And they lead the state (or come close to leading the state) in almost every indicator of money spent: per pupil, number of administrators, pay for administrators, extracurricular, “support services” (whatever that is), and yes, even teacher salaries.

Now based on that alone, I bet I can tell you something about the district: it's rich, white, and Republican.

Now I’ll look it up…


Why am I not surprised?

Median household income: $134K. Republicans outnumber Democrats five to one. And it’s 0.75% African American (which in real numbers means 24 black people).

My point is (and was) simply this: no matter what school system you have, money makes it better. Or at least that's what rich people think when it comes to their own children.

I’m not talking about efficiency or even money well spent. But if you spend more, kids will benefit.

If not, are the good people of Saddle River stupid to spend that money on education? Of course not.

Is it only in rich districts that teacher pay correlate with teacher quality? Of course not. If it were, the rich white Republicans of Saddle River must be the stupidest people in New Jersey. I don't think so.

And my second point is that these same people don’t want to spend their money on other people’s children. When it comes to other people's children, they say, it’s not about the money... it’s about how it’s spent. Maybe it is (in part), but until rich districts spend less on school than poor districts, I don’t want to hear them say it's not about the money.

Now, should you want to reply--and I welcome your response--please do this for me: pretend my sweet, smiling face is right in front of you, eager to hear what you have to say. Remember that I'm not some anonymous archetype to slag off. In other words: mind your manners.

brandsinger said...

I appreciate your newly assumed civil tone (I believe you earlier began by calling me a "bully") and carefully reasoned argument. Good healthy dialogue is what we both want.

My point to you (and I'd love to meet your smiling face over a beer... uh, probably a Chardonnay, I'm guessing) is that you are writing publicly on blogs and that it is best try to avoid slinging around political stereotypes.

"Rich Republicans" are not, as implied by your original provocation, selfishly focused on their own kids' education with no regard for the education of others. The broader picture shows that Republicans favor educational reforms that benefit inner-city kids desperately trying to escape the Democratic-Teachers Union death-grip on their futures. Republicans are willing to fund new charter schools and voucher programs to improve the schooling and lives of these less privileged kids.

You seem to be struggling here (and I don't mean this to be or sound like a put-down -- on the contrary it's a compliment) with your political prejudices... examining them... re-assessing. That's great. Keep it up. Read Orwell, Hitchenson, Mamet and others who realized in their lives the oppression of leftist dogma reinforced by the phony righteousness that the other side doesn't care. The other side does care. The other side opposes the caring done by self-serving technocrats whose goal is, really, well cared-for dependents.

Good luck. We're square.

PCM said...

Well, I tried the nice/civil approach... and it didn't work. At least not in achieving my goal of having a friendly discussion rather than having the strong urge to slug you.

Political stereotypes are certainly better than the personal stereotypes you engage in. More accurate too, since political parties, after all, do have platforms.

And no, I don't like Chardonnay. (Funny... of all the drinks in the world, you pick the one I don't like.)

You are incredible smug, presumptuous, assuming, and... and this part is important: wrong.

I can't say I wasn't warned, but from now on I will simply ignore your comments. Good day.

The proudly ignorant bully has won. The playground, a mighty small playground at that, is yours.