Experiments and the Real World

May 26, 2018
Posted by Jay Livingston

Two days ago, the NY Times published an op-ed by Tali Sharot and Cass Sunstein, “Would You Go to a Republican Doctor?” It is based on a single social psychology experiment. That experiment does not involve going to the doctor. It does not involve anything resembling choices that people make in their real lives. I was going to blog about it, but Anderw Gelman (here) beat me to it and has done a much better and more thorough job than I could have done. Here, for example, is a quote from the op-ed and Gelman’s follow-up.

“Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it?”

I followed the link to the research article and did a quick search. The words “doctor,” “accountant,” and “architect” appear . . . exactly zero times.

Gelman takes the article apart piece by piece. But when you put the pieces together, what you get is a picture of the larger problem with experiments. They are metaphors or analogies. They are clever and contrived. They can sharpen our view of the world outside the lab, the “real” world  — but they are not that world.

 “My love is like a red, red rose.” Well, yes, Bobby, in some ways she is. But she is not in fact a red, red rose.

Here is the world of the Sharot-Sunstein experiment.

We assigned people the most boring imaginable task: to sort 204 colored geometric shapes into one of two categories, “blaps” and “not blaps,” based on the shape’s features. We invented the term “blap,” and the participants had to try to figure out by trial and error what made a shape a blap. Unknown to the participants, whether a shape was deemed a blap was in fact random.

The 97 Mechanical Turkers in the experiment had to work with a partner (that is, they thought they would work with a partner – there was no actual collaboration and no actual partner). Players thought they would be paid according to how well they sorted blaps. The result:

[The players] most often chose to hear about blaps from co-players who were politically like-minded, even when those with different political views were much better at the task.

To repeat, despite the title of the article (“Would You Go to a Republican Doctor?”), this experiment was not about choosing a doctor. To get to New York Times readers choosing doctors, you have to make a long inferential leap from Mechanical Turkers choosing blap-sorters. Sharot-Sunstein are saying, “My partner in sorting ‘blaps’ is like a red, red rose a doctor or an architect.” Well, yes, but . . . .

See the Gelman post for the full critique.

Full disclosure: my dentist has a MAGA hat in his office, and I’m still going back for a crown next month. A crown is like a hat in some ways, but not in others. 


Unknown said...

I'd recommend going to a different dentist. You'll feel better if your money isn't going to someone trying to destroy your country for their own benefit.

Jay Livingston said...

David, I've been going to this same dentist for several decades, he's very good, and I'm not going to switch now. Besides, my note was a bit misleading. Trump had been a client of my dentist and gave him the hat, which sits on a shelf in his private office, not on display in the waiting room. I doubt that the dentist voted for him.

Trump told him he'd support an area of research the dentist is part of. I don't know whether that turned out to be the case, but at the time, there was not so much evidence that Trump could reverse himself on everything. In any case, we live in New York State so our votes don't really matter.

brandsinger said...

David Littleboy -- Good thinking. You should start questioning everyone you deal with to discover their political affiliations. You might begin with your mail carrier -- probably a Democrat but could be a secret Trump supporter. Then investigate how your plumber voted. If for Trump, change plumbers. Check the hardware store where you buy bulbs, fixtures and tools. Do you have a cable repair person? How about the ticket takers at the airport? I think before you take a trip you should investigate carefully lest you're in the hands of a Republican pilot and flight attendants. You never want to give your money -- or even be polite -- to people "trying to destroy your country for their own benefit." I'm joking, of course. I hope you were, too. Otherwise: narrow-minded and moronic.