Experiments and the Real World

May 26, 2018
Posted by Jay Livingston

Two days ago, the NY Times published an op-ed by Tali Sharot and Cass Sunstein, “Would You Go to a Republican Doctor?” It is based on a single social psychology experiment. That experiment does not involve going to the doctor. It does not involve anything resembling choices that people make in their real lives. I was going to blog about it, but Anderw Gelman (here) beat me to it and has done a much better and more thorough job than I could have done. Here, for example, is a quote from the op-ed and Gelman’s follow-up.

“Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it?” I followed the link to the research article and did a quick search. The words “doctor,” “accountant,” and “architect” appear . . . exactly zero times.

Gelman takes the article apart piece by piece. But when you put the pieces together, what you get is a picture of the larger problem with experiments. They are metaphors or analogies. They are clever and contrived. They can sharpen our view of the world outside the lab, the “real” world  — but they are not that world.

 “My love is like a red, red rose.” Well, yes, Bobby, in some ways she is. But she is not in fact a red, red rose.

Here is the world of the Sharot-Sunstein experiment.

We assigned people the most boring imaginable task: to sort 204 colored geometric shapes into one of two categories, “blaps” and “not blaps,” based on the shape’s features. We invented the term “blap,” and the participants had to try to figure out by trial and error what made a shape a blap. Unknown to the participants, whether a shape was deemed a blap was in fact random.

The 97 Mechanical Turkers in the experiment had to work with a partner (that is, they thought they would work with a partner – there was no actual collaboration and no actual partner). Players thought they would be paid according to how well they sorted blaps. The result:

[The players] most often chose to hear about blaps from co-players who were politically like-minded, even when those with different political views were much better at the task.

To repeat, despite the title of the article (“Would You Go to a Republican Doctor?”), this experiment was not about choosing a doctor. To get to New York Times readers choosing doctors, you have to make a long inferential leap from Mechanical Turkers choosing blap-sorters. Sharot-Sunstein are saying, “My partner in sorting ‘blaps’ is like a red, red rose a doctor or an architect.” Well, yes, but . . . .

See the Gelman post for the full critique.

Full disclosure: my dentist has a MAGA hat in his office, and I’m still going back for a crown next month. A crown is like a hat in some ways, but not in others.