Replication Complications

December 14, 2019
Posted by Jay Livingston

Some people can tell a joke. Others can’t. Same joke. One person has everyone laughing, the other gets zilch. Does the null response mean that the joke isn’t funny?

 What we have here is a failure to replicate.

 A couple of days ago, the Psychology Archive (PsyArXiv) published results showing a failure to replicate an experiment on Terror Management Theory (TMT).* Among the possible reasons for this failure, the authors say,

There was substantial nuance required in implementing a successful TMT study. . . . These nuances include how the experimenter delivers the experimental script (tone, manner ). . .

I offered this same idea five years ago. I didn’t use the term “nuance.” Instead, I speculated that some experimenters knew how to “sell it” —  “it” in this case being the basic manipulation or deception in the experimental set-up. You can read the whole post (here), but here’s a somewhat shorter replication. I’m copy-and-pasting because as we get more results from replication studies, it’s still relevant. Also, I liked it.

*              *            *             *

One of the seminal experiments in cognitive dissonance is the one-dollar-twenty-dollar lie, more widely known as Aronson and Carlsmith, 1963. Carlsmith was J. Merrill Carlsmith. The name itself seems like something from central casting, and so did the man – a mild mannered, WASP who prepped at Andover, etc. Aronson is Eliot Aronson, one of the godfathers of social psychology, a Jewish kid from Revere, a decidedly non-preppy city just north of Boston.

In the experiment, the subject was given a boring task to do — taking spools out of a rack and then putting them back, again and again — while Carlsmith as experimenter stood there with a stopwatch. The next step was to convince the subject to help the experimenter. In his memoir, Not by Chance Alone, Aronson, describes the scenario.

[Merrill] would explain that he was testing the hypothesis that people work faster if they are told in advance that the task is incredibly interesting than if they are told nothing and informed, “You were in the control condition. That is why you were told nothing.” At this point Merrill would say that the guy who was supposed to give the ecstatic description to the next subject had just phoned in to say he couldn't make it. Merrill would beg the “control” subject to do him a favor and play the role, offering him a dollar (or twenty dollars) to do it. Once the subject agreed, Merrill was to give him the money and a sheet listing the main things to say praising the experiment and leave him alone for a few minutes to prepare.

But Carlsmith could not do a credible job. Subjects immediately became suspicious.

It was crystal clear why the subjects weren't buying it: He wasn't selling it. Leon [Festinger] said to me, “Train him.”

Sell it. If you’ve seen “American Hustle,” you might remember the scene where Rosenfeld (Christian Bale) is trying to show the FBI agent disguised as an Arab prince how to give a gift to the politician they are setting up.  (The relevant part starts at 0:12 and ends at about 0:38)

Here is the script:

Aronson had to do something similar, and he had the qualifications. As a teenager, he had worked at a Fascination booth on the boardwalk in Revere, Massachusetts, reeling off a spiel to draw strollers in to try their luck.

Walk right in, sit in, get a seat, get a ball. Play poker for a nickel. . . You get five rubber balls. You roll them nice and easy . . . Any three of a kind or better poker hand, and you are a winner. So walk in, sit in, play poker for a nickel. Five cents. Hey! There’s three jacks on table number 27. Payoff that lucky winner!

Twenty years later, he still had the knack, and he could impart it to others.

I gave Merrill a crash course in acting. “You don't simply say that the assistant hasn't shown up,” I said. “You fidget, you sweat, you pace up and down, you wring your hands, you convey to the subject that you are in real trouble here. And then, you act as if you just now got an idea. You look at the subject, and you brighten up. ‘You! You can do this for me. I can even pay you.’”

The deception worked, and the experiment worked. When asked to say how interesting the task was, the $1 subjects give it higher ratings than did the $20 subjects. Less pay for lying, more attitude shift.

 The experiment is now part of the cognitive dissonance canon. Surely, others have tried to replicate it. Maybe some replications have not gotten similar results. But that does not mean we should toss cognitive dissonance out of the boat. The same may be true for TMT. It’s just that some experimenters are good at instilling terror, and others are not.

----------------------------------
  * If you’ve never heard of TMT (I hadn’t), it’s basically the idea that if you get people to think about their own mortality, their attitudes will become more defensive about themselves and their group. Of the twenty-one replications, a very few got results that supported TMT, a very few got results that contradicted TMT. Most found no statistically significant or meaningful differences. 

Here’s the set-up for the independent variable: The subjects in the Terror condition were asked to write about “the emotions they experienced when thinking about their own death, and about what would happen to their physical body as they were dying and once they were dead.” The non-Terror subjects were asked to write about the same things about watching television — e.g., what happens to your physical body when you watch TV. (I am not making this up.)

Methodological Trees and Forests

December 12, 2019
Posted by Jay Livingston

The units of analysis that researchers choose usually constrain the explanations they come up with. Measuring variables on individuals makes it harder to see the effects of larger units like neighborhoods.

For example, much research has found a correlation between female-headed households and crime. Most explanations for this correlation focus on the households, with much talk about the lack of role models or the quality of parent-child interaction. But these explanations are looking at individual trees and ignoring the forest. The better question is not “What are the effects of growing up in a single-parent home?” It’s “What are the effects of growing up in a neighborhood where half the households are headed by single mothers?”

In the early 1990s, I wrote a criminology textbook, and one of the things that differentiated it from others was that it took seriously the idea of neighborhoods and neighborhood-level variables.

That was then. But now, Christina Cross in a recent Times op-ed makes a similar argument. Research generally shows that it’s better for kids to grow up with two parents rather than one. That fits with our assumptions about “broken homes” even if we now call them “single-parent households.” But Cross’s research finds a crucial Black-White difference in the importance of this one dimension of family structure.

Looking at educational outcomes, she finds that White kids from two-parent families do much better than their single-parent counterparts. But for Black kids, the advantage of a two-parent home is not so great.

living in a single-mother family does not decrease the chances of on-time high school completion as significantly for black youths as for white youths. Conversely, living in a two-parent family does not increase the chances of finishing high school as much for black students as for their white peers.

 Why does a two-parent family have less impact among Blacks? Cross looks at two explanations. The first is that the effect of a very low-income neighborhood (“socioeconomically stressful environments”) is so great that it washes out most of the effect of the number of parents inside the home. For a kid growing up in an area with a high concentration of poverty, having a father at home might make a difference, but that difference will be relatively small, especially if the father is unemployed or working for poverty-level wages.

The other explanation is that having other relatives close by mitigates the impact of having only one parent in the home. Cross says that her data supports this idea, but the extend-family network explanation is not nearly as powerful as the neighborhood-poverty explanation.

For policy-makers, what all this means is that the traditional conservative, individual-based solutions miss the point. Exhorting people to stay married (and providing costly government programs along the same lines) aren’t going to have much impact as long as we still have racially segregated neighborhoods with high levels of unemployment and poverty.

The message for researchers is similar: if you confine your thinking or your variables to individuals, you risk ignoring more important variables.

Whatever Happened to “Broken Homes”?

December 11, 2019
Posted by Jay Livingston

“Just think about the last time you heard someone use the term “broken home” or “single-parent household” to explain the misbehavior or misfortune of a person in your social circle.” That’s from a Times op-ed (here) by Christina Cross, a sociology post-doc at Harvard.

I think the last time I heard “broken homes” was before Ms. Cross was born. It’s so 1950s, with its judgmental pronouncement on families that didn’t look like “Ozzie and Harriet” or “Father Knows Best.” In the 70s, as more middle-class people were getting divorced, we needed a less value-laden term. Enter “single parent.”

(Google nGrams shows the frequency of words in books, so the change in the use of these terms in the media and everyday talk probably happens a year or two earlier.)

“Single-parent” is not as blatantly stigmatizing as “broken homes,” but when we hear it, we still think that something is wrong. The more important point that Ms.  Cross makes is that broken homes — the harmful outcomes they bring — may be much more consequential for Whites than for Blacks. I hope to get to that in a later post. But for now, I'll just point out that the sharp decline in mentions of “single-parent” starting in the early 90s tracks with the decline of teen crime and teen pregnancy in the same period.

Alumn. . .us / a / i / ae / x?

December 9, 2019
Posted by Jay Livingston

Welsey Yang, a writer by trade, must have been browsing messages from his alma mater, Rutgers, when this caught his attention. He posted it to Twitter.

(Click on an image for a larger view. I have added the blue oval for emphasis.)

English has a problem with words that are imported from Latin and Romance languages. What do we do about the gender and number that require Latin to have four different forms of the word? Alumnus, alumna, alumni, alumnae. It’s just too many to keep track of, especially since the plurals, masculine and feminine, sound alike.

The simple solution is to use a single form of the word to cover everyone, and that’s what many people have done. For that single form, there are two choices. One is alum, plural alums, which conveniently gets rid of those gendered endings in the Latin versions.

The other is the all-purpose alumni, which many Engish speakers now use indiscriminately for either gender and for singular or plural. It must drive Latin scholars up the ivy-covered wall. Both of these have been getting more popular lately. I searched for them on Nexis-Uni’s database of news sources.

(Some of the increase in these numbers may be attributable to the increased size of the Nexis database. But I doubt that it has grown by the multiple of 20-30 since the 1990s that we see for an alum and an alumni.)

A Google search for “she is an alumni of” gets 177,000 results, slightly less than the 191,000 for “she is an alumna of.” For men, “an alumnus” still outscores “an alumi” by a factor of 10, but that includes a lot of old sources. I would bet that the ratio decreases with time.

Personally, I would avoid the problem completely and go with graduates or, in less formal settings, grads.

The Rutgers Linguistic department has a different solution to the problem of gender: the very recent coinage alumnx, the word that inspired Yang to tweet. It’s a different sort of solution. The ungendered alum and the all-purpose alumni seem to have cropped up unplanned and without any ax to grind. But alumnx is a deliberate effort to change the language. What’s interesting here, as one of the Yang’s commenters points out, is that as linguists the department members are descriptivists, more interested in describing how people actually use language than in telling people which words to use. But here they are prescribing alumnx as the correct way avoid the less woke Latin forms.

My guess is that these x-words will have a short life. Most people don’t care much about the politics of speech and have little interest in changing. Remember “Freedom Fries”? Worse, alumnx and Latinx don’t resemble real words in English or in any other language.* It’s one thing to replace the “man” in policeman or fireman. “Police officer” and “fire fighter” were already part of the language well before the feminists of the 1960s called our attention to the sexism of the more frequently used terms.

Maybe I’m wrong. Maybe these x-words will become as much a part of the language as Ms. In any case, I expect that the members of the department will, as descriptivist language researchers, monitor how well they are doing as prescriptivist language changers. (Or is it changerx?)

------------------
* Latinx has similar problems. It’s not very popular among the people it is meant to designate, probably because neither English nor Spanish has words in this form. Terry Blas suggests (here) that a better gender-neutral solution would be an “e” rather than an “x” — Latine and Latines.