David Baxter PhD
Late Founder
Now John Bargh’s Famous Hot-Coffee Study Has Failed To Replicate
By Jesse Singal, BPS Research Digest
January 2, 2019
If you Google “holding a warm cup of coffee can” you’ll get a handful of results all telling the same story based on social priming research (essentially the study of how subtle cues affect human thoughts and behavior). “Whether a person is holding a warm cup of coffee can influence his or her views of other people, and a person who has experienced rejection may begin to feel cold,” notes a New York Times blog post, while a Psychology Today article explains that research shows that “holding a warm cup of coffee can make you feel socially closer to those around you.”
These kind of findings are most often associated with John Bargh, a Yale University professor and one of the godfathers of social priming. In his 2017 book Before You Know It: The Unconscious Reasons We Do What We Do, Bargh goes further, even suggesting – based on social priming studies and a small study that found two hours of “hyperthermia” treatment with an infra lamp helped depressed in-patients – that soup might be able to treat depression. “After all,” he writes, “it turns out that a warm bowl of chicken soup really is good for the soul, as the warmth of the soup helps replace the social warmth that may be missing from the person’s life, as when we are lonely or homesick.” He continues, “These simple home remedies are unlikely to make big profits for the pharmaceutical and psychiatric industries, but if the goal is a broader and more general increase in public mental health, some research into their possible helpfulness could pay big dividends for individuals currently in distress, and for society as a whole.”
Of course, it’s a pretty big leap from social-priming studies conducted in labs, or from a small study of a very unusual population, to the idea that maybe chicken soup can rival Prozac — especially when one remembers that millions of clinically depressed people drink hot coffee every day, to no apparent effect. But this nicely captures a common, and important, critique of social priming: Social-priming enthusiasts, both within and outside of academia, have sometimes over-extrapolated on the basis of limited or questionable findings.
Over-extrapolation is one thing, but what if the original studies simply weren’t even robust in the first place? Recently, a team led by Christopher Chabris and Dan Simons (best known for their “invisible gorilla” work on inattentional blindness), plus other colleagues, sought to replicate two of Bargh’s famous temperature-based findings from 2008. As reported by Research Digest at the time, Bargh and his colleague Lawrence Williams contrived a way to have undergrad participants in a lab hold either a hot or a cold drink and then rate a target individual. Students who held the warm drink rated the individual higher on traits having to do with warmth than students who held the cold drink. In a follow-up study conducted out in the world, Williams and Bargh also found that those “holding a hot (versus cold) therapeutic pad were more likely to choose a gift for a friend instead of for themselves,” the theory here being that a sense of warmth makes people more giving and other-focused.
Chabris and his colleagues attempted to replicate both these findings, following the format of the original research quite closely, but on participant samples that were triple the size of the originals, and which likely more closely resemble the general population (for both experiments, rather than relying on undergrads, they recruited people off the streets of Saratoga Springs, New York). As they report in their preprint available at PsyArXiv, and due to be published in Social Psychology, they found no effects of drink temperature or hot pads on their participants’ judgments or behaviour. Moreover, Chabris and his colleagues used Bayesian analysis (a way of estimating probabilities) to show that “there is substantially more evidence for the null hypothesis of no effect than for the original physical warmth priming hypothesis.”
Of course, as Chabris et al note, this still leaves open the possibility that the hot-coffee and hot-pad effects are real – perhaps they simply failed to observe a significant result because of statistical bad luck. Or maybe the effects are real, but bounded in certain important ways. Maybe they only work in a lab, or only among student participants, or only under certain other conditions met in the first study but not the second. Therein lies the problem for social priming both as a scientific field and as a pop-cultural phenomenon: As failed replications pile up, as they seem to be in this domain (other famous social priming effects that have proven hard to re-create include words related to old-age leading participants to walk more slowly; washing hands cleaning the conscience; and thoughts of money making us selfish), it gets harder and harder to come up with theories that can 1) explain both the original, exciting-seeming findings and the failed replications; and which 2) aren’t incredibly tangled or boring.
I’ll use an extreme example to illustrate the point: Imagine if it really is the case that this particular hot-coffee priming effect only works on college students under a certain age in lab settings, as opposed to out in the world. What would anyone do with that fact? Why would it matter? It certainly couldn’t be justifiably used to make sweeping statements, presumably applicable to humans writ large, about how “Holding a warm cup of coffee can” do X or Y (let alone that hot soup can cure depression).
But regrettably that’s how these sorts of findings have all been too often communicated: “Holding a warm cup of coffee can make you feel warmer toward others!” And so on. It’s much rarer for social-priming findings — or other sexy research findings from behavioural science — to be presented in an appropriately hedged, scientifically responsible way. That wouldn’t be exciting enough.
Reference
No Evidence that Experiencing Physical Warmth Promotes Interpersonal Warmth: Two Failures to Replicate Williams and Bargh (2008)
By Jesse Singal, BPS Research Digest
January 2, 2019
If you Google “holding a warm cup of coffee can” you’ll get a handful of results all telling the same story based on social priming research (essentially the study of how subtle cues affect human thoughts and behavior). “Whether a person is holding a warm cup of coffee can influence his or her views of other people, and a person who has experienced rejection may begin to feel cold,” notes a New York Times blog post, while a Psychology Today article explains that research shows that “holding a warm cup of coffee can make you feel socially closer to those around you.”
These kind of findings are most often associated with John Bargh, a Yale University professor and one of the godfathers of social priming. In his 2017 book Before You Know It: The Unconscious Reasons We Do What We Do, Bargh goes further, even suggesting – based on social priming studies and a small study that found two hours of “hyperthermia” treatment with an infra lamp helped depressed in-patients – that soup might be able to treat depression. “After all,” he writes, “it turns out that a warm bowl of chicken soup really is good for the soul, as the warmth of the soup helps replace the social warmth that may be missing from the person’s life, as when we are lonely or homesick.” He continues, “These simple home remedies are unlikely to make big profits for the pharmaceutical and psychiatric industries, but if the goal is a broader and more general increase in public mental health, some research into their possible helpfulness could pay big dividends for individuals currently in distress, and for society as a whole.”
Of course, it’s a pretty big leap from social-priming studies conducted in labs, or from a small study of a very unusual population, to the idea that maybe chicken soup can rival Prozac — especially when one remembers that millions of clinically depressed people drink hot coffee every day, to no apparent effect. But this nicely captures a common, and important, critique of social priming: Social-priming enthusiasts, both within and outside of academia, have sometimes over-extrapolated on the basis of limited or questionable findings.
Over-extrapolation is one thing, but what if the original studies simply weren’t even robust in the first place? Recently, a team led by Christopher Chabris and Dan Simons (best known for their “invisible gorilla” work on inattentional blindness), plus other colleagues, sought to replicate two of Bargh’s famous temperature-based findings from 2008. As reported by Research Digest at the time, Bargh and his colleague Lawrence Williams contrived a way to have undergrad participants in a lab hold either a hot or a cold drink and then rate a target individual. Students who held the warm drink rated the individual higher on traits having to do with warmth than students who held the cold drink. In a follow-up study conducted out in the world, Williams and Bargh also found that those “holding a hot (versus cold) therapeutic pad were more likely to choose a gift for a friend instead of for themselves,” the theory here being that a sense of warmth makes people more giving and other-focused.
Chabris and his colleagues attempted to replicate both these findings, following the format of the original research quite closely, but on participant samples that were triple the size of the originals, and which likely more closely resemble the general population (for both experiments, rather than relying on undergrads, they recruited people off the streets of Saratoga Springs, New York). As they report in their preprint available at PsyArXiv, and due to be published in Social Psychology, they found no effects of drink temperature or hot pads on their participants’ judgments or behaviour. Moreover, Chabris and his colleagues used Bayesian analysis (a way of estimating probabilities) to show that “there is substantially more evidence for the null hypothesis of no effect than for the original physical warmth priming hypothesis.”
Of course, as Chabris et al note, this still leaves open the possibility that the hot-coffee and hot-pad effects are real – perhaps they simply failed to observe a significant result because of statistical bad luck. Or maybe the effects are real, but bounded in certain important ways. Maybe they only work in a lab, or only among student participants, or only under certain other conditions met in the first study but not the second. Therein lies the problem for social priming both as a scientific field and as a pop-cultural phenomenon: As failed replications pile up, as they seem to be in this domain (other famous social priming effects that have proven hard to re-create include words related to old-age leading participants to walk more slowly; washing hands cleaning the conscience; and thoughts of money making us selfish), it gets harder and harder to come up with theories that can 1) explain both the original, exciting-seeming findings and the failed replications; and which 2) aren’t incredibly tangled or boring.
I’ll use an extreme example to illustrate the point: Imagine if it really is the case that this particular hot-coffee priming effect only works on college students under a certain age in lab settings, as opposed to out in the world. What would anyone do with that fact? Why would it matter? It certainly couldn’t be justifiably used to make sweeping statements, presumably applicable to humans writ large, about how “Holding a warm cup of coffee can” do X or Y (let alone that hot soup can cure depression).
But regrettably that’s how these sorts of findings have all been too often communicated: “Holding a warm cup of coffee can make you feel warmer toward others!” And so on. It’s much rarer for social-priming findings — or other sexy research findings from behavioural science — to be presented in an appropriately hedged, scientifically responsible way. That wouldn’t be exciting enough.
Reference
No Evidence that Experiencing Physical Warmth Promotes Interpersonal Warmth: Two Failures to Replicate Williams and Bargh (2008)
