David Baxter PhD
Late Founder
Research linking violent entertainment to aggression retracted after scrutiny
by Cathleen O’Grady, ScienceFeb 16, 2021
Do violent movies like Sin City lead to aggression? Retracted studies have muddied the debate. (AF archive/Alamy Stock Photo)
As Samuel West combed through a paper that found a link between watching cartoon violence and aggression in children, he noticed something odd about the study participants. There were more than 3000—an unusually large number—and they were all 10 years old. “It was just too perfect,” says West, a Ph.D. student in social psychology at Virginia Commonwealth University.
Yet West added the 2019 study, published in Aggressive Behavior and led by psychologist Qian Zhang of Southwest University of Chongqing, to his meta-analysis after a reviewer asked him to cast a wider net. West didn’t feel his vague misgivings could justify excluding it from the study pool. But after Aggressive Behavior published West’s meta-analysis last year, he was startled to find that the journal was investigating Zhang’s paper while his own was under review.
It is just one of many papers of Zhang’s that have recently been called into question, casting a shadow on research into the controversial question of whether violent entertainment fosters violent behavior. Zhang denies any wrongdoing, but two papers have been retracted. Others live on in journals and meta-analyses—a “major problem” for a field with conflicting results and entrenched camps, says Amy Orben, a cognitive scientist at the University of Cambridge who studies media and behavior. And not just for the ivory tower, she says: The research shapes media warning labels and decisions by parents and health professionals.
The investigations were triggered by Illinois State University psychologist Joe Hilgard, who published a blog post last month cataloging his concerns about Zhang’s work. Hilgard was initially impressed when he came across a 2018 paper of Zhang’s in Youth & Society, another study with 3000 subjects. “I was like, holy smokes!” he says. The study found some teenagers were more aggressive after playing violent video games. Given the huge sample size, it had the potential to be a “powerful chunk of evidence,” Hilgard says.
But he found the paper’s statistics mathematically impossible. Zhang and his co-authors reported high levels of statistical significance for their finding, but the reported differences in the effects of violent games versus nonviolent games were too small for that high statistical significance to be possible. Hilgard alerted Zhang and the journal, and Zhang submitted a correction. Hilgard says that made the statistics seem more plausible, but they were still incorrect.
Hilgard says he found problems in other papers of Zhang’s, such as nearly identical results reported in three different papers. He emailed Zhang and asked to see his data, but he says Zhang refused. Hilgard then contacted Dorothy Espelage, a psychologist at the University of North Carolina, Chapel Hill, and co-author with Zhang on multiple papers. She told Hilgard that Zhang had refused to send her the data, too. It was only after Hilgard asked Southwest University to investigate that Zhang sent Hilgard data for a Youth & Society paper on movie violence.
But the data were odd, Hilgard says, and missing features normally found in similar experiments. He sent his findings to Zhang’s university, which said it found problems, but not fraud, with Zhang’s data, and that Zhang was “deficient in statistical knowledge and research methods.” Dissatisfied with the response, Hilgard sent his observations to all the journals involved.
Marc Zimmerman, a psychologist at the University of Michigan, Ann Arbor, and the editor of Youth & Society, says Hilgard’s email came a week after another source questioned Zhang’s data. Two months later, in December 2019, the journal retracted Zhang’s two papers. Without both whistleblower reports, Zimmerman says he might not have acted so promptly or decisively. “No journal editor wants to publish wrong data,” he says, “but it’s a serious accusation.”
In an email to Science, Zhang denied any misconduct, and wrote “there is never just one ‘right’ way to look at the data.” He wrote that Hilgard, who is skeptical of the link between violent games and aggression, is “trying to make his name based just on claiming that everyone else does bad research.” But Jay Hull, a psychologist at Dartmouth College whose work shows a link between violent video games and aggression, agrees with Hilgard’s concerns about Zhang’s work. More generally, Hull is worried about a “science that’s built on shifting sands.”
Other journals, including Aggressive Behavior, have not retracted any of Zhang’s research, even though Espelage supported a retraction of the Aggressive Behavior paper on which she was a co-author. Craig Anderson, a psychologist at Iowa State University and editor-in-chief at Aggressive Behavior, declined to comment on the record. The paper’s results will live on within West’s meta-analysis, leaving him unsure how to correct his own work. It’s “profoundly frustrating,” West says.
Zhang’s recent papers avoid the red flags Hilgard has complained about. But Hilgard isn’t satisfied. The delays and lack of action, he says, make him “really anxious about how hard it’s going to be to detect data that is flawed in more subtle ways.”
