Or, conversely, researchers might study an already-known effect, find no evidence for it, and conclude that they must have messed up somewhere…even if the original, “known” effect isn’t real and the original authors were the ones with the weird sample! Both versions of the “file-drawer” problem can hurt psychologists’ understanding of how the mind works.
Another result of this lack of incentive to replicate is that researchers tend to wade into hyper-specific areas of study. So, there are thousands, if not tens of thousands, of researchers worldwide studying minutiae [though I doubt that they’d put it that way!] related to how the visual system works, or how neurons are organized in a certain part of the brain. Such a strategy ensures that the researcher is contributing new findings and getting published…even if few people actually care about the results!
But very few researchers bother to re-do existing studies…because that won’t help you get tenure!
Some researchers have calculated that there are likely thousands of ‘things we know that just ain’t so’ in the psychological literature! This number could be — and should be! — vastly reduced by simply running some replication studies, as in the Many Labs Project.
In 2015, the Open Science Collaboration brought widespread attention to the replication issue by redoing 100 published studies in psychology. They found that over 50% of those studies failed to replicate; an alarming result (but Gilbert, King, Pettigrew, and Wilson’s reply pointed out some errors in that paper)! Of course, that procedure begs the question of which finding is “real” — the original study, or the single replication attempt?
When a study is replicated, that replication increases our confidence that the effect isn’t just a mirage. And when a study fails to replicate, it doesn’t mean that the original researcher was wrong, or lying, or careless, or stupid — it simply means that one of the groups of people had some unusual characteristics. To determine which result is the “weird” one, we need to conduct further replication attempts.
This is why replication is such a critical piece of the research puzzle: we don’t care if something happens only once; we do care if it happens often!
But, as the field of psychological research stands today, there’s very little incentive to conduct replications — and plenty of incentive not to! This is why I was actually pleased to see the headline that Dr. Carney has recanted her stance on the “power pose.”
Like any good scientist, she changed her mind when the preponderance of the evidence changed. When the “power pose” effect failed to be reproduced—and failed, and failed, and failed again—it became obvious that the initial finding wasn’t due to some psychological process that happens when you stand in certain postures, but was instead due to some other unknown factors.
Researchers are human, too. They often tend to cling to their own ideas, even in the face of strong counterevidence. So, Dr. Carney should be commended — not scorned — for her willingness to change her mind in the face of new evidence. That’s exactly what scientists are supposed to do!
The entertainment industry, however, has no such practices. That’s why movies such as Limitless and Lucy continue to portray the popular — and thoroughly debunked — myth that we only use 10% of our brains. Occasionally, a person’s entire brain is active at once. But it doesn’t result in increased intelligence or superpowers. It’s called a seizure.
A reporting gap occurs between the excitement to publish a surprising new finding and the long, tedious, and unexciting replication process. After all, nobody remembers what Buzz Aldrin said when he was the second man to step on the Moon! We’ll see if Dr. Cuddy’s TED talk is amended or removed in light of this new evidence…I suspect that it won’t be.
It’s best if we avoid the temptation to jump the gun and talk about something as fact before it’s been replicated. But, as long as we only reward the first researchers to “discover” some effect and ignore the important work of replication, we will get fooled again.
This article was revised from its original version on September 25, 2017.