A new study warns that research suggesting a genetic-environmental linkage for the development of some psychiatric illnesses needs additional scrutiny and replication before being accepted as true.
McLean Hospital/Harvard Medical School investigator Laramie Duncan, Ph.D., and co-author Matthew Keller, Ph.D., at the University of Colorado conducted a comprehensive review of a decade of mental health research linking genetics and environmental variables.
“Based on our calculations and data from related fields, we estimate that many of the positive findings in this particular area of research may (unintentionally) be incorrect,” said Duncan.
A core tenet of scientific research is the need to replicate or redo studies to see if the same results occur time after time. Conceptually, after the research is replicated on several occasions, then the findings can be trusted and introduced into clinical practice.
“What we suggest, to be certain about such correlations, is a focus on the cornerstone of scientific investigation—which is replication. The more we can replicate original findings in follow-up studies, the more we can be sure the results are accurate,” says Duncan.
Duncan stressed that her paper is not meant to call for skepticism about the existence of gene-by-environment interactions, or psychiatric research in general, but to shed light on the fact that consistent, replicable results deserve more attention than novel findings and indirect replications.
“Genetic research is like trying to identify a needle in a haystack and statistically, it is predicable that investigators will find and report false positives,” said Duncan. “To separate the wheat from the chaff, we need to do follow up studies and replicate the outcomes. That’s the only way we can differentiate between accurate findings and the inevitable false positives.”
Duncan and her co-author categorized each of the 103 research studies conducted in the field of gene-by-environment interaction research in psychiatry between 2000 and 2009 as either novel–representing the first reports of particular interactions–or replication studies—efforts to confirm other researchers’ results.
Upon analyzing the data, Duncan and her team found that the rate of published significant novel or first-time results far outnumbered the rate of replicated results of the same studies.
Past reviews have observed that novel findings are more likely to be published but are also more likely to be false positives.
“Upon comparing novel with replication studies, we realized that the many discrepancies are likely a result of publication bias toward positive findings,” said Duncan.
According to Duncan and Keller, the tendency to favor publishing significant results over non-significant results is common in research, due both to the desire of publications to showcase groundbreaking findings, and to the decision of authors not to submit null findings.
But Duncan warns that this bias can be misleading, if one is unaware of it and does not take into account in interpreting the validity of published findings.
“Publication bias is problematic because it produces a distorted representation of findings in an area of study,” said Duncan.
“Through our research we found that 96 percent of novel studies were significant compared with just 27 percent of replication attempts, suggesting that novel findings appear much stronger than they actually are.”
The study, “A Critical Review of the First 10 Years of Candidate Gene-by-Environment Interaction Research in Psychiatry,” is now available online ahead of print publication in the American Journal of Psychiatry.