Contradicted and Initially Stronger Effects in Highly Cited Clinical Research

By John M Grohol PsyD
July 13, 2005

So you know all that research published and trumpeted in the nightly news (and all over the Net) about the latest health findings? Well, 32% of that research is subsequently discovered not to be so strong, or downright wrong:

Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with “negative” results.

This is, of course, one of the points of research — to have the results replicated (or not) to discover if the results are robust, and therefore, that treatment decisions should be made upon them. You simply can’t do that unless you do the initial research and publish the results somewhere (like in a peer-reviewed journal).

Via JAMA.

Dr. John Grohol is the CEO and founder of Psych Central. He has been writing about online behavior, mental health and psychology issues, and the intersection of technology and psychology since 1992.