BROOKE GLADSTONE:
When a paper released by a scientific journal turns out to be wrong, either due to human error or intentional fraud, the journal’s editors often will issue a retraction advising scientists to disregard the research.
A recent Wall Street Journal study has found that the number of such retractions is soaring. In fact, since 2001 the number of retractions issued by journals has increased 15-fold.
We asked Wired Magazine science writer Jonah Lehrer what he thinks is going on.
JONAH LEHRER:
The explanation I’m must attracted to is simply that there’s been a norm change. In years past they used to wait for serious evidence of scientific fraud. And often co- authors would have to insist on retraction. Now I think they're much more willing to retract a paper if there’s simply an error.
BROOKE GLADSTONE:
Can you give me a couple of examples, so we can understand what's at stake?
JONAH LEHRER:
The kind of attractions that are most worrisome are attractions that very quickly become the basis for policy. So we're talking here mostly about retractions in clinical trials.
BROOKE GLADSTONE:
So an example might be a kidney drug.
JONAH LEHRER:
It could be a kidney drug, it could be things like hormone replacement therapy. The list goes on and on and on. I mean, you know, I think people are now pushing back against anti-oxidants, against the beneficial effects of various vitamins. So there are lots and lots of clinical trials that have gotten lots of media attention. And then the retraction is published in the fine print.
I also want to make the point that I don't think there's anything necessarily bad about an increase in retraction. I think, if anything, this represents, you know, a new kind of transparency in peer review.
BROOKE GLADSTONE:
You weren't that bothered by the retraction study, but there was a related study that really did worry you.
JONAH LEHRER:
Yes. This was a rather obscure paper, published [LAUGHS] in a rather obscure journal called Ecosphere, which looked at a small subset of papers, in the fishery sciences. It was done by researchers at the University of Washington. And they were interested in what happens after a paper is refuted or retracted.
So let’s say a paper’s published in Nature or Science, and they just looked at papers published in these two very prestigious journals. And then a few years go by, numerous papers have come out, and they proved that this paper just isn't true, that these researchers made a mistake. What happens next? Is this falsified paper, is it still cited, or do other scientists recognize the mistake?
And what they discovered is, I think, deeply depressing, if you believe in peer review, because they found that the number of citations for the paper which has been soundly refuted don't decrease, and, in fact, that they remain years and years after being refuted at 17 times higher than the refuting papers.
BROOKE GLADSTONE:
Mm-hmm.
JONAH LEHRER:
So it’s as if the mistake has been demonstrated, it just still persists. It’s very tough to get rid of.
BROOKE GLADSTONE:
Well, we know that in the mainstream media once an impression is made, it's very hard to unmake. But you would hold fellow scientists to a higher standard. You would think that they would at least keep up with their own research.
JONAH LEHRER:
You would certainly hope so. And, [LAUGHS] in fact, I think one of the most troubling part of this paper is they show that even when these papers which are showing that the original papers wrong are cited, about 10 percent of the time they're cited as demonstrating affirmative proof.
[BROOKE LAUGHS]
So scientists cite them in the wrong way.
BROOKE GLADSTONE:
So they don't even read their own citations.
[JONAH LAUGHS]
Is this because of an over-reliance on grad students?
JONAH LEHRER:
[LAUGHS] No. I think lots of people use citations without fully reading the papers. They look at the title and maybe the abstract.
I think one thing that's happening is that it's tough to let go of a beloved theory. These papers were very influential often for up to a decade. So the mind, for all sorts of reasons, struggles to let go of those ideas.
BROOKE GLADSTONE:
So what do we do about this?
JONAH LEHRER:
One of the very simple fixes is simply when scientists call up a paper that’s been refuted in Pub Med or Google Scholar or one of these search engines that scientists all over the world use, that they should be automatically linked to the refuting papers.
BROOKE GLADSTONE:
Mm-hmm.
JONAH LEHRER:
So – so if a study isn’t true, we should know that it’s not true, and the papers that prove that it's not true should be right there, as well, so they’re easy to cite and the record is automatically corrected.
BROOKE GLADSTONE:
On the other hand, you said that even when scientists are aware of the contradictory papers, they either don't read them or misread them or use them to support the original wrong conclusions.
JONAH LEHRER:
Well, I think we'd have to invent a new format, be some new kind of link where you’d know that these were citations which weren't supporting the original paper –
[BROOKE LAUGHS]
- but were actually contradicting it.
BROOKE GLADSTONE:
Red font!
JONAH LEHRER:
Exactly.
[BROOKE LAUGHS]
In the scientific literature, the vast majority of citations, of course, are positive. They're showing that they're building on these earlier discoveries. This would be the opposite kind of citation.
BROOKE GLADSTONE:
Mm-hmm.
JONAH LEHRER:
The more ambitious, I think, fixes focus on things like forcing scientists to declare in advance what their hypotheses might be.
BROOKE GLADSTONE:
Mm!
JONAH LEHRER:
So, you know, in a lot of scientific fields, one of the problematic things has been that scientists – let’s say you’re doing some brain scan research and you’re trying to find some significant correlation between people doing X and Y happening in some particular brain area. Because of the standards of scientific publishing, you’ll often have to get a result that passes the mystical test of significance, which shows that it's very, very unlikely that these results were a byproduct of randomness.
BROOKE GLADSTONE:
Mm-hmm.
JONAH LEHRER:
And so, scientists will often keep on searching, keep on replicating experiments until they get that result that passes the test of significance.
And so, the hope of having this kind of database where scientists have to register their original hypotheses is that it’ll make that harder to do, that scientists will have to say in advance what they're looking for, and if they can't find that original relationship, then that’ll be part of the paper they eventually publish. The hope is that it would introduce a new kind of transparency to the scientific process.
BROOKE GLADSTONE:
Jonah, thank you very much.
JONAH LEHRER:
Think you.
BROOKE GLADSTONE:
Jonah Lehrer is a science writer for Wired.
