In two recent editorials addressing the self-correction of science, Nature editors make genuinely revolutionary promises, apparently catapulting themselves into the vanguard of innovators amongst journals and leaving reactionaries like Cell Press in the dust. Congratulations and encouragement are due, even if the editorials do minimise Nature’s role in creating the current dire situation. However, we argue that Nature cannot and will not keep those promises, because of editorial and corporate conflicts of interest. At best the promises are wishful thinking and at worst cynical window-dressing.
We start with the promises.
Go forth and replicate concludes: “We welcome, and will be glad to help disseminate, results that explore the validity of key publications, including our own.”
Post-publication criticism is crucial, but should be constructive ends with: “criticism itself must be embraced.” (We presume this is intended to apply to Nature.)
To those who have attempted to get a flagship Nature journal to address problems in a paper, these undertakings are nothing short of mind-blowing. So the revolution has started? Maybe not quite yet.
First, let’s agree that Nature journals are currently light years from living up to these ideals. We have in mind a particularly egregious example of continuing evasion and stonewalling at Nature Materials, but readers could certainly list many papers with serious, unaddressed problems where Nature (group) editors are suppressing all discussion and action.
Next, notice that the editorials give no practical information. How exactly will Nature embrace criticism? How will they welcome exploration of the validity of papers? How will they gladly help disseminate said exploration? The procedure remains the black hole it is today—nearly all criticism is swallowed without trace. Ironically, the first noticeable change following their call to embrace criticism was the total disappearance of comments site-wide; at the time of writing they are still unavailable. Even if it is the result of a technical problem, rather than of insecurity about feedback, fixing the problem is clearly not a priority.
Part of the problem is that Nature don’t appear to plan on doing more than publishing, very occasionally, a photogenic complete replication where no fault attaches to the original authors or to the editors who accepted the paper. Everything would be “constructive” and cuddly. The reality of “exploring validity” is quite different and much messier. A scientific proof is often a hugely complex and fragile construction that can be demolished by a single problem. Worse, that problem will usually be of the “definitely should have known better” variety, exposing all involved in the original publication to embarrassment or worse. Another issue is that a fatal criticism may be based upon settled knowledge alone. We observe that editors seem to be particularly impervious to criticism involving logic and calculation, while it is all too easy to put up roadblocks by requiring difficult and unnecessary experimental replication. What do Nature plan to do in such cases?
The potential editorial embarrassment raises the question of conflicts of interest. Nature have never set out any procedure for avoiding the acute conflict of interest and lack of accountability of editors (and editorial teams) who accept low-quality or erroneous work. The fiction that editors are perfectly impartial and competent is not credible. This problem is magnified by the almost universal refusal of editors to consider criticism of work that appeared in other journals. In this respect, Nature’s invitation to criticize work published elsewhere would be of great significance—if carried through. One hopeful example is the recent preprint Magnetic magic in Nature by Markus Meister, subsequently published as Physical limits to magnetogenetics. No doubt eLife enjoyed destroying papers in three Nature journals in one go (see also the discussion on PubPeer). More exceptionally still, Meister was not required to redo all the experiments with ten times the original sample sizes. However, today such cases remain extraordinarily rare.
There are also obvious problems of volume and impact. Nature only publish tiny numbers of papers—those that make the most outrageous claims. How do they plan to disseminate what may prove to be a significant volume of criticism? What format will they use? How will they link criticisms and papers? Laughably, they say that “replication studies are not automatically or consistently linked to the original papers on journal web sites”. Might this state of affairs have something to do with the fact that journals like Nature do everything in their power to avoid such feedback? Nature can have access to our API tomorrow! And how will Nature evaluate criticism against their impact criterion? Currently, most criticism unsurprisingly fails to attain the impact threshold. There needs to be a clear commitment to treat criticism as having the same impact as the original paper—in other words, a commitment to evaluate criticism, rather than suffocating it editorially. The editorial’s analysis that replications lack value and visibility is absolutely correct, but it skirts around the elephant in the room: that lack arises precisely because Nature won’t publish them! Instead of recklessly encouraging researchers to undertake costly replications only to discover that Nature still won’t publish them, the editors should lead by example and act on the cases currently on their desks. (Did we mention problems at Nature Materials?)
Beyond these critical practical matters entirely glossed over by the editorials, we believe that money—the Nature business model—represents an insurmountable obstacle to fulfilling their promises. The profits of a glamour journal like Nature flow from the exploitation of two monopolies—those on information and on scientific glory. The monopoly on information is being quite rapidly eroded by open access and preprints, leaving only scientific glory. Publishing criticism of papers in Nature would damage the image of definitive quality and impact that is absolutely central to their brand and to their surviving market power. Would Nature willingly commit corporate suicide and blow up profit margins of ~35%? Dream on. This will only happen when forced upon them by the whole scientific community; journals may follow but will not lead.
The latest editorial, Post-publication criticism is crucial, but should be constructive, which reads amusingly like an uneasy collaboration between the office idealist and a grizzled corporate apparatchik, also slings some mud at PubPeer, fusses about politeness in scientific discussion and exhibits a worrying view of scientific quality. We address these points next.
The editorial snidely implies that a random comment on PubPeer can unfairly result in a retraction and even job loss. This is of course ridiculous. An institution will only fire a researcher following commentary on PubPeer if thorough investigation subsequently uncovers very serious problems. Similarly, we would have noticed if every comment on PubPeer had editors scrambling to retract the paper! In reality, a retraction is never undertaken lightly and usually only with great editorial reluctance, if not outright obstruction. We are confident that Nature editors would require quite incontrovertible proof (in triplicate) before they would consider such a move. The introduction to the editorial refers in particular to Statcheck comments on PubPeer. Contrary to the editors’ bald assertions, those carefully worded comments do not “lack… context” and they are perfectly harmless. Moreover, we bet that Statcheck has a much lower false discovery rate than publications in Nature.
PubPeer commenters and other post-publication peer reviewers are variously accused of lacking nuance, being discourteous, being undiplomatic, lacking humility (unlike Nature editors and authors of course) and gloating with a sense of “gotcha”. These are largely unpleasant traits, we would agree, but we challenge the editors to provide some concrete examples that are actually problematic. Even if critics really did have all of these character flaws, would that be a serious matter if they were also right? We believe science would progress more quickly in a world where researchers feel free to thrash out ideas in public than in the current system where garbage is politely allowed to pile up and fester unchallenged.
Nature’s fixation with nuance and courtesy rather than truth and precision is eerily reminiscent of the arguments of the editor of Plant Physiology, Michael Blatt. While Blatt and Nature are very concerned with protecting the feelings of authors (and maybe editors), we at PubPeer unequivocally aim to put first the readers and users of publications. As we stated in the linked blog: “Frankly, a few ruffled academic feathers pale into insignificance when patients’ lives, taxpayer billions and young researchers’ careers are at stake.” Maybe the editors would like to explain how nuance and courtesy help to protect their readers from unreliable science?
Nature also elect to follow Blatt in relativising scientific quality. They argue that “the scientific paper is a marker on the way to scientific progress, not itself a destination”, suggest that “insightful reasoning can lead to incorrect conclusions that still advance science” and claim that a typical paper is a “jumble of context” whose “Outcomes can depend on apparently trivial differences in methods”. This sounds awfully like an apology for compromising on rigor and robustness in favor of impact. We believe such relativism is anti-scientific and its expression confirms some of our fears about the custodians of scientific success. One imagines the editors saying, “Well, it would be really important if it were true,” in justification of yet another far-fetched story. We believe that the criteria for quality are straightforward: work should be honest, clear and contain no known flaws according to the state of the art at the time of publication, without compromise for impact or titillation of the tabloids. (Note that these criteria would not preclude publication of interesting ideas and partially proven hypotheses, but one should be honest about their limitations, something that is vanishingly rare today.) Put another way, we believe that extraordinary claims require extraordinary evidence, not extraordinary publicity.
In conclusion, the Nature editors have made quite revolutionary promises, but, for the reasons outlined above, we firmly believe that those promises won’t be kept in any meaningful way. The time is coming when the Nature editors will have to put up or shut up.
If you would like to comment on this post you can do so here: https://pubpeer.com/topics/1/AE11BE44CF3C40A558F3B453BF53C7