Science self-corrects – instantly

Publishing a paper is still considered a definitive event. And what could be more definitive than publishing two Nature papers back to back on the same subject? Clearly a great step forward must have occurred. Just such a seismic event happened on the 29th of January, when Haruko Obokata and colleagues described a revolutionarily simple technique for producing pluripotent cells. A short dunk in the acid bath or brief exposure to any one of a number of stressors sufficed to produce STAP (Stimulus-Triggered Acqusition of Pluripotency) cells, offering enormous simplification in stem cell research and opening new therapeutic avenues.

Nature were presumably conscious of a potential problem when processing the paper. The field of stem cell research had seen the huge Woo-Suk Hwang scandal. It was also shaken by the hasty three-day review by Cell of a paper from the Mitalipov group that contained several very careless errors (and whatever happened to those promised external verifications of its central claims?). Obviously, it wouldn’t do to get caught out like Science and Cell.

So Nature and its referees must have known that these papers would have a bullseye painted on their backs. And, knowing this, one suspects that they would have been extra-careful. Their most trusted referees would take their time to ensure that the authors had dotted all the ‘i’s and crossed all the ‘t’s. The papers indeed underwent a respectable 9 months gestation from submission to acceptance; the authors even complained about the strictness of the referees. So how would Nature’s high(est) quality output get on in the big bad world? One would maybe have expected a few quibbles about finer points of interpretation, but surely the critics of pre-publication review would find little ammunition in such carefully prepared papers.

The papers came out to blaring publicity. The mainstream media were all over the story. Nature had another breakthrough; Riken and Harvard were revelling in their glory. But the internet allows quite accelerated feedback, from anybody and everybody. Inveterate stem-cell blogger Paul Knoepfler was immediately on the case, sharing his field’s bewilderment, a mixture of skepticism and hope. Our site received some chatter, mostly about the predicted consequences of the stated T-cell origin of the stem cells.

Then, on February 4th, less than a week after publication, an anonymous comment on PubPeer pointed out that a gel showed signs of having a lane spliced in ( Although some people don’t consider this practice very serious, most ‘Peers’ take quite a dim view of it. In any case, an unannounced splice was potentially deceptive and probably caused people to examine the papers with a more critical eye. Over the following weeks, quite a number of comments highlighting small (inconsistent scale bars) and potentially serious (possible figure duplications) problems were posted. Numerous comments linked back to Japanese blogs, whose interest was understandable. At the very least, the papers were shown to contain a disconcerting number of unfortunate errors.

In parallel, Knoepfler was giving the stem cell field weather report (turning gloomy), interviewed coauthors Charles Vacanti and Teruhiko Wakayama, and also ran an innovative crowd-sourced replication page, which soon showed that promises of simplicity and hopes of generality would not easily transfer from the abstract to the lab. We learnt that both Riken and Nature were ‘investigating’. Questions were also raised about papers by Vacanti and Wakayama.

Suspense turned to outright febrility following new comments posted on PubPeer this weekend, which may turn out to be the straws that broke the donkey’s back. They claim that figure panels in one of the Nature papers appear to have been duplicated from different experiments in Obokata’s PhD thesis (unfortunately not online). Knoepfler then ran a hugely entertaining piece speculating that Riken might choose to precipitate a retraction in order to avoid being blamed by Harvard, presumably according to the theory that senior authors are blameless victims, especially if they work at Harvard. We also heard that coauthor Wakayama now says he has lost faith in the work, despite being recently quoted as having replicated Obokata’s technique for himself.

One suspects that the denouement is not far off. But some conclusions can already be drawn. We see that post-publication peer review easily outperformed even the most careful reviewing in the best journal. The papers’ comment threads on PubPeer have attracted some 40000 viewers. It’s hardly suprising they caught issues that three overworked referees and a couple of editors did not.

Science is now able to self-correct instantly. Post-publication peer review is here to stay.

6 thoughts on “Science self-corrects – instantly

  1. Post-pub peer review is central to new, democratic, and open science. We all have seen our share of deeply flawed papers – both empirical and theoretical – including those in luxury journals. The papers continue to be sited, their authors getting new grants, the viscous cycle of irreproducibility being self-reinforced. Post-pub peer review can break the cycle and bring much needed accountability to science. An ability to easily cite PubPeer comments will be instrumental in increasing the power of post-pub peer-review (would not it be great if each comment has its own easily citable DOI number?).

  2. I’m not a scientist either by trade or training, but to be pedantic, isn’t there a difference instantly self-correcting and instantly peer reviewed?

    After all, we still don’t know until the experiment is reproduced whether or not STAP cells even exist. Long-term studies would be a lot more difficult to correct.

    That said, I’m certainly not complaining about the fact that scientists can peer review instantly now. It’s a silver lining in yet another story of massive deception coming from my country this year (the other being Samuragochi’s ghost-written music).

  3. Let’s be a bit more precise: post-publication review *of very high-profile papers* is here to stay. Post-publication review of the vast majority of papers doesn’t even exist and shows no sign of coming into being. Please correct me if I’m wrong, but I’m betting that the vast majority of papers attract no comments on PubPeer, for instance. Just as the vast majority of papers attract no comments on PubMed Central, or on existing journal-based comment systems, or on blogs or Twitter, or anywhere else.

    Which maybe is fine. Indeed, I suspect one could make a good argument for a “hybrid” system not to dissimilar from the current one, in which all papers get pre-publication review from 2-3 referees, and then a very small fraction of high-profile papers are heavily scrutinized by lots of people post-publication. But it’s important to recognize that, in this system, post-publication review is a supplement to pre-publication review–perhaps a very important and useful supplement, but not a replacement. And that’s the way it will remain, at least for a long time, I think, since I don’t know of any plausible schemes to ensure that every paper gets serious post-publication review. (Remember that the vast majority of readers of most papers are just reading the abstract or skimming, so even if every paper gets, say, hundreds of readers, that doesn’t necessarily mean every paper gets “reviewed” in any meaningful sense).

    • @disqus_uLKOPqfMfi:disqus
      Sure, the vast majority of papers have not been commented on PubPeer. Almost as many have probably never been seriously read, even by their assigned referees (see recent scandals involving computer-generated papers). But we have nevertheless learnt that post-publication peer review (PPPR) can add value, even to the highest profile cases. Platforms like PubPeer also allow any comments that are made to be accumulated and made available to all, rather than to be discussed between a few friends and then lost. By no means all of the papers commented on PubPeer are high-profile. Many comments seem to arise simply because somebody read the paper and noticed something. Feel free to contibute your own expertise and knowledge to ensure that it is not wasted.

      As I read it, the blog post isn’t necessarily suggesting that PPPR replace standard reviewing. You seem to consider such a situation as utopian. But did you know that most of physics works exactly like that? Preprints are posted to the ArXiv before being submitted to journals and reviewed on a much slower time scale under no pressure at all. Everybody works with the ArXiv, which doesn’t even have a commenting system. People just read what the find interesting and work with what they find useful.

  4. Pingback: Opens Roundup (March 25) | PLOS OpensPLOS Opens

Leave a Reply

Your email address will not be published. Required fields are marked *