PubPeer response to ACSNano editors

The editorial board of the journal ACSNano recently angered chemistry bloggers and many of their readers with an editorial implying that bloggers often make irresponsible accusations of fraud. The editorial went on to suggest that bloggers should refrain from such unfair and damaging behavior, and instead allow journal editors to process misconduct investigations with their usual diligence and deliberation. It has been suggested that the editorial was in fact an indirect response to the intense commenting on PubPeer about papers by one of the board members.

In our view the editorial rather clumsily conflated two separate processes. The first, commenting on published data, including highlighting any inconsistencies, is absolutely legitimate. The second, accusing people of misconduct, should indeed not be done lightly. Lumping the two processes together enabled the editors to imply that bloggers were making accusations of fraud, when in fact they were only commenting on published data.

PubPeer, a possible target of the editorial, illustrates the fault in this logic. Thus, although we encourage anonymous comments on papers, we forbid any accusations of fraud, misconduct etc. We have made this explicit on a page explaining that comments should be based on easily verifiable information. This policy has been enforced since the inception of the site. While it is true that some of the anomalies highlighted by commenters would require the most fabulous coincidences to have arisen innocently, accusations of misconduct are still not allowed. They are also unnecessary. Any scientist worth his/her salt can interpret a screaming inconsistency.

The editorial also suggested that bloggers unfairly deny authors the possibility to defend their work. This is laughable to those following PubPeer. Commenters have repeatedly sought responses from authors, who are automatically notified of comments. Authors can always respond, either anonymously or signing their comments. However, they rarely exercise their right to reply to comments unless to clarify a simple misunderstanding.

We therefore believe the editorial cited no examples of bad behavior by “bloggers” for a very good reason: they are few and far between. Instead of loose accusations of fraud and authors being denied the chance to reply, they could only have pointed to long, restrained threads of careful scientific discussion, to which the authors have refused to respond. We would be very interested to hear the opinion of the board members about these threads and their content.

PubMed Commons

We think that although it is off to a rocky start by being overly exclusive, PubMed Commons is a great initiative and a big step in the right direction towards effective post-publication peer review. We shall of course be following the experiment with interest.

The obvious difference with PubPeer is the lack of any anonymity. PubPeer offers three levels of anonymity: registered academics can post signed or anonymous comments, and we also host moderated comments from unregistered contributors, who could be anyone, anywhere.

Our own experience suggests that strong anonymity is the key to encouraging useful comments, as do the failed experiments with journal-run commenting systems. We suspect that PubMed will eventually come to the same conclusion. Thus, a majority of comments on PubPeer are from unregistered contributors, whereas only a tiny minority are signed; registered academics commenting anonymously make up the balance.

Examination of the typical contents of PubPeer comments can easily explain why users choose anonymity. It turns out that the strongest motivation to comment arises when people see a problem with a paper, often a serious one indicative of incompetence, deceit or misconduct. But such critical comments are those most likely to attract reprisals. Most of the comments we receive would not have been made in the absence of the anonymity we provide.

Anonymity does allow low quality and bad faith comments to be made with impunity, but we have found this concerns only a small minority of comments and we feel that it is a necessary price to pay to encourage frank and worthwhile discussion.

Finally, if PubMed does start offering strong anonymity, that is likely to disrupt our plans (and save us money!) but for the moment we are continuing to develop PubPeer to facilitate all formats of scientific discussion.

*Note that PubPeer is open to all sciences, not just biomedical sciences.

Anonymous cowards vs the scientific establishment

Battle lines are being drawn on the internet, between the scientific establishment and volunteer vigilantes trying to impose their own vision of the scientific process through “post-publication peer review”.

On one side is the cream of the scientific aristocracy: a professor with a meteoric career trajectory at Imperial College London, one of the best universities in the world, and the top academic publisher, Nature Publishing Group. On the other side: a few anonymous malcontents carping on an obscure web site called PubPeer (welcome to our site!).

A couple of years ago, the professor’s group published in Nature journals a short series of papers reporting an ultra-sensitive assay, in principle for anything that could be recognized by an antibody. The assay, called “Plasmonic ELISA”, was hailed as a breakthrough in diagnostic medicine and is typical of the kind of sensational “high impact” work encouraged by Nature journals. The story was widely reported in the mainstream press.

However, a few scientists felt that the results were a little too good to be true, or at least that unexpected observations were not well explained and that supporting evidence was missing. One of the critics wrote to the journal in private, detailing those concerns. The journal considered them but, with its referees, decided that there was no substance to the complaints. So the publications were not only initially accepted for publication by the journal and its editors, after a typically rigorous review, but their quality was reconfirmed by a second round of review that specifically addressed the criticisms. A stronger proof of quality is hard to come by: Nature says “yes”, twice.

Scientists can be stubborn, especially when they think they are right and have been told they are wrong. The unhappy critic therefore took advantage of our site, which enables anonymous comments on scientific articles, to air his concerns. Site visitors found them convincing and chimed in with their own remarks and analyses. The flow has been essentially one way, with nearly all commenters agreeing that the publications appear to contain serious problems. The difference of opinion with the professor, Nature Nanotechnology and their referees could not be stronger. Only one side can be right. To date, no substantive rebuttal of the criticisms has been posted, though of course the authors and the referees of the papers would be free to defend their work and judgement, anonymously or otherwise. They haven’t responded because dealing with the ignorant internet riff-raff is beneath them or because they have no answer to the criticisms?

Who do you believe – a prestigious professor publishing in a high-impact Nature journal or the anonymous cowards? You can make up your own mind and join in the discussion here (please be polite and factual):

Example case showing why letters to the editor can be a waste of time.

A lively discussion has developed around some recent high-profile publications reporting some amazing results that were covered in many major media outlets. The comments on one of these papers exemplify perfectly why we created PubPeer and why we feel that post-publication peer review needs to have a formal home. We would like to take a minute to point out these comments and underline why we feel that they are so important.

The traditional routes for scientists to raise questions about a publication are to publish their own paper, write to the authors, or write to the journal. All of these avenues have their disadvantages. A new publication requires a major time (and financial) investment, while an impasse is often reached in interactions with the authors. Both of these approaches identify and expose the critic to reprisals. The final option, writing to the journal, does mostly preserve anonymity, but at the cost of a loss of transparency. Below is a perfect example of this from one of the articles getting a lot of attention on PubPeer at the moment:

In that thread, Peer 2 recounts writing to Nature Nanotechnology, who, to their credit, did take the time to follow up the issue via correspondence with the authors. Ultimately however, although Peer 2 was not satisfied, the editor was, and closed the case. The authors, who have confirmed to us that they are aware of the comments, seem to feel that they have already addressed the issues with the editor of Nature Nanotechnology and that the scientific community should be content with that. However, at the end of the correspondence, the only people aware of the potential issues were Peer 2, the editor of the journal and the authors.

Peer 2 eventually posted the issues on PubPeer, where others interested in the field saw them and added to them with some very thorough reviews of the paper. We are not experts in that field, and we can’t predict what will eventually happen to the paper, but, through reading PubPeer, many others in the field now know about and can evaluate for themselves the issues raised. We consider this to be the main benefit of post-publication review: the dissemination of follow-up discussion so that it is available to all interested parties. Post-publication review can take many avenues (blogs, tweets, etc.) and we encourage the use of all of them. But we also feel that it is essential that others in the field are able to find these reviews easily. That is why a centralized database is essential. If you prefer one of the other methods of review, please also consider cross-posting your remarks to PubPeer, so that your colleagues are more likely to discover your comments.