Anonymity

Quite a few people have expressed disapproval of the anonymity of PubPeer comments (and also of its organizers). This criticism is understandable, but we don’t think the problem is as simple to solve as some suggest. We examine the issues below.

Firstly, note that commenters on PubPeer DO have the option of displaying their real name (on a per publication basis). So those who want the fame and glory associated with criticizing their department chairman’s papers are of course free to get it in full. It seems that most people using PubPeer don’t choose that option; many moreover prefer the ultra-anonymous procedure of “unregistered submissions” that do not require account creation. Our own experience of commenting on papers is consistent with this impression: people are more inhibited by fear than by the fact that they must be anonymous (when in any case they don’t have to be). Many other factors come into play, including just finding the time to study papers with sufficient care to comment properly.

Authors whose papers have been criticized often vociferously complain about the anonymity. One suspects that they don’t always want to enter into a collegial discussion. Legal threats have already been received. I think we all know powerful scientists who can not be guaranteed to be reasonable about criticism of their work. Peer review is anonymous for a very good reason.

Many people suggest providing nicknames and possibly a reputation system. Of course we have considered both of these and may in the future provide them on a optional basis. But the big disadvantage we see is that in the long run they will destroy anonymity without the commenter wishing it, realizing it or being able to prevent it. This will happen because the nicknames would allow a commenter profile to be built up. In the end that profile or a single specific comment will allow somebody to identify a commenter. From that point on, all comments associated with the nickname can be traced back to the true commenter. A similar problem arises with the reputation system, although exploiting the information exposed is marginally more technical. It works by tracking comments that have correlated reputations in time. Whenever your reputation is updated, that update will occur synchronously on all of your comments. A similar commenter profile can thus be constructed (even in the absence of nicknames). The end result would be the same. Thus, we feel that nicknames and/or a reputation system would lull users into a false sense of security but ultimately and irreversibly compromise the anonymity that they might wish to maintain.

For the reasons given above, we have decided to allow anonymous comments on PubPeer, although the site may miss out on “social” driving forces as a result. The absence of any commenter identification also means that readers simply have to read and understand any comments for themselves; it is not possible to rely on any reputation. Although this is a little disorienting, it does have the advantage of forcing readers to exercise their own judgment.

PubPeer organizers remain anonymous for similar reasons. Imagine comments criticizing papers by their PI or department chairman. At the very least, pressure could be applied to censor the unwanted criticism. We just don’t want to deal with that situation. And, analogously to the situation for anonymous comments, the identity of the PubPeer organizers should have no bearing on the pertinence of any comments submitted.

Introducing PubPeer

The process of reviewing published science is constantly occurring and is now commonly being called post-publication peer review. It occurs in many places including on blogs such as this one, review articles, at conferences around the world, and has even been encouraged on the websites of some journals. However, the process of recording and searching these comments is, unfortunately, inefficient and underused by the larger scientific community for several reasons: To successfully impact the publication process, this database of knowledge has to accomplish two important tasks. First it requires participation by a large part of a given scientific community so that it reflects an average impression instead of an outlier’s impression. Second, it requires that the collective knowledge is centralized and easy to search in order find out what the community collectively thinks about an individual paper or a body of work. A recent initiative, the San Francisco Declaration on Research Assessment (DORA), echoes many of these same concerns.

In an attempt to assemble such a database, a team of scientists, have put together a website called PubPeer.com that is searchable and encourages participation by the larger scientific community. With a critical mass of usage an organized system of post-publication review could improve both the process of scientific publication as well as the research that underlies those publications.

Those of us involved in the creation of PubPeer.com believe that in an ideal world, a scientist’s main goal would be to discover something interesting about the world and simply report it to other scientists to use and build upon. This idealistic view of the scientific process is however not matched in reality because, for academic scientists, our publications count for much more than a simple contribution to the scientific record. For example, the majority of candidates are eliminated from consideration for tenure track positions at a major universities based on the names of the journals that have published their recent findings.

Review committees use this method because publications are the best measure of past and potential scientific output, but by potentially overvaluing “high impact” journal names, these committees and study sections effectively defer to journal editors to help them choose the best candidates for jobs and grants. However, these journals select their articles based on more than just good science – the papers also need to be of ‘wider interest’ and this can sometimes skew the publications towards ‘exciting’ results over those that are more measured, and perhaps more likely to be correct (for instance). The sometimes disproportionate attention given to a high profile paper also makes it a tempting target for more unscrupulous scientists.

It’s never going to be possible for us to thoroughly read all of the papers submitted to a job advertisement, nor all of the papers referenced in grant applications, but we can easily reduce the importance that journal names play in decisions and replace it with something that is more meaningful and directly in our hands instead of the hands of publishers. After reading any publication, we all have impressions about whether the reported observations are useful, interesting, elegant, irrelevant, flawed, etc. If a particular scientific field that is interested in a given publication were able to compile all of it’s impressions of that publication, that collective information would be infinitely more useful to search committees and study sections than the name of the journal in which it was published.

Outlined below are a few aspects of PubPeer.com that differentiate it from the current post-publication review systems and which will hopefully make it more widely used.

  • A key issue that we have decided on is the importance of anonymity. One of the reasons that we have never commented on articles directly on journal websites is because the colleagues whose publications we are most qualified to comment on are likely reviewing our publications and grant proposals. Even the most well-intentioned criticism could potentially irk these potential reviewers. Since publications are so precious to everyone’s future career advancement, there is a huge psychological barrier for early stage scientists to attach names to any comments that could be considered critical.Therefore, in order to encourage as much participation in this post-publication review process as possible, PubPeer allows comments to be left anonymously if someone is so inclined. Critics of this feature sometimes email us to point out that anonymity allows for baseless slander or to proclaim that a commenter’s name is essential for judging the validity of a comment. We strongly disagree with this second point because good comments are good regardless of whether they come from a senior scientist or a graduate student. We can all judge for ourselves the content of comments and on PubPeer it is possible to vote the good comments up and the bad comments down into the noise so that community as a whole can decide together what is worth paying attention to. Baseless defamation, rumors, and ad hominen attacks are not tolerated at all and are immediately removed from the site.The people involved with PubPeer are all active scientists and we are trying to remain anonymous for the time being for several reasons: 1) we can imagine scenarios in which pressure could be put on us to remove/alter comments if our identities were known and 2) we would like to protect our families and private bank accounts from the more litigious among our readers.
  • A main drawback of the current practice of post-publication peer review is that the reviews can be spread across many different blogs and journal websites. If one wants to know what the community thinks of a given body of work (whether it be a discipline, an author’s output, a university department, etc.) it takes a major time investment to track down the information from all of these different sources. PubPeer provides a centralized and easily searchable database that contains comments on all published articles.
  • PubPeer also provides for a system of alerts. In order to be effective, authors and others interested should be able to be alerted to comments on their favorite publications or topics. PubPeer automatically notifies corresponding authors of new comments on their articles and anyone can set up email alerts on articles they find interesting.

This was reposted from realclimate.org