Quite a few people have expressed disapproval of the anonymity of PubPeer comments (and also of its organizers). This criticism is understandable, but we don’t think the problem is as simple to solve as some suggest. We examine the issues below.

Firstly, note that commenters on PubPeer DO have the option of displaying their real name (on a per publication basis). So those who want the fame and glory associated with criticizing their department chairman’s papers are of course free to get it in full. It seems that most people using PubPeer don’t choose that option; many moreover prefer the ultra-anonymous procedure of “unregistered submissions” that do not require account creation. Our own experience of commenting on papers is consistent with this impression: people are more inhibited by fear than by the fact that they must be anonymous (when in any case they don’t have to be). Many other factors come into play, including just finding the time to study papers with sufficient care to comment properly.

Authors whose papers have been criticized often vociferously complain about the anonymity. One suspects that they don’t always want to enter into a collegial discussion. Legal threats have already been received. I think we all know powerful scientists who can not be guaranteed to be reasonable about criticism of their work. Peer review is anonymous for a very good reason.

Many people suggest providing nicknames and possibly a reputation system. Of course we have considered both of these and may in the future provide them on a optional basis. But the big disadvantage we see is that in the long run they will destroy anonymity without the commenter wishing it, realizing it or being able to prevent it. This will happen because the nicknames would allow a commenter profile to be built up. In the end that profile or a single specific comment will allow somebody to identify a commenter. From that point on, all comments associated with the nickname can be traced back to the true commenter. A similar problem arises with the reputation system, although exploiting the information exposed is marginally more technical. It works by tracking comments that have correlated reputations in time. Whenever your reputation is updated, that update will occur synchronously on all of your comments. A similar commenter profile can thus be constructed (even in the absence of nicknames). The end result would be the same. Thus, we feel that nicknames and/or a reputation system would lull users into a false sense of security but ultimately and irreversibly compromise the anonymity that they might wish to maintain.

For the reasons given above, we have decided to allow anonymous comments on PubPeer, although the site may miss out on “social” driving forces as a result. The absence of any commenter identification also means that readers simply have to read and understand any comments for themselves; it is not possible to rely on any reputation. Although this is a little disorienting, it does have the advantage of forcing readers to exercise their own judgment.

PubPeer organizers remain anonymous for similar reasons. Imagine comments criticizing papers by their PI or department chairman. At the very least, pressure could be applied to censor the unwanted criticism. We just don’t want to deal with that situation. And, analogously to the situation for anonymous comments, the identity of the PubPeer organizers should have no bearing on the pertinence of any comments submitted.

6 thoughts on “Anonymity

  1. Pingback: Open Review of Scientific Literature | Adventures in Signal Processing and Open Science

  2. I had not read your thoughts on a reputation system until now.
    I must say that I was quite convinced that a “nickname” set-up would do the trick and enable a reputation system. Now, after reading your thoughts on it, I am not so sure. I guess you may be right about your worries that a sufficiently active user profile could be identified and in particular that a user’s comments on different papers could be associated through the reputation score.
    OK, if this does not work, then could scoring individual comments be useful? I guess it could risk creating more noise than actual value?

    • As you mentioned on Twitter, there is a social aspect to scoring that could be harnessed to help build the critical mass of commenters necessary to make post-publication peer review successful.

      Also, it is likely that participation in an online scientific community will be valued by universities/granting agencies/etc one day. This could encourage users to sign their comments and take credit for them. Ratings of these comments would then be very useful.

      So in summary, we feel that there is a value to rating comments. Keep doing it!

  3. I have to say, your maintaining anonymity cuts both ways as well. What guarantee is there that you will not censor submissions?

    • One good thing with internet communities is that when someone knows, others easily knows.
      Specifically here, the PubPeer’s users will quickly know if PubPeer’s organizers are trying to protect a particular journal.

  4. If the time correlation between a comment and reputation profile is an issue, why not randomly delay the contributions to the reputation scores using a large window ? (e.g. 3 months to 1 year). Google Scholar and ResearchGate often take time to update, but it’s not a big issue since nobody needs to follow the weekly updates of a h-index or i10-index.

Leave a Reply

Your email address will not be published. Required fields are marked *