publications - Towards a Stackexchange-like comment/reputation system for research papers

We have had quite a few discussions on how to gauge the quality of research papers and academic journals. The consensus is that it is tough to identify a single metric which could reveal this quality factor.

I had an idea in this context. In the past newspapers used to have a single "Letters to Editor" section to which comments on all articles were sent. With the advent of the Net, we have a comments section under every article.

• Similarly, why don't reputable journals introduce a comment space for their published papers, where fellow researchers could appreciate/criticise/query the works?


• What hinders us from having a reputation system for published papers - something where registered users could vote based on their perceived utility?

Answer

Similarly, why don't reputable journals introduce a comment space for their published papers, where fellow researchers could appreciate/criticise/query the works?

They don't need to create such a space, since these comments can already appear elsewhere on the internet. Nevertheless, some journals have tried, but they typically attract very few comments, and nobody knows how to create a viable community this way. See that link (and the blog posts it links to) for further discussion.

What hinders us from having a reputation system for published papers - something where registered users could vote based on their perceived utility?

One major obstacle is that not enough people seem to want to use such a system. There have been many attempts to set one up, but none have caught on.

However, there's a deeper obstacle. In order to get any meaningful results at all, you need to solve at least three problems: ensuring that votes come only from competent researchers, ensuring that nobody can deliberately manipulate the system or cheat, and ensuring that voters have the right incentives even if they are not trying to be dishonest. In principle, it might be possible to solve these problems, but it would take a nontrivial infrastructure. However, the more elaborate the system becomes, the harder it will be to get lots of users. A large majority of researchers won't pay any attention to a system like this, and some of the ones who do care will be strongly opposed, so getting widespread adoption will be an uphill battle.