productivity - Scientifically meaningful sources of bibliometrics

Whether we like it or not, modern academia is increasingly being measured, in some vain attempt to get objective measures. Although it is unwise to fight 'being measured', it is at least possible to steer the measures away from meaningless ones, backed by peer-reviewed research that establishes this unrefutably.

There are a lot of different metrics that have been defined - I am not looking for those. What I want is pointers to the research behind the scientific validity of those metrics.

So the question is: where should I look for scientific assessments of bibliometrics?

Answer

Johan Bollen and Herbert van de Sompels are two researchers to follow in this area. Bollen did an analysis of 39 different citation-based metrics which is a good place to start. However, it's crucial to note that there are serious errors in trying to use citation-counting methods as some sort of ground truth. Citation counting is problematic because:

• Different fields have different citation practices. In biology it's common to have 10 or more authors on one paper, whereas in math you often have only one or two.


• Citations take a long time to accumulate, penalizing early-stage researchers.


• Citations only tell part of the story, leaving out the useful contributions made by researchers in the form of code written and datasets released.


• Citations often mutate over time.

It's now possible to get more information about a paper than just who cited it, and it's possible to get this information before several years have passed and before the information about the impact of the paper becomes old and less useful. The Public Library of Science makes detailed article-level metrics available and Mendeley has an API from which you can collect real-time data about how many readers a paper has, as well as social metadata such as tags and annotations and reader demographics. These metrics are being consumed by services such as Total Impact and combined with data from Github, Twitter, and traditional citation metrics. My bet is that if you're looking for a meaningful set of measures, you're going to find it in these richer sets of aggregated data.