Yesterday, during some idle time in the office, a colleague of mine has discovered that I am currently ranked at position 20 worldwide by Microsoft Academic Search for my subfield regarding the last 5 year impact. While certainly flattering, this ranking is also quite clearly bogus, as many much more eminent academics are ranked far behind me in this list.
The Occam's razor explanation that there simply is another researcher of my name in my field can be discarded, as Microsoft Academic profile does indeed list my, and only my, publications (as well as a profile pic that they seem to have grabbed from an old university web page).
So my question is the following: how does Microsoft Academic Search actually generate rankings (of individual academics, but there are also rankings of journals)? It seems clear that not all citations and publications are considered equal (otherwise I would not end up so far to the front with a comparatively meager set of papers and citations), but how do they decide what is "worth" how much?
I am mostly looking for answers that refer to papers or web links coming from MS insiders on their ranking algorithms. Barring that, some well thought-through speculation from outsiders is also ok :)
Edit:
I just discovered that Microsoft Academic Search seems to employ the notion of a field rating for conferences and journals, and the rating for some of the venues that we prefer seems to be unreasonably high. However, that still does not fully explain my curious case (and it still leaves open the question of how these field ratings are generated in the first place).
Answer
how does Microsoft Academic Search actually generate rankings?
Since Microsoft Academic Search, like Google Scholar, is a proprietary vertical search engine and its algorithms and ranking systems are not open source, you're unlikely to get a detailed "how" answer unless a disgruntled former Microsoft employee decides to chime in.
Microsoft Academic Search does make some details of its rankings and results system available, however. This page offers a basic explanation of the search engine's approach. For both Microsoft and Google, citations play a major part in search result rankings, which has subjected academic search engines to criticism in the past. As the Academic Search page above indicates,
the information associated with Microsoft Academic Search author profiles is derived from the tens of millions of scholarly publications that are currently indexed by Microsoft Academic Search. Most of these publications have reference lists that Microsoft processes. The indexed publications and reference lists help create a snapshot of individual authors' publication history, productivity, and impact. As more content is indexed within Microsoft Academic Search, the accuracy and completeness of the author profile data improves.
Because Academic Search rankings can be distorted by factors including citation counts, incomplete publication indexes, and the field rating system you mentioned, it will inevitably produce some inaccurate results.
Speculation alert: it seems to me that the lion's share of ranking is accomplished through automatic citation list analysis, which is not necessarily a good indicator of
- Prominence in one's field,
- Number of publications (instead, it seems to emphasize how often a publication is cited elsewhere).
Finally, Academic Search is still in Beta, and I suspect that their rapidly growing index may not always be in perfect step with their ranking algorithms - in my personal experience with the search engine, there do seem to be cases of highly-ranked scholars or papers suddenly plummeting, then re-appearing again, which I assume is a consequence of ranking algorithm tinkering.
No comments:
Post a Comment