I often get asked to review papers on new software tools. This always makes me wonder how one should review such a paper, or even whether software papers should undergo a typical review process at all?
My private checklist typically consists of:
- Is it available for download for others?
- Does it run?
- Did the authors provide benchmark problems? Do they give expected results?
- Is documentation sufficient?
- Go through numerical methods used in the software (briefly, no meticulous code analysis)
If everything runs fine and produces expected results, I honestly have nothing else to “demand” from the authors. My referee reports are positive and (embarrassingly) brief in those cases. I’m of the opinion that the user community should “do the rest”; evaluate the software and decide whether it’s useful in their workflows.
No comments:
Post a Comment