In computer science, we usually implement our work into algorithms and codes. So in order to compare our work to others, we also need to implement their approaches as well.
But the problem is that usually there are recently published works in each area which have used complicated mathematical methods and claimed to beat other previous methods. But some of them are difficult to implement by just reading the papers; or they have not mentioned all the necessary details in the paper for a precise implementation of their work and neither have they make their code available.
How should I compare my work to these most recent algorithms for which the code is not available?
Answer
Asking the authors for the code is of course always an option, but from my personal experience this almost never works (there is a reason why they did not put it online in first place).
I don't know which area in computer science you are from but normally there is some quantitative measure to determine which method is better and those measures are somehow in the paper. Those can be things like run time or precision-recall curves on certain data sets.
If you want to compare on the same dataset, you can ask the authors for the numbers / plots themselves and then plot your own curves together with them.
If you want to compare on a different data set, you can ask the authors to run the code for you. In my experience, this works more often because many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it.
If you have Bachelor/Master students to supervise, you can give the implementation as a project to them (although you need to give them enough time to do it).
In my personal opinion, it is not worth the effort to implement a method which is not described in enough detail anywhere. It will probably cost you an unreasonable amount of time without even reproducing the quality shown in the paper (for various reasons).
No comments:
Post a Comment