Monday 23 July 2018

reproducible research - How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?


I am graduate student, to finish my degree I need to build methods outperform what is already there. An issue that I came across with, is that two papers reported way (I mean more than 20%) more than what resulted from my reimplementation. This could be due to two reasons:




  1. I missed something during the implementation. Which is what I have been telling myself. For months, I tried all possible combinations and possible paths. One of the methods is straightforward. Still, I could not reach their claimed performance.


    I contacted the corresponding authors, and no one replied. So I tried to contact the other authors.


    The first paper, the author replied and sent me the code. He/she told me to keep all details ”confidential”. Well, it turns out they they are not using the data they claim in their the paper, of course their results are different than my reimplementation. And my implementation was correct.



    The second paper author also replied and they didn’t send me the code because they say it is easy to implement, but confirmed that what I did is correct still I couldn’t understand why such difference.


    Both papers are published in <2 impact factor journals. Their web servers are not working.




  2. They are not honest.




Now I am stuck, my method does outperform my reimplementation of their methods but not what they claim. The first paper I can’t say anything because “it is confidential” the second paper I can only confirm that I correctly implemented their method for the most part (based on my chat with the authors)


I know that I probably could not publish on this part of my work, because who is going to believe a young scientist who just started her way? But not sure how the committee are going to believe me. What can I say or do? Please help me




No comments:

Post a Comment

evolution - Are there any multicellular forms of life which exist without consuming other forms of life in some manner?

The title is the question. If additional specificity is needed I will add clarification here. Are there any multicellular forms of life whic...