Wednesday 19 December 2018

ethics - Are P-hacking and hypothesising after results are known considered misconduct in academia?


This question is inspired by a comment to another question where I asked for help on how to argue against P-hacking and hypothesising after results are known (Harking). Someone questioned the classification of these two behaviours as misconduct, and my general experience (around my close academic circle) is that many see these two activities as part of the way we do science.


Here is what I refer to by P-hacking and Harking.


P-hacking is when someone collects more data, changes the specification of a statistical model, change the analysis sample, or does other changes to the study until the results become statistically significant. Many of these things can be done with a justification, but the p-hacker (p-fisher) does them solely with the intent of obtaining a significant result. In doing so, he or she risks capitalising on statistical error (type 1 error) and publishing results that are basically a false positive.


Harking is when someone generates a scientific hypothesis about the data after seeing the data. It would be innocuous if the researcher acknowledged the exploratory nature of the study and sought to confirm the findings in another set of data (or if he or she used cross validation techniques). It becomes a problem when researchers pretend that they had the hypothesis a priori and that the study was done to confirm it, hiding the exploratory nature of the study and conferring more strength to the results than they actually have.


I am not asking for opinions on whether these things should or should not be considered misconduct. Rather, I would like to know the overall position of scientists in fields where statistics are used. I know of no survey on how scientists view these two behaviours, but I welcome answers that include such data.



Answer



The Declaration of Helsinki was updated in 2013 to "mandate" that research involving human subjects must be pre-registered. While not perfect, pre-registration prevents many of these statistical manipulations. The idea of pre-registration is that publicly stating your hypotheses and the details of how they will be tested in advance reduces questionable statistical practices. For example, changing the number of subjects, the inclusion/exclusion criteria, or the statistical model are not allowed.



From my understanding, failure to comply with the Declaration of Helsinki would be considered unethical in Medicine, while in other fields the pre-registration aspect is being actively ignored. For example, articles are now being published with disclaimers like "this research was conducted in accordance with the 2013 Declaration of Helsinki except the study was not pre-registered".


No comments:

Post a Comment

evolution - Are there any multicellular forms of life which exist without consuming other forms of life in some manner?

The title is the question. If additional specificity is needed I will add clarification here. Are there any multicellular forms of life whic...