What if p = 0.06?
Unless an a-priori (beforehand) power analysis is performed, one cannot determine if a dataset is sensitive enough to detect a true effect when using p values. Therefore, when dealing with non-significant p values, the first question to be asked is, “what is the power of the study to detect a true difference?”.
If p = 0.06, the researcher should ask if the effect is worth exploring further. As previously mentioned, Fisher intended the use of p values as screening devices in a series of experiments to help rule out chance. When the p values were between 0.05 and 0.2, he always attempted to improve the study design so that there was a better possibility of ruling out chance/ finding the truth.
In quantitative research, consistent smallish probabilities from multiple studies in the same direction permit one to conclude the direction of an effect. Statistically significant results that are replicated provide the basis of scientific advance.
Tukey suggested that statistical testing should not be treated as an all or none procedure. Instead, we should use appropriate wording to describe our reluctance to bet on the direction of the true difference/ relationship. He suggested saying that an effect leans in a certain direction when the p is greater than 0.05 and less than 0.15, and that there is a hint about the true direction when the p is greater than 0.15 but less than 0.25.
Instead of suggesting that the null hypothesis was accepted by stating that “there was no difference among the treatments”, it is recommended that researchers simply state that more data are needed before determining the direction of difference(s)- “the direction of the differences among treatments was undetermined”. This approach suggests that the findings are not conclusive, and more data are needed before arriving at a conclusion.
Link to Part 1 of this series:
Link to Part 2 of this series:
Link to Part 3 of this series:
Link to a previous article on Null Hypothesis: