Null Hypothesis Significance Testing: What you should know (Part 4)

What if p = 0.06?

Unless an a-priori (beforehand) power analysis is performed, one cannot determine if a dataset is sensitive enough to detect a true effect when using p values. Therefore, when dealing with non-significant p values, the first question to be asked is, “what is the power of the study to detect a true difference?”.

If p = 0.06, the researcher should ask if the effect is worth exploring further. As previously mentioned, Fisher intended the use of p values as screening devices in a series of experiments to help rule out chance. When the p values were between 0.05 and 0.2, he always attempted to improve the study design so that there was a better possibility of ruling out chance/ finding the truth.

In quantitative research, consistent smallish probabilities from multiple studies in the same direction permit one to conclude the direction of an effect. Statistically significant results that are replicated provide the basis of scientific advance.

Tukey suggested that statistical testing should not be treated as an all or none procedure. Instead, we should use appropriate wording to describe our reluctance to bet on the direction of the true difference/ relationship. He suggested saying that an effect leans in a certain direction when the p is greater than 0.05 and less than 0.15, and that there is a hint about the true direction when the p is greater than 0.15 but less than 0.25.

Instead of suggesting that the null hypothesis was accepted by stating that “there was no difference among the treatments”, it is recommended that researchers simply state that more data are needed before determining the direction of difference(s)- “the direction of the differences among treatments was undetermined”. This approach suggests that the findings are not conclusive, and more data are needed before arriving at a conclusion.

Useful Links:

Link to Part 1 of this series:

https://communitymedicine4all.com/2020/09/09/null-hypothesis-significance-testing-what-you-should-know-part-1/

Link to Part 2 of this series:

https://communitymedicine4all.com/2020/09/16/null-hypothesis-significance-testing-what-you-should-know-part-2/

Link to Part 3 of this series:

https://communitymedicine4all.com/2020/09/23/null-hypothesis-significance-testing-what-you-should-know-part-3/

Link to a previous article on Null Hypothesis:

https://communitymedicine4all.com/2014/04/05/the-null-hypothesis-why-it-can-never-be-accepted/

1 thought on “Null Hypothesis Significance Testing: What you should know (Part 4)

  1. Pingback: Confidence Intervals: The basics | communitymedicine4all

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.