Null Hypothesis Significance Testing: What you should know (Part 4)

Errors in determining the truth

It has been mathematically demonstrated that the point estimate of a statistically significant result is more likely to overestimate the truth when the power is less than 100%.

There is a failure to account for systematic errors as well- the p value of a study is the probability that a test statistic calculated from the data would be greater than or equal to its observed value, assuming that

  1. the test hypothesis is true and
  2. there is no uncontrolled source of bias in the data collection or analytical processes.

The latter assumption is equivalent to assuming the absence of uncontrolled confounding, selection bias, measurement error, etc. When such errors are present, the p value is a measure of the compatibility between a study’s observed data and what would be predicted if all the assumptions used to compute the p value were correct.

A low p value does not inform us which of the two assumptions mentioned above is incorrect. Rather than checking if the second assumption regarding bias is incorrect, one takes decisions regarding the first assumption.


1 thought on “Null Hypothesis Significance Testing: What you should know (Part 4)

  1. Pingback: Confidence Intervals: The basics | communitymedicine4all

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.