Errors in determining the truth

It has been mathematically demonstrated that the point estimate of a statistically significant result is more likely to overestimate the truth when the power is less than 100%.

There is a failure to account for systematic errors as well- the p value of a study is the probability that a test statistic calculated from the data would be greater than or equal to its observed value, assuming that

1. the test hypothesis is true and
2. there is no uncontrolled source of bias in the data collection or analytical processes.

The latter assumption is equivalent to assuming the absence of uncontrolled confounding, selection bias, measurement error, etc. When such errors are present, the p value is a measure of the compatibility between a study’s observed data and what would be predicted if all the assumptions used to compute the p value were correct.

A low p value does not inform us which of the two assumptions mentioned above is incorrect. Rather than checking if the second assumption regarding bias is incorrect, one takes decisions regarding the first assumption.