Arbitrary alpha levels

One of the criticisms of NHST is the use of arbitrary alpha levels (level of significance). The choice of an alpha of 5% (which corresponds to a p value of 0.05 since 5% equates with 0.05 in decimals) is arbitrary.

Fisher suggested that researchers not adhere to a fixed value of alpha, but to determine alpha separately for each null hypothesis. The level of alpha chosen should correspond to the researcher’s threshold to say that chance has been eliminated. However, using different values for different sets of data is likely to cause confusion.

Perhaps using an alpha level of 0.05 as a screening device in small-scale studies that are part of an ongoing series of studies is acceptable. However, when one-time studies with high stakes are conducted, then much smaller alpha levels are recommended (although it is very unusual to take high-stakes decisions based on a single study).