# Types of errors in hypothesis tests The end result in a hypothesis test is a decision: 1. Reject the null hypothesis 2. Fail to reject the null hypothesis Due to the inherent randomness in data, it is possible to make an incorrect decision. Since there are two hypotheses in a hypothesis test, there are two ways to be wrong. This can be visualized in the following confusion matrix: | | $H_0$ is the true state of the world | $H_0$ is not the state of the world | | ---------------------- | ------------------------------------ | ----------------------------------- | | $H_0$ rejected | Incorrect decision (Type-I error) | Correct decision | | Failed to reject $H_0$ | Correct decision | Incorrect decision (Type-II error) | This matrix parallels a similar matrix that can be constructed for prediction/classification tasks. ## Making decisions for hypothesis tests - [[P-values]]: if the p-value is less than the pre-determined significance level, then we reject the null hypothesis - [[Confidence Intervals]]: Equivalently, if the confidence interval does not contain the parameter under the null hypothesis, we also reject. ## Things to consider If multiple hypothesis tests are being done, we have a [[Multiple Testing Problem|multiple testing problem]]. In order to preserve the false positive rate, the significance threshold for each test needs to be changed. Hypothesis tests need to be decided before any data is collected. Trying to find a test that produces a significant result *after* the data is collected is a form of [[P-hacking|p-hacking]] and should be avoided. --- # References [[Applied Linear Regression#6. Testing and Analysis of Variance]]