Use a critical region

Significance and errors

Fix the level and alternative before collecting evidence. Allocate the rejection probability to the tail or tails named by the alternative. Keep the long-run error rate separate from any claim about the probability that this particular hypothesis is true.

A significance level describes the testing rule across repeated samples when the null model is true. At $\alpha=0.05$, an exact continuous test is designed to reject on 5% of those repeated samples. It does not assign a 5% probability to the null hypothesis being true or to this particular conclusion being wrong.

Imagine repeatedly checking a filling machine whose true mean is its claimed 500 ml. Before collecting data, fix the alternative and the rejection region. A result in that region supplies evidence against the claim, while leaving the possibility of an unusual sample from a correctly set machine.

Use $\alpha$ as a probability: 5% means $0.05$. For the same alternative and data, reducing the level shrinks the rejection region, so evidence sufficient at 5% may be insufficient at 1%. The test’s stated significance requires its sampling and distribution assumptions to hold.

Reference: terminology used in the retained source

A Type I error rejects a true $H_0$; a Type II error does not reject a false $H_0$. The four source outcomes are:

Do not reject

$H_0$ true: Correct decision

$H_0$ false: Type II error

Reject

$H_0$ true: Type I error

$H_0$ false: Correct decision

Use $\alpha$ as a probability: 5% means $\alpha=0.05$, not 5. In this exact continuous test,

$$P(\text{reject }H_0\mid H_0\text{ true})=\alpha$$

Reducing $\alpha$ makes the rejection region smaller and reduces Type I error; it need not reduce every kind of error. It can increase Type II error. The source’s unqualified “smaller error” sentence must be read as referring to Type I error.

Critical regions and calculator tails

Fix the rejection region from the alternative and $\alpha$ before observing the sample. At 10%, the three rules are:

• Left-sided: reject for $z\le-1.28155$.
• Right-sided: reject for $z\ge1.28155$.
• Two-sided: reject for $|z|\ge1.64485$. The source’s two-sided inequality contains absolute-value bars, although native extraction loses them.

For a 10% two-sided test the central area is 0.90. The source uses invNorm(0.9, 0, 1, CENTER); its distinct Teacher note also teaches invNorm(0.05, 0, 1, LEFT) for the negative boundary and invNorm(0.05, 0, 1, RIGHT) for the positive boundary. These are equivalent ways to allocate 0.05 to each tail. The signs differ, while the magnitudes agree.