Maths › Further Statistics 1 › The quality of tests
The quality of tests
Every test can fail two ways. It can raise a false alarm, or it can miss a real change. The size measures the first and the power measures the second, and pushing one down pushes the other up.
Builds on Hypothesis tests for Poisson and geometric models and Hypothesis testing with the binomial.
IN THIS TOPIC
- Distinguish Type I and Type II errors and compute their probabilities.
- Find the size of a test from its critical region.
- Evaluate and interpret the power function at particular alternatives.
COMMON MISCONCEPTION
A test at the 5% level is wrong 5% of the time.
Two ways to be wrong
A Type I error rejects a true H₀, which is a false alarm. Its probability is the size of the test, meaning the actual probability of landing in the critical region when H₀ holds. For a discrete distribution that usually falls a little below the nominal level. A Type II error fails to reject H₀ when some alternative is true, which is a miss.
The 5% figure describes only the first kind of failure, and only in the world where H₀ is true. Reading it as an overall error rate mistakes a conditional probability for one, and ignores Type II errors entirely.
WORKED EXAMPLE
Size and Type II probability
For X ~ B(20, p), H₀: p = 0.3 against H₁: p < 0.3, with critical region X ≤ 2. Find the size, and the probability of a Type II error when p = 0.15.
Size = P(X ≤ 2 | p = 0.3) = 0.0355, below the nominal 5%, because the discrete jump from X = 2 to X = 3 overshoots.
When p = 0.15: P(X ≤ 2) = 0.4049, so the test rejects with probability 0.4049.
The Type II probability is 1 − 0.4049 = 0.5951. This test misses a drop to 0.15 more often than it catches it.
The power function
The power at a particular alternative is the probability of correctly rejecting H₀ there, so power equals 1 − P(Type II error). Plot it against the parameter and you have the power function. It sits low near H₀, where the truth is hard to tell apart from the null, and it rises towards 1 as the alternative moves further away.
WORKED EXAMPLE
Reading the power function
For the test above, the power is 0.0355 at p = 0.3, 0.2061 at p = 0.2, 0.4049 at p = 0.15 and 0.9245 at p = 0.05. Comment.
At p = 0.3 the power equals the size, as it must. H₀ is true there, so a 'correct rejection' is really a false alarm.
Detection improves steadily as p falls. Even at p = 0.15, though, nearly six departures in ten go unnoticed.
Only for large drops does the test become reliable. Widening the critical region would raise the power everywhere, but it would also increase the size, and with n fixed there is no way to make both risks smaller at once.
GUIDED PRACTICE
Comparing two tests
Test A has size 0.05 and power 0.62 at a given alternative; test B has size 0.01 and power 0.44 at the same alternative. Which is preferable, and on what grounds?
Show the working
Neither dominates. A detects the alternative more often, B raises fewer false alarms.
The choice depends on the costs. Where a missed change is expensive, prefer A. Where a false alarm triggers something costly, prefer B.
Raising the sample size is the only way to improve both at once.
Which risk the context cares about
Exam questions increasingly ask you to argue for a level rather than apply one. A screening test that misses a disease costs a patient far more than a false positive costs the clinic, so a larger size is worth accepting to buy power. A test that shuts down a production line every time it fires had better keep its size small.
Two sentences will do it. Name what each error would mean in the situation described, then say which is worse and which way that pushes the significance level.
ASSESSMENT FOCUS
- Compute the size from the critical region, not from the nominal level. For discrete tests the two differ.
- Type II probabilities need a specific alternative value. Without one the question is incomplete.
- Power = 1 − P(Type II error), evaluated at the stated alternative.
- When comparing tests, mention both size and power. Neither number judges a test on its own.
CHECK YOURSELF
A test has size 0.043 and, at a particular alternative, a probability of 0.68 of failing to reject H₀. State the probability of a Type I error and the power at that alternative.
Show a hint
Size is the Type I probability; power is one minus the Type II probability.
Show the answer
P(Type I error) = 0.043. Power = 1 − 0.68 = 0.32, so this test misses the alternative about two thirds of the time.
Type I rejects a true H₀, with probability equal to the test's size; Type II misses a real change.
Power = 1 − P(Type II error), rising as the alternative moves away from H₀, and with n fixed size and power trade off.
WORKBOOK
Printable practice for this topic: original exam-style questions with room to work, and a fully worked answer book. Free to use; please do not redistribute or sell.
Or read them with their worked answers on the the quality of tests questions page.
CHECK YOUR PROGRESS
Rate how confident you feel with each objective for this lesson. Ratings are saved in this browser, on this device, unless you sign in.
- Distinguish Type I and Type II errors and compute their probabilities.
- Find the size of a test from its critical region.
- Evaluate and interpret the power function at particular alternatives.
Open the full revision checklist to see every objective in the course in one place.