Practise › Questions › The quality of tests
The quality of tests questions
Every test can fail two ways. It can raise a false alarm, or it can miss a real change. The size measures the first and the power measures the second, and pushing one down pushes the other up.
7 original questions · 26 marks · the the quality of tests notes · Further Statistics 1
Every question here is written for this library rather than taken from a past paper. Write your answer out before opening the worked one: the answers award marks point by point, and the marks are easier to see when you have something of your own to compare against.
Define a Type I error and a Type II error.
Worked answer
A Type I error rejects H₀ when H₀ is true: a false alarm. A Type II error fails to reject H₀ when a particular alternative is true: a missed change. B1 B1 for the two definitions. The first needs no alternative specified; the second cannot be computed without one.For X ~ B(10, p) with H₀: p = 0.5, H₁: p < 0.5 and critical region X ≤ 1, find the size of the test.
Worked answer
Size = P(X ≤ 1 | p = 0.5) = (1 + 10)/2¹⁰ = 11/1024 = 0.0107. M1 for the lower tail under H₀, A1 for 11/1024, A1 for 0.0107. That sits well below a nominal 5%. The distribution is discrete, so the jump from X ≤ 1 to X ≤ 2 overshoots the level and no region has size exactly 0.05.A test of H₀: p = 0.5 against H₁: p < 0.5 uses X ~ B(10, p) with critical region X ≤ 1. Find the probability of a Type II error when p = 0.2, and the power at that value.
Worked answer
P(X ≤ 1 | p = 0.2) = 0.8¹⁰ + 10(0.2)(0.8⁹) = 0.1074 + 0.2684 = 0.3758, so the test rejects with probability 0.3758: that is the power. The Type II probability is 1 − 0.3758 = 0.6242, so the test misses a drop to 0.2 nearly two times in three. M1 for the binomial terms, A1 for 0.3758, A1 for naming that as the power, A1 for 0.6242.A test of H₀: p = 0.5 against H₁: p < 0.5 uses X ~ B(10, p) with critical region X ≤ 1, and its power at p = 0.2 is 0.376. Find the power at p = 0.1 and comment on how the two compare.
Worked answer
P(X ≤ 1 | p = 0.1) = 0.9¹⁰ + 10(0.1)(0.9⁹) = 0.3487 + 0.3874 = 0.7361. M1 for the two binomial terms, A1 for 0.7361, B1 for the comparison. The power has almost doubled: the further the truth lies from H₀, the more easily the test detects it.A Poisson test has H₀: λ = 5 against H₁: λ > 5 with critical region X ≥ 10. Find its size, and its power when λ = 8.
Worked answer
Size = P(X ≥ 10 | λ = 5) = 1 − P(X ≤ 9) = 1 − 0.9682 = 0.0318. Power at λ = 8 is P(X ≥ 10 | λ = 8) = 1 − P(X ≤ 9) = 1 − 0.7166 = 0.2834. M1 for the one minus form, A1 for 0.0318, M1 for the same at λ = 8, A1 for 0.2834. A rise from 5 to 8 is caught barely more than a quarter of the time by a single observation. Both figures come from the same cumulative table, read at different values of λ.Explain why widening a critical region raises the power of a test but is not automatically an improvement.
Worked answer
A wider region rejects H₀ more readily, so it catches more true departures and the power rises at every alternative. It also rejects more often when H₀ is true, so the size rises with it and there are more false alarms. With the sample size fixed the two move together, and the right balance depends on which error costs more. B1 for the power rising, B1 for the size rising, B1 for the trade-off. Only a larger sample improves both at once.X ~ B(20, p) is used to test H₀: p = 0.3 against H₁: p < 0.3 at the 5% level. Given P(X ≤ 2 | p = 0.3) = 0.0355 and P(X ≤ 3 | p = 0.3) = 0.1071, find the critical region and the size of the test, then find the power at p = 0.15 and the probability of a Type II error there.
Worked answer
The alternative is one-tailed and low, so the critical region sits in the lower tail. P(X ≤ 3) = 0.1071 exceeds 0.05, so 3 cannot be included, while P(X ≤ 2) = 0.0355 does not. The critical region is X ≤ 2.
The size is the probability of rejecting H₀ when it is true, that is 0.0355, not the nominal 0.05. A discrete distribution rarely offers a region of size exactly 0.05, and the actual size is what must be quoted.
Power at p = 0.15 = P(X ≤ 2 | p = 0.15) = 0.85²⁰ + 20(0.15)(0.85¹⁹) + 190(0.15²)(0.85¹⁸) = 0.0388 + 0.1368 + 0.2293 = 0.405.
P(Type II error) = 1 − 0.405 = 0.595. The test misses a fall from 0.3 to 0.15 in nearly six cases out of ten, so a non-significant result here is weak evidence for H₀ rather than support for it.
M1 for comparing the tail probabilities with 0.05, A1 for X ≤ 2, B1 for the size 0.0355, M1 for the binomial terms at p = 0.15, A1 for 0.405, A1 for 0.595, B1 for the closing comment.
Power is always computed with the critical region fixed and the parameter changed. Recomputing the region at p = 0.15 is the standard error.
The same practice on paper: the printable workbook for this topic, questions and a worked answer book.
Practise the quality of tests one question at a time
The player marks nothing for you. It shows one question, waits, then shows the worked answer so you can mark yourself, and brings a question back sooner when it went badly.