Maths › Further Statistics 2 › Least squares regression and residuals
Least squares regression and residuals
A line of best fit chosen by a rule instead of by eye. The rule is to make the vertical gaps as small as possible in total, and the gaps left over tell you whether a line was the right idea.
Builds on Correlation and regression and Representing and interpreting data.
IN THIS TOPIC
- Calculate regression coefficients from summary statistics and write the line down.
- Compute residuals and use them to judge a fit and spot outliers.
- Find the residual sum of squares and say what it measures.
COMMON MISCONCEPTION
The least squares line is the one that passes closest to all the points, measuring distance perpendicular to the line.
Minimising the vertical gaps
For each data point the residual is the vertical gap between the observed y and the value the line predicts, so it is y − (a + bx). The least squares line makes the sum of the squares of those residuals as small as possible. The distances measured are vertical, never perpendicular; minimising perpendicular distances would give a different line altogether.
That choice is what separates the regression of y on x from the regression of x on y. Each minimises gaps in its own direction, so the two lines differ unless the points lie exactly on one. Use y on x when x is the controlled or explanatory variable and y is the one you want to predict. Squaring keeps the algebra differentiable, and it punishes one large miss more heavily than several small ones.
Both are in the booklet under Correlation and regression, as are the computational forms of Sxx and Sxy, so the marks are awarded for the arithmetic rather than the recall. The second formula guarantees that the line passes through the mean point, whatever the data. That is a fast check on a computed line and a fast way to recover the intercept if you lose it.
WORKED EXAMPLE
A line from summary statistics
For the five points (1, 2), (2, 4), (3, 5), (4, 4), (5, 6), find the regression line of y on x.
Σx = 15, Σy = 21, Σx² = 55, Σxy = 71, so Sxx = 55 − 15²/5 = 10 and Sxy = 71 − (15)(21)/5 = 8.
b = 8/10 = 0.8, and a = 4.2 − 0.8 × 3 = 1.8.
The line is y = 1.8 + 0.8x. Predicted values are 2.6, 3.4, 4.2, 5.0, 5.8, and the mean point (3, 4.2) sits on the line as it must.
What the leftovers tell you
Residuals are worth plotting against x. If a straight line is the right model they scatter about zero with no pattern. A curve in the residual plot means the relationship was not linear and a transformation is needed. A fan shape means the spread of y grows with x, which breaks the assumption behind the fit even though the line itself may look fine.
One residual far larger than the rest flags a possible outlier or a recording error. Investigate it before deleting it, and say in your answer why you would or would not drop the point. The residuals always sum to zero, so their sizes carry the information and their total carries none. That is a consequence of fitting the intercept: choosing it to minimise the squared total is exactly the condition that forces the residuals to balance, and a line pinned to a fixed point instead carries no such guarantee.
The residual sum of squares
Add the squares of the residuals and you have a single number for what the line failed to explain:
The booklet prints it, and prints the equivalent Syy(1 − r²) beside it, which is the quicker route when r is already in front of you. Syy is the total variation in y, so the subtracted term is the part the line accounts for. A small RSS relative to Syy means the line has done most of the work. Squaring the residuals directly gives the same answer, and comparing the two routes is a good way to check a slip.
GUIDED PRACTICE
Residuals and RSS
For the same five points, find the residuals and the residual sum of squares, and comment.
Show the working
Observed minus predicted: −0.6, 0.6, 0.8, −1.0, 0.2. They sum to zero, as they always do.
Sum of squares: 0.36 + 0.36 + 0.64 + 1.00 + 0.04 = 2.4.
By the formula: Syy = 97 − 21²/5 = 8.8, so RSS = 8.8 − 64/10 = 2.4, which agrees.
The line explains 6.4 of the 8.8 total variation. The largest residual, −1.0 at x = 4, is not extreme enough to call an outlier.
ASSESSMENT FOCUS
- Quote the formulae for the coefficients and show the summary statistics you put into them.
- Residual means observed minus predicted. The other order is the commonest sign error on this topic.
- Interpret a residual plot in words. A pattern rules the model out; formless scatter supports it.
- For the RSS, either sum the squared residuals or use the formula, and say which route you took.
CHECK YOURSELF
A regression line gives Sxx = 20, Sxy = 30 and Syy = 50. Find the gradient and the residual sum of squares.
Show a hint
One division for the gradient, then the RSS formula.
Show the answer
b = 30/20 = 1.5. RSS = 50 − 900/20 = 50 − 45 = 5, so the line explains 45 of the 50 total variation.
Least squares minimises the sum of the squared vertical residuals, giving b = S(xy)/S(xx), with the line through the mean point.
Residuals sum to zero, their pattern tests the model, and RSS = S(yy) − S(xy)²/S(xx) measures what is left unexplained.
WORKBOOK
Printable practice for this topic: original exam-style questions with room to work, and a fully worked answer book. Free to use; please do not redistribute or sell.
Or read them with their worked answers on the least squares regression and residuals questions page.
CHECK YOUR PROGRESS
Rate how confident you feel with each objective for this lesson. Ratings are saved in this browser, on this device, unless you sign in.
- Calculate regression coefficients from summary statistics and write the line down.
- Compute residuals and use them to judge a fit and spot outliers.
- Find the residual sum of squares and say what it measures.
Open the full revision checklist to see every objective in the course in one place.