Linear regression
Fit one line, let it bend, then add another measured input. The calculation stays the same: choose what each column represents, and find the combination whose predictions have the smallest squared residuals.
Choose a line by its residuals
Imagine recording displacement y in centimetres at times t in seconds relative to a reference instant. Use these chosen teaching values; they are not measurements from an experiment. Negative t means before the reference instant.
| Time t (s) | Displacement y (cm) |
|---|---|
| −1 | 2 |
| 0 | 1 |
| 1 | 2 |
| 2 | 6 |
We want one rule that predicts displacement from time. Start with a line $\widehat y=\beta_0+\beta_1t$. The hat distinguishes a fitted value from the observation y. The intercept β₀ is the prediction at t=0; the slope β₁ is the change in predicted centimetres per additional second. These are the two unknown coefficients.
No line passes through all four points: the first one-second change is -1 cm, while the next is +1 cm. For observation i, define its residual $r_i=y_i-\widehat y_i$. A positive residual puts the observation above the fitted line. We choose the line that minimizes the sum of squared residuals, abbreviated SSE. Squaring prevents positive and negative misses from cancelling.
$$\operatorname{SSE}(\beta)=\sum_{i=1}^{4}(y_i-\beta_0-\beta_1t_i)^2$$
The index i runs over observations. SSE has units of squared centimetres.
A design matrix puts the same prediction rule into every row. Fix coefficient order as intercept, then slope. At t=-1, the row [1,-1] computes β₀-β₁. The response column contains the observed displacements in that same row order.
$$X=\begin{bmatrix}1&-1\\1&0\\1&1\\1&2\end{bmatrix},\quad \beta=\begin{bmatrix}\beta_0\\\beta_1\end{bmatrix},\quad y=\begin{bmatrix}2\\1\\2\\6\end{bmatrix}$$
X is 4×2, β is 2×1, and Xβ is the 4×1 column of predictions. Thus r=y−Xβ and SSE=rᵀr.
Why do normal equations find the best line? Changing the coefficients by a column h changes the fitted values by Xh. If a candidate residual r is perpendicular to both columns of X, then $r^TXh=0$. Expanding the squared error of the changed fit gives the following certificate.
$$\|r-Xh\|^2=\|r\|^2-2r^TXh+\|Xh\|^2=\|r\|^2+\|Xh\|^2\geq\|r\|^2$$
Here ∥v∥² means the sum of squared entries of v. A dot product of zero is the perpendicularity condition.
Consequently a solution of $X^Tr=0$, or $X^TX\widehat\beta=X^Ty$, minimizes SSE. These are the normal equations. The transpose Xᵀ turns columns into rows. The residual must be perpendicular in the four-dimensional space of response lists; it need not be perpendicular to the line drawn in the time–displacement plot.
Solve the two coefficients together
- The column dot products give $X^TX=\begin{bmatrix}4&2\\2&6\end{bmatrix}$: sum 1² to get 4, sum t to get 2, and sum t² to get 6. Similarly, $X^Ty=(11,12)^T$, from sum y=11 and sum ty=12.
- Solve $4\widehat\beta_0+2\widehat\beta_1=11$ and $2\widehat\beta_0+6\widehat\beta_1=12$. Subtract half the first equation from the second: $5\widehat\beta_1=13/2$, so $\widehat\beta_1=13/10$. Substitution gives $\widehat\beta_0=21/10$.
- The fitted line is $\widehat y=21/10+(13/10)t$. At the four inputs it predicts $(4/5,21/10,17/5,47/10)$. Subtracting from the observed values gives $r=(12,-11,-14,13)^T/10$.
- Its squared error is $(144+121+196+169)/100=63/10$. Check the certificate: sum r=(12-11-14+13)/10=0, and sum tr=(-12-14+26)/10=0. Both design columns are perpendicular to r.
The coefficients are unique here: a line that is zero at two distinct times must have zero intercept and slope, so X has independent columns. In general, that condition permits the formula $\widehat\beta=(X^TX)^{-1}X^Ty$. For this example, the inverse is $\frac1{20}\begin{bmatrix}6&-2\\-2&4\end{bmatrix}$; multiplying it by (11,12) gives (42,26)/20, the same coefficients. If a question supplies the inverse and right side, this final multiplication is all that remains.
At t=3 the model predicts 21/10+39/10=6 cm. This is extrapolation: 3 lies outside the observed interval [-1,2]. The computed value is a prediction, not another observation. For any new time, use the same units and the same row [1,t]. A good fit on these four points alone cannot establish how future readings will behave.
What the matrix fit establishes
With an intercept, the residuals sum to zero; individual residuals need not vanish. The fitted-value matrix $H=X(X^TX)^{-1}X^T$ gives $\widehat y=Hy$: it selects the closest response list expressible by the design columns. This is an algebraic statement about these inputs, not a claim that time causes the fitted change in displacement.