Porisma
Back to reading

Correlation and linear regression

Choose the response and fit a line · Diagrams

Choose another diagram

Vertical and horizontal residuals

Choose the response before measuring a residual. Both sketches keep x horizontal and y vertical. Their point positions and lines are schematic; they are not numerical data or fits calculated from those positions.

y on x: vertical residuals

Ten schematic source points and a retained illustrative line. Four dashed vertical segments mark source residual roles; they are not perpendicular distances.

Predict y from x

Keep the observed x coordinate fixed. The line supplies the predicted y response at that same input.

y=a+bxy=a+bx

Measure the vertical residual

The residual is observed y minus predicted y. It is signed; the dashed segment shows its magnitude along the response coordinate.

ei=yi−y^i=yi−(a+bxi)\begin{aligned}e_i&=y_i-\hat y_i\\&=y_i-(a+bx_i)\end{aligned}

Minimise squared vertical residuals

Choose a and b to minimise this sum over every observation. Squaring prevents positive and negative residuals from cancelling. The drawing marks four residual roles, not only four terms in the sum.

Sy=∑i=1nei2S_y=\sum_{i=1}^n e_i^2

Compare each observation with its prediction.

Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₁.

e₁: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

e1=y1−y^1<0e_{1}=y_{1}-\hat y_{1}<0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₂.

e₂: above its prediction

The ringed observation lies above its prediction, so observed y minus predicted y is positive.

e2=y2−y^2>0e_{2}=y_{2}-\hat y_{2}>0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₃.

e₃: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

e3=y3−y^3<0e_{3}=y_{3}-\hat y_{3}<0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: eₙ.

eₙ: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

en=yn−y^n<0e_{n}=y_{n}-\hat y_{n}<0

x on y: horizontal residuals

Ten schematic source points and a retained illustrative line. Four dashed horizontal segments mark source residual roles; they are not perpendicular distances.

Predict x from y

Keep the observed y coordinate fixed. The line supplies the predicted x response at that same input.

x=c+dyx=c+dy

Measure the horizontal residual

The residual is observed x minus predicted x. It is signed; the dashed segment shows its magnitude along the response coordinate.

ei=xi−x^i=xi−(c+dyi)\begin{aligned}e_i&=x_i-\hat x_i\\&=x_i-(c+dy_i)\end{aligned}

Minimise squared horizontal residuals

Choose c and d to minimise this sum over every observation. Squaring prevents positive and negative residuals from cancelling. The drawing marks four residual roles, not only four terms in the sum.

Sx=∑i=1nei2S_x=\sum_{i=1}^n e_i^2

Compare each observation with its prediction.

Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₁.

e₁: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

e1=x1−x^1>0e_{1}=x_{1}-\hat x_{1}>0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₂.

e₂: left of its prediction

The ringed observation lies left of its prediction, so observed x minus predicted x is negative.

e2=x2−x^2<0e_{2}=x_{2}-\hat x_{2}<0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₃.

e₃: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

e3=x3−x^3>0e_{3}=x_{3}-\hat x_{3}>0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: eₙ.

eₙ: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

en=xn−x^n>0e_{n}=x_{n}-\hat x_{n}>0
  • Changing the response changes the least-squares problem. Rearranging one fitted equation does not generally give the other fitted regression.
  • Measure along the response coordinate, not perpendicular to the line. A residual may be positive, negative or zero; its square is nonnegative.
  • The two source panels have distinct schematic observations. Do not treat them as numerical fits of one shared dataset, or calculate coefficients or residual totals from their drawing coordinates.
  • The two sums measure errors in different response coordinates and may have different units. Compare candidate fits for the same response on the same observations; do not rank y-on-x against x-on-y by their raw residual totals.

Compare with the source residual sketches

Expand diagram

Use the scrollbars or arrow keys to inspect details.

Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Open original diagram (new tab)
Back to reading