Porisma
Back to reading

Correlation and linear regression

Choose the response and fit a line · Diagrams · Complete reading

Choose another diagram

Two regression directions

Compare the fitted response on the same x-horizontal and y-vertical axes. These source-role sketches preserve illustrative points and lines, not numerical observations or regressions calculated from the drawing.

Weak linear association

  • Solid: y on x
  • Dashed: x on y
In the source's near-zero-correlation sketch, the two line directions differ greatly.

Strong linear association

  • Solid: y on x
  • Dashed: x on y
In the source's near-perfect positive-correlation sketch, the two lines nearly coincide. The notes also make this comparison near r = −1.

Keep the two response models distinct

The y-on-x fit minimises squared vertical residuals; the x-on-y fit minimises squared horizontal residuals on these axes. Rearranging one fitted equation does not generally produce the other fitted regression.

y=a+bxx=c+dy\begin{aligned}y&=a+bx\\x&=c+dy\end{aligned}

Both fitted lines pass through the means

For paired data with both variables varying, both least-squares regressions pass through this mean point. Use the actual observations or fitted equations to find it; the schematic positions supply no numerical mean.

(xˉ,yˉ)(\bar x,\bar y)

At exactly zero linear correlation

When both variables vary and r is zero, both regression coefficients are zero. On the same axes, y-on-x is horizontal and x-on-y is vertical. The vertical line cannot be rewritten as a single-valued function y of x.

y=yˉx=xˉ\begin{aligned}y&=\bar y\\x&=\bar x\end{aligned}
  • Use the same axes and scale when comparing the two directions. A drawn angle is not a scale-independent measure of correlation; rescaling an axis changes the angle. Do not read a numerical r from these sketches.
  • With perfect linear association and both variables varying, the fitted lines coincide. A constant variable makes the usual correlation coefficient undefined, rather than providing a zero-correlation example.
  • Choosing a regression for prediction still depends on the prescribed explanatory role. Use the linked method to distinguish direct prediction from inverse calibration.
  • Zero linear correlation does not rule out a nonlinear relationship. Inspect the scatter before accepting a line.

Compare with the source direction sketches

Expand diagram

Use the scrollbars or arrow keys to inspect details.

Two source-role schematics on common x-horizontal and y-vertical axes: near-zero correlation gives nearly horizontal y-on-x and nearly vertical x-on-y lines; near-perfect positive correlation gives almost coincident lines. The same convergence applies near perfect negative correlation. Positions and lines illustrate roles, not numerical fits.
Two source-role schematics on common x-horizontal and y-vertical axes: near-zero correlation gives nearly horizontal y-on-x and nearly vertical x-on-y lines; near-perfect positive correlation gives almost coincident lines. The same convergence applies near perfect negative correlation. Positions and lines illustrate roles, not numerical fits.
Two source-role schematics on common x-horizontal and y-vertical axes: near-zero correlation gives nearly horizontal y-on-x and nearly vertical x-on-y lines; near-perfect positive correlation gives almost coincident lines. The same convergence applies near perfect negative correlation. Positions and lines illustrate roles, not numerical fits.
Open original diagram (new tab)

Vertical and horizontal residuals

Choose the response before measuring a residual. Both sketches keep x horizontal and y vertical. Their point positions and lines are schematic; they are not numerical data or fits calculated from those positions.

y on x: vertical residuals

Ten schematic source points and a retained illustrative line. Four dashed vertical segments mark source residual roles; they are not perpendicular distances.

Predict y from x

Keep the observed x coordinate fixed. The line supplies the predicted y response at that same input.

y=a+bxy=a+bx

Measure the vertical residual

The residual is observed y minus predicted y. It is signed; the dashed segment shows its magnitude along the response coordinate.

ei=yi−y^i=yi−(a+bxi)\begin{aligned}e_i&=y_i-\hat y_i\\&=y_i-(a+bx_i)\end{aligned}

Minimise squared vertical residuals

Choose a and b to minimise this sum over every observation. Squaring prevents positive and negative residuals from cancelling. The drawing marks four residual roles, not only four terms in the sum.

Sy=∑i=1nei2S_y=\sum_{i=1}^n e_i^2

Compare each observation with its prediction.

Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₁.

e₁: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

e1=y1−y^1<0e_{1}=y_{1}-\hat y_{1}<0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₂.

e₂: above its prediction

The ringed observation lies above its prediction, so observed y minus predicted y is positive.

e2=y2−y^2>0e_{2}=y_{2}-\hat y_{2}>0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: e₃.

e₃: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

e3=y3−y^3<0e_{3}=y_{3}-\hat y_{3}<0
Dashed vertical residual from the ringed observation to its prediction.
  • Ringed point: eₙ.

eₙ: below its prediction

The ringed observation lies below its prediction, so observed y minus predicted y is negative.

en=yn−y^n<0e_{n}=y_{n}-\hat y_{n}<0

x on y: horizontal residuals

Ten schematic source points and a retained illustrative line. Four dashed horizontal segments mark source residual roles; they are not perpendicular distances.

Predict x from y

Keep the observed y coordinate fixed. The line supplies the predicted x response at that same input.

x=c+dyx=c+dy

Measure the horizontal residual

The residual is observed x minus predicted x. It is signed; the dashed segment shows its magnitude along the response coordinate.

ei=xi−x^i=xi−(c+dyi)\begin{aligned}e_i&=x_i-\hat x_i\\&=x_i-(c+dy_i)\end{aligned}

Minimise squared horizontal residuals

Choose c and d to minimise this sum over every observation. Squaring prevents positive and negative residuals from cancelling. The drawing marks four residual roles, not only four terms in the sum.

Sx=∑i=1nei2S_x=\sum_{i=1}^n e_i^2

Compare each observation with its prediction.

Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₁.

e₁: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

e1=x1−x^1>0e_{1}=x_{1}-\hat x_{1}>0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₂.

e₂: left of its prediction

The ringed observation lies left of its prediction, so observed x minus predicted x is negative.

e2=x2−x^2<0e_{2}=x_{2}-\hat x_{2}<0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: e₃.

e₃: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

e3=x3−x^3>0e_{3}=x_{3}-\hat x_{3}>0
Dashed horizontal residual from the ringed observation to its prediction.
  • Ringed point: eₙ.

eₙ: right of its prediction

The ringed observation lies right of its prediction, so observed x minus predicted x is positive.

en=xn−x^n>0e_{n}=x_{n}-\hat x_{n}>0
  • Changing the response changes the least-squares problem. Rearranging one fitted equation does not generally give the other fitted regression.
  • Measure along the response coordinate, not perpendicular to the line. A residual may be positive, negative or zero; its square is nonnegative.
  • The two source panels have distinct schematic observations. Do not treat them as numerical fits of one shared dataset, or calculate coefficients or residual totals from their drawing coordinates.
  • The two sums measure errors in different response coordinates and may have different units. Compare candidate fits for the same response on the same observations; do not rank y-on-x against x-on-y by their raw residual totals.

Compare with the source residual sketches

Expand diagram

Use the scrollbars or arrow keys to inspect details.

Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Two distinct source residual schematics: ten observations with vertical differences for y-on-x, and ten observations with horizontal differences for x-on-y. Four residual roles e1,e2,e3, en marked in each. Dashed distances are measured along the response coordinate, not perpendicular to the line. Positions are schematic; formulas are in the accompanying text.
Open original diagram (new tab)
Back to reading