Correlation and linear regression

Calculate Pearson’s linear correlation coefficient

Look at the scatter before reading the coefficient. Describe the direction and strength of a linear pattern, then check for curvature and unusual observations. A coefficient near zero rules out neither a curved relationship nor a useful model of another form.

Pearson’s product moment correlation coefficient $r$ describes the strength and direction of linear association between paired numerical variables. Calculate it alongside the scatter, because a curved pattern is not summarised adequately by linear correlation.

$$r=\frac{\sum(x-\bar x)(y-\bar y)}{\sqrt{\sum(x-\bar x)^2\sum(y-\bar y)^2}}=\frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}},$$

where

$$S_{xy}=\sum xy-\frac{(\sum x)(\sum y)}n,\quad S_{xx}=\sum x^2-\frac{(\sum x)^2}n,\quad S_{yy}=\sum y^2-\frac{(\sum y)^2}n.$$

These are the centred and summary-data forms printed in MF27. Use the calculator directly when the paired observations are supplied; use the summary formula when only totals are available. The denominator requires both variables to vary, so $S_{xx}>0$ and $S_{yy}>0$. Otherwise $r$ is undefined, rather than zero.

Read the coefficient cases

For defined $r$, $-1\le r\le1$:

$r=-1$

All points lie on a line with negative gradient.

$-1<r<0$

Negative linear association: $y$ tends to decrease as $x$ increases.

$r=0$

Zero linear correlation; a strong curved relationship may remain.

$0<r<1$

Positive linear association: $y$ tends to increase as $x$ increases.

$r=1$

All points lie on a line with positive gradient.

The source’s $r<0$ and $r>0$ rows include the perfect endpoints; the separate rows above make the cases disjoint. “Tends to” describes the overall association, not every consecutive observation. The five source illustration roles are retained in Five coefficient cases.

Interpret the coefficient with its scatter

A strong correlation does not establish that one variable directly causes the other. The source compares a person’s right-foot size (horizontal) with left-foot size (vertical): its five-point rising schematic can reflect a shared factor such as genes. Its sentence “there is no causal relationship” is too general: correlation alone establishes neither the presence nor the absence of causation.

The three source calculator plots all report $r=0.86$, yet depict a roughly linear scatter, a curve, and a close trend with an unusual observation. Causation and equal-coefficient patterns preserves those distinct roles. The source supplies no raw coordinates for these three plots; the reconstructed illustrations must not be treated as datasets whose exact coefficient is 0.86.

Explain the effect of an unusual observation

An outlier can increase or decrease the correlation, or change its sign. In the source’s rising-scatter illustration, the point $(2,7)$ sits above the main trend; removing it gives a higher positive $r$. Outlier in a rising scatter preserves this example. Investigate unusual observations rather than deleting points just to increase $|r|$. The radiation example applies the formula and explains the effect of removing an observation without recalculating.

From association to a regression model

Decide which coordinate is the response before fitting. Regression of y on x minimises squared vertical residuals on the usual axes; regression of x on y minimises squared horizontal residuals. Rearranging a fitted line does not generally fit the other response.

Linear regression fits a line to observed paired data. Check the scatter and coefficient before deciding whether a straight-line model is suitable; the usual fitting method is least squares. There are two directions: regression of $y$ on $x$ and regression of $x$ on $y$. A model can support estimation and, with appropriate assumptions, inference about population parameters; a fitted line alone does not establish those assumptions.

The source compares the two lines on the same axes: when $r\approx0$, their directions differ markedly; near $r\approx1$ they almost coincide, also near $r\approx-1$. Two regression directions preserves both schematic comparisons. This describes the same dataset and fixed axis scales; the visual angle also depends on axis scaling. Later source pages supply the residual definitions, coefficients and conditions for choosing the appropriate direction.

Fit the response coordinate: two least-squares problems

For regression of $y$ on $x$, write $y=a+bx$, with $x$ explanatory and $y$ response. Plot $x$ horizontally and $y$ vertically. Each residual is the vertical difference between an observed $y_i$ and its prediction at the same $x_i$:

$$e_i=y_i-(a+bx_i),\qquad S=\sum_{i=1}^{n}e_i^2=\sum_{i=1}^{n}(y_i-a-bx_i)^2.$$

Choose $a,b$ to minimise $S$. It is the sum of squared residuals, not a signed total residual or the sum of perpendicular distances. The source calls $S$ “Total residual”; the precise objective is the sum of squares. It depends on both $a$ and $b$.

$$b=\frac{S_{xy}}{S_{xx}}=\frac{\sum(x-\bar x)(y-\bar y)}{\sum(x-\bar x)^2}=\frac{\sum xy-(\sum x)(\sum y)/n}{\sum x^2-(\sum x)^2/n},$$ $$a=\frac{\sum y_i}{n}-b\frac{\sum x_i}{n}=\bar y-b\bar x,\qquad y-\bar y=b(x-\bar x).$$

Thus $a$ is the $y$-intercept and $b$ the gradient/regression coefficient. For defined $r$, the gradient and correlation have the same sign. The $y$-on-$x$ line passes through $(\bar x,\bar y)$ and requires $S_{xx}>0$. Use the calculator for complete paired data; MF27 formulas are useful when calculator use is not permitted or only sufficient summary information is given.

Fit x as the response

For regression of $x$ on $y$, take $y$ as explanatory and $x$ as response and write $x=c+dy$. On the same conventional axes, its residuals are horizontal:

$$e_i=x_i-(c+dy_i),\qquad S=\sum_{i=1}^{n}(x_i-c-dy_i)^2.$$ $$d=\frac{S_{xy}}{S_{yy}}=\frac{\sum(x-\bar x)(y-\bar y)}{\sum(y-\bar y)^2}=\frac{\sum xy-(\sum x)(\sum y)/n}{\sum y^2-(\sum y)^2/n},$$ $$c=\bar x-d\bar y,\qquad x-\bar x=d(y-\bar y).$$

This line also passes through $(\bar x,\bar y)$ and requires $S_{yy}>0$. The two source residual schematics are retained in Vertical and horizontal residuals. Swapping regression direction changes the quantity minimised; rearranging one fitted equation does not generally produce the other fitted regression. If $d=0$, the reverse line is $x=\bar x$ and cannot be written as a single-valued $y$ function.

The eight-observation example fits both lines, sketches them on the same axes, and uses their intersection to recover the means. Keep full coefficients for this calculation; premature rounding changes the intersection.

Predict within the observed explanatory-variable range

Identify the explanatory variable and its observed range before substituting into a fitted model. Check whether the prediction is interpolation or extrapolation. Then explain why the sample and model are relevant to the situation being predicted.

Interpolation predicts a response for an explanatory value within the range used to fit the model. Extrapolation uses an explanatory value outside that range. The source’s twelve-point sketch distinguishes the middle interval from both outside intervals; see Interpolation and extrapolation.

The source calls interpolation “generally safe”, but being inside the range does not guarantee an accurate individual prediction. A suitable form of relationship, a relevant sample and appropriate measurement still matter. With only two observations, a straight line can pass through both without providing evidence that the underlying relationship is linear.

Explain the limits of extrapolation

Extrapolation is more uncertain because the observed relationship may change outside the sample range. The source warns against using an input of $400$ when the observed inputs run from $10$ to $60$. It also warns against applying one population’s fitted model to a different population.

Check the scatter and strength of linear association before using a straight-line prediction.

The source’s word “significant” is a modelling guideline here, not a reported statistical significance test.

Choose the regression direction from the modelling setup

If the problem fixes an explanatory variable, keep its fitted response model, even when solving it backwards for a calibration estimate. If neither variable has a prescribed explanatory role, regress the variable being predicted on the supplied predictor.

$x$ is prescribed explanatory

Estimate $y$ from $x$: Use $y$ on $x$

Estimate $x$ from $y$: Rearrange the same $y$-on-$x$ equation

$y$ is prescribed explanatory

Estimate $y$ from $x$: Rearrange the same $x$-on-$y$ equation

Estimate $x$ from $y$: Use $x$ on $y$

Neither is prescribed explanatory

Estimate $y$ from $x$: Use $y$ on $x$

Estimate $x$ from $y$: Use $x$ on $y$

These six completions fill the blank Student selection table, checked against the supplied worked directions in Example 7. Solving a fixed model backwards is not the same as fitting the other regression direction, and a zero fitted gradient cannot be inverted. Calibration also has its own uncertainty; the table is the source exercise’s selection convention, not a guarantee of optimal prediction in every statistical setting.

Transform units and track the sign of correlation

Write the variable transformation explicitly. For correlation, check the signs of both scale factors. For a transformed regression, identify the scale on which residuals were minimised, then undo the transformation to interpret the prediction.

For defined correlation and nonzero scale factors, if $X=a+bU$ and $Y=c+dV$, then

$$r_{XY}=\operatorname{sgn}(bd)\,r_{UV}.$$

Translations do not change correlation. Positive rescaling of either variable does not change it; reversing just one axis changes its sign. Reversing both leaves the sign unchanged. Scaling a variable by zero makes it constant, so correlation is undefined.

The source introduces $p=0.3x+7$ and $q=-4y-50$, then says such linear transformations do not affect correlation. For this particular pair, the product of scale factors is negative, so $r_{pq}=-r_{xy}$: the magnitude stays the same but the sign reverses. Its later precise condition $bd>0$ is the condition for an unchanged value; the general “multiply or divide” wording must retain that condition.

Track what changes in the fitted line

For regression, translating data preserves the gradient but can change the intercept. Rescaling axes generally changes the gradient as well. In Example 9, the scales are both positive, so the two calculated coefficients really are equal. The source notes that a covariance proof lies beyond the syllabus; the sign rule here makes the needed qualification explicit without requiring that proof.

Linearise a nonlinear relationship

A nonlinear pattern can sometimes be represented by a straight line after transforming variables. The source introduces quadratic, logarithmic and reciprocal relationships; fit the response against the correct transformed explanatory variable and inspect the transformed scatter too. Logarithms require positive arguments, and reciprocals require nonzero inputs.

A straight-line trend changes at a constant rate. Curvature may show an increasing or decreasing rate of change, but that description alone does not identify an exponential model. The source’s broad exponential wording is therefore a description to check against the actual proposed formula. Example 10 fits logarithms of both variables, giving a power law; Example 11 compares speed and squared speed as explanatory inputs.