Calculate unbiased estimates

Why the mean and variance estimates are unbiased

For iid observations with common finite mean $\mu$ and variance $\sigma^2$, let $\bar X=\frac1n\sum_{i=1}^nX_i$. By linearity,

$E(\bar X)=\frac1n E(X_1+\cdots+X_n)=\frac1n[E(X_1)+\cdots+E(X_n)]=\frac{n\mu}{n}=\mu.$

This expectation calculation only needs the common mean. Independence is needed when dropping covariances in the variance calculation:

$\operatorname{Var}(\bar X)=\frac1{n^2}\operatorname{Var}(X_1+\cdots+X_n)=\frac1{n^2}\sum_{i=1}^n\operatorname{Var}(X_i)=\frac{n\sigma^2}{n^2}=\frac{\sigma^2}{n}.$

The standard error is $\sigma/\sqrt n$. For $n>1$ and positive variance, this variance is smaller than the individual variance; at $n=1$ it is equal.

For dependent observations, covariance terms must be included. An exact finite-population sample without replacement may require a finite-population correction; do not silently replace that sampling design with iid draws.

For $n>1$, define

$S^2=\frac1{n-1}\sum_i(X_i-\bar X)^2=\frac1{n-1}\left[\sum_i X_i^2-\frac{(\sum_iX_i)^2}{n}\right].$

Expand the centred sum:

$\sum_i(X_i-\bar X)^2=\sum_iX_i^2-2\bar X\sum_iX_i+n\bar X^2=\sum_iX_i^2-n\bar X^2,$

because $\sum_iX_i=n\bar X$. Here $\bar X$ is common to every summand, but it is still a random variable; the source’s phrase “$\bar X$ is a constant” is only meaningful with respect to the summation index. We do not pull it out of an expectation as a deterministic quantity.

Using $E(X_i^2)=\sigma^2+\mu^2$ and $E(\bar X^2)=\operatorname{Var}(\bar X)+[E(\bar X)]^2=\sigma^2/n+\mu^2$,

$E(S^2)=\frac{n(\sigma^2+\mu^2)-n(\sigma^2/n+\mu^2)}{n-1}=\frac{(n-1)\sigma^2}{n-1}=\sigma^2.$

Thus the observed $s^2$ is an unbiased variance estimate. This does not say that $S=\sqrt{S^2}$ is generally unbiased for $\sigma$.

Formula and calculator notation tables

Choose the case that matches the supplied data. Calculate the mean and variance separately, then check which denominator the calculator uses.

Given data: $\sum x,\sum x^2$ and $n$

• Mean estimate: $\bar x=\sum x/n$
• Unbiased variance estimate: $s^2=[\sum x^2-(\sum x)^2/n]/(n-1)$

Given data: $\sum(x-a),\sum(x-a)^2$ and $n$

• Mean estimate: $\bar x=a+\sum(x-a)/n$
• Unbiased variance estimate: $s^2=[\sum(x-a)^2-(\sum(x-a))^2/n]/(n-1)$

Given data: Variance computed with denominator $n$

• Mean estimate: Use the observed mean
• Unbiased variance estimate: Multiply that variance by $n/(n-1)$

Quantity: Population parameter

• Mean notation: $\mu$
• Variance notation: $\sigma^2$

Quantity: Observed sample summaries

• Mean notation: $\bar x$
• Variance notation: Source “sample variance”: $\sum(x-\bar x)^2/n$; calculator $\sigma_x^2$

Quantity: Unbiased population estimates

• Mean notation: $\bar x$
• Variance notation: $s^2=\sum(x-\bar x)^2/(n-1)$; calculator $S_x^2$

“Sample variance” is used with different denominator conventions in different texts. This source uses the denominator-$n$ version in its MF27 comparison, then multiplies by $n/(n-1)$. Check the formula rather than inferring it from the name.

For nonconstant data, calculator $S_x$ is larger than $\sigma_x$ because its squared value has denominator $n-1$; both are zero for constant data.

Square the stored $S_x$ to obtain $s^2$ and avoid premature rounding.

Raw values go in L1; a frequency table also requires its frequencies in L2, as Examples 2 and 3 show.

Why shifting simplifies the arithmetic

The lecture contrasts $\sum x=1000005$ with $\sum(x-10000)=5$ for $n=100$. A shift changes the mean but not the centred sum of squares. First,

$\frac{\sum(x-a)}n+a=\frac{\sum x-na}n+a=\frac{\sum x}n.$

Next, expand both terms:

$$\begin{aligned} \sum(x-a)^2-\frac{[\sum(x-a)]^2}{n} &=\sum x^2-2a\sum x+na^2-\frac{(\sum x-na)^2}{n}\\ &=\sum x^2-2a\sum x+na^2-\frac{(\sum x)^2}{n}+2a\sum x-na^2\\ &=\sum x^2-\frac{(\sum x)^2}{n}. \end{aligned}$$

Dividing by $n-1$ proves the shifted variance formula. Do not add the shift back to a variance.

Calculate, then explain what the estimate means

For ten observations, $\sum(x-50)=20$ and $\sum(x-50)^2=130$. Find estimates of the population mean and variance. Explain which estimate needs the shift restored and why the variance estimate uses 9.

After your attempt: compare the reasoning

The mean estimate is $50+20/10=52$. The unbiased variance estimate is $[130-20^2/10]/9=10$. Shifting changes the mean but leaves centred deviations unchanged. The deviations from the sample mean must sum to zero, so only nine can vary freely; this motivates the denominator $n-1$. The estimate 10 is not a guarantee that the population variance equals 10.