Sampling and estimators
An observed average is one possible answer from a sampling procedure. Follow that procedure across possible samples to understand uncertainty, estimate spread, and recognize when independence or a normal approximation is justified.
One sample gives one estimate
You measure four outputs from a process and record $2,2,6,2$. Their average is $(2+2+6+2)/4=3$. Another four measurements could give a different average. Sampling begins with that distinction: the values we happened to observe are only one possible set of data.
A population is the collection, or probability model, we want to learn about. A parameter is a fixed feature of that population, such as its mean $\mu$ (“mu”). A sample is the set of observations we collect. Write $X_1,\ldots,X_n$ for the random measurements before collection and $x_1,\ldots,x_n$ for their observed values afterward. The positive integer $n$ is the sample size.
$$\overline X=\frac1n\sum_{i=1}^nX_i,\qquad \overline x=\frac1n\sum_{i=1}^nx_i.$$
The sum adds the observations indexed 1 through n. The bar denotes their arithmetic mean.
The rule $\overline X$ is an estimator: it produces an answer from whatever sample arrives. Its observed answer $\overline x=3$ is an estimate. Neither notation says that the population mean must be 3. To investigate the rule, temporarily choose a population whose mean we can calculate.
| Possible value $x$ | Probability $P(X=x)$ |
|---|---|
| $2$ | $3/4$ |
| $6$ | $1/4$ |
These probabilities specify a chosen example, not measured frequencies. The expected value is the probability-weighted average: multiply each possible value by its probability and add. Here $\mu=E[X]=2(3/4)+6(1/4)=3$. A sample can contain many more 2s than 6s, but its exact composition remains random.
For now, take independent measurements with this same distribution. Independent means learning one value does not change the probabilities for the others. Identically distributed means every measurement follows the same individual law. Together these assumptions are abbreviated IID.
Average the rule across all possible samples
- Every $X_i$ has expected value $\mu$. Expectation distributes over finite sums and constant multiples: averaging the sum of several quantities gives the sum of their averages.
- Therefore $E[\overline X]=(E[X_1]+\cdots+E[X_n])/n=(n\mu)/n=\mu$. This remains true whenever the observations have the same finite mean, even if they are dependent.
- An estimator with expectation equal to its target is called unbiased. This describes the rule over repeated sampling. A particular estimate can still be too high or too low.
Changing units also changes the average predictably: if $Y_i=aX_i+b$, then $\overline Y=a\overline X+b$ for fixed numbers $a,b$. For example, $Y_i=2X_i-1$ turns the observed mean 3 into 5.
Which population did the sample represent?
Unbiasedness above uses the distribution actually being sampled. If we only measure unusually fast outputs, averaging more of them does not repair that selection bias. The sampling procedure connects the observed data to the intended population.