Probability, Sampling & Benchmark Statistics
Ask a language model the same question twice and you may get two different answers. Time the same operation twice and you will get two different numbers. Both facts have the same mathematical shape, a draw from a probability distribution, and this lesson is the small dose of probability and statistics the course needs to handle both honestly.
As with the rest of this module, nothing goes past napkin level. By the end you should be able to look at any number built on randomness, a model's answer or a benchmark's summary line, and ask the two questions that keep it honest: what distribution is this one number standing in for, and how many draws from it were actually taken.
Distributions over outcomes
When an outcome is uncertain, describing it takes a list: every outcome that could happen, each with a weight saying how likely it is. Make every weight zero or larger, and make the weights sum to exactly one, and the list is a probability distribution. A die is the classic example: six outcomes, each with weight one sixth.
The two rules are the whole definition, and each says something plain. Nonnegative means no outcome can be less than impossible. Summing to one means the outcomes on the list are all the outcomes there are.
The previous lesson ended with softmax, a function that turns any list of scores into positive weights summing to one. That output satisfies the definition exactly. So a language model, at each step of generating, hands you a probability distribution: one weight for every entry in its vocabulary, the fixed list of tokens it is able to produce, 128,256 of them for Llama 3.1 8B.
After “The capital of France is”, a well-trained model might put weight 0.62 on “ Paris”, 0.31 on “ located”, and scatter the remaining 0.07 across the other 128,254 entries. The numbers are made up, and understanding one token measures the real ones, which turn out flatter than this. What is not made up is the mechanism: a set of plausible continuations carries most of the weight, and a vast floor of near-zero weights carries the rest. The model is a die with 128,256 faces, and every prompt loads it differently.
Sampling from a distribution
A distribution describes possibility; generation needs one actual token. There are two ways to get one from the list.
The first is to take the heaviest outcome, every time. The rule is deterministic: the same distribution yields the same choice, so the same prompt yields the same reply, run after run.
The second is to draw in proportion to the weights, which is what sampling means. Picture the weights laid end to end on a line of length one; because they sum to one, they tile it exactly. Pick a random point on the line and take the outcome whose segment it covers. “ Paris” at 0.62 owns sixty-two percent of the line, so it wins about three draws in five; a token carrying weight 0.001 wins one draw in a thousand. Unlikely is not impossible, and that is the property sampling buys: the same prompt can produce a different reply on every run.
Why accept that? Because taking the heaviest token at every step tends to produce flat, repetitive text, and a measured amount of chance reads as variety. How much chance, and how systems reshape the distribution before drawing from it, is covered in understanding one token; for this module the two bare policies are enough.
The randomness itself is tamer than the word suggests. The random point comes from a pseudorandom number generator: an algorithm that turns a starting value, the seed, into a long sequence of numbers that behaves like chance but is fully determined by the seed. Record the seed and the run can be replayed, draw for draw; that is all reproducible means in a pipeline with sampling inside it.
Common misconception. A model that answers the same prompt differently is not glitching, and nothing inside it changed between runs. The weights are frozen, and the distribution they produce for a given prompt is the same every time; the variation is injected after the distribution, by the draw, on purpose. If you need the variation gone you do not need a different model: take the heaviest token, or pin the seed.
Sampling has a second meaning in this field, and this lesson needs both. When you time twenty runs of an operation, a statistician says you have drawn a sample: twenty draws from the distribution of all the runs you could have timed. Generating a token and timing a run are the same mathematical act.
The difference is that the model's distribution can be printed before you draw from it, while a measurement's distribution is visible only through the draws. The rest of the lesson is about reading it there.
Summarizing repeated measurements
Time the same operation twenty times and you get twenty different numbers. The machine was doing other things, memory was in a different state, something on a network hesitated; the causes are endless and mostly invisible, so they get a collective name, noise. Here is such a set of timings, in milliseconds, already sorted:
88, 92, 94, 95, 97, 98, 99, 100, 100, 101, 101, 103, 104, 106, 108, 110, 112, 115, 121, 900
Nineteen of them tell one story: the operation takes about a hundred milliseconds. The twentieth tells another: occasionally, for reasons the list alone cannot explain, it takes nine times that.
The most common summary is the mean: add everything, divide by the count. Here that is 2,844 ÷ 20, or 142.2 ms, and it is a strange headline for this data, because nineteen of the twenty runs were faster than the average run. The mean is an accounting number. Multiplied by a run count it returns total time, and total time is what a budget is made of; the same property lets one run at 900 drag the summary far above where the runs actually live.
The median resists the drag. Sort the values and take the middle one, here the average of the tenth and eleventh, both 101: the median run took 101 ms. Push the outlier from 900 to 9,000 and the median does not move; the mean climbs to 547.2. When the two disagree, the data is lopsided, and the direction of the gap points at the long side.
Lopsided is the natural state of timing data. The lesson memory and throughput arithmetic put a floor under an operation's time, a bound its arithmetic cannot beat, so the left side of a timing distribution is a wall. No matching wall bounds the right side, because delays stack: a busy machine, a retry, a pause. Runs pile up just above the floor and straggle off to the right, which is why the mean sits above the median here.
Spread has a standard number too. Take each run's distance from the mean, square it, and average the squares: the variance, here 30,285 square milliseconds, a unit nobody can picture. Take the square root to get back to milliseconds: the standard deviation, about 174 ms.
Read it with care, because squaring makes large deviations dominate: the single run at 900 contributes about 95 percent of the variance, and dropping it collapses the standard deviation of the remaining nineteen runs to 8 ms. No single number for spread can describe a tail.
Percentiles and the tail
Percentiles read the distribution directly instead of compressing it. Sort the runs ascending; the p-th percentile is the level that p percent of runs came in at or below. Three of them have become nouns engineers use daily: P50, which is the median by another name; P95, the level 95 percent of runs came in at or below; and P99, the level 99 percent did.
On the twenty runs, P50 is 101 ms, P95 is 121 ms, and P99 is 900 ms. The first two describe the crowd, the third describes the straggler, and no averaging blurs which is which. That is the trio's appeal: a distribution's shape in three numbers, middle, edge of normal, and tail.
Run it and the page's numbers come back: mean 142.2, median 101, then 101, 121, 900. Note what each percentile refuses to see: P95 is identical whether the worst run took 900 milliseconds or 90 seconds. A percentile is one rung of a ladder, and you read the tail by asking for higher rungs.
Small samples
On twenty runs, P99 is simply the worst run: the nearest-rank rule asks for the smallest value with 99 percent of runs at or below it, and with twenty values that is the twentieth. The page's P99 of 900 ms is one unlucky draw promoted to a summary statistic, not a property of the operation.
Small samples destabilize even the definition. Conventions differ on what to do between ranks. Nearest rank says P95 = 121, while a common interpolating convention, the default in several standard tools, walks five percent of the way from the nineteenth value toward the twentieth: 121 + 0.05 × (900 − 121), about 160. Same twenty numbers, two standard answers, about a third apart; statisticians catalogue nine such conventions in common software. On a thousand runs the conventions agree to within noise. The disagreement itself is the signal: twenty runs cannot pin down a tail, and the conventions diverge exactly where the data runs out.
So the habit I most want you to take from this lesson is a question, not a formula: before trusting any statistic, ask how many measurements are underneath it. A tail percentile needs hundreds of runs before it measures the operation rather than one run; a mean needs fewer, but more than the two or three a quick check produces. The working rule of measuring until the number stops moving, and the right way to run and report such measurements when the runs are inference requests, belongs to inference benchmarking, which gives P50, P95, and P99 a lesson of their own.
The summaries, side by side:
| Summary | What it answers | Where it misleads |
|---|---|---|
| Mean | Total ÷ count, the accounting view | One outlier moves it; most runs can be faster than it |
| Median / P50 | The typical run | Says nothing about how bad the bad runs are |
| P95, P99 | How bad the edge and the tail get | On small samples they are individual runs |
| Standard deviation | Spread around the mean, in one number | Squaring lets one outlier dominate it |
That completes the module's mathematics: amounts, rates, and ratios from memory and throughput arithmetic, shapes and attention arithmetic in the two lessons between, and now distributions and the statistics of repeated runs. What remains is tooling. The next lesson sets up the Python and PyTorch working habits that the course's code, starting with your first real model run in running your first LLM, will assume.