What is a good cross entropy loss value

Ask what is a good cross entropy loss value and most answers say some version of lower is better, zero is perfect. True, and useless: it cannot tell you whether the 2.3 on your screen is progress or a bug.

The useful answer is that a good cross entropy loss value is relative to two numbers you can compute before training starts. The uniform-guessing baseline sits at the top, and the entropy of your data sits at the bottom.

One popular claim is worth killing first: that the loss ranges from 0 to 1. It does not. Probabilities live between 0 and 1; the loss is their negative log, and it has no upper bound.

The scale has no ceiling

Cross entropy charges you minus the log of the probability your model assigned to the correct class. Assign 0.7 and pay 0.36. Assign 0.5 and pay 0.69. Assign 0.01 and pay 4.6.

As the assigned probability approaches zero, the loss grows without limit. A single confidently wrong prediction can cost more than dozens of correct ones, which is the point: the loss punishes confident error hardest.

Fanout's softmax and cross entropy explainer walks through how the probabilities are produced from logits in the first place.

Start from the uniform baseline

A model that knows nothing should predict the uniform distribution: probability 1/C for each of C classes. Its loss is minus the log of 1/C, which is ln C. You can compute this before the first gradient step.

2 classes: ln 2, about 0.693.
10 classes: ln 10, about 2.303.
1,000 classes: ln 1000, about 6.91.
A 50,257-token GPT-2 vocabulary: about 10.8.

This gives you your first sanity check. A freshly initialized model should start near ln C; starting far above it usually means an initialization or logit-scale problem, not a hard dataset.

Read the value against the baseline

A loss of 2.0 means three different things depending on class count. On 2 classes it sits far above 0.693, so the model is worse than a coin flip and something is broken: shuffled labels, a target off by one, or the wrong loss reduction.

On 10 classes, 2.0 sits barely under the 2.303 baseline. The model's typical correct-class probability is e to the minus 2.0, about 0.135, against 0.10 from pure guessing. It has learned almost nothing.

On a 50,257-class language modeling task, 2.0 would beat every GPT-2 size on record. Same number, three verdicts, which is why absolute thresholds fail.

The binary case has one value worth memorizing. A validation loss parked at 0.69 means the model is a fair coin: whatever the training loss is doing, no signal is reaching held-out data.

One caveat before trusting 0.693: it assumes balanced classes. The better baseline is the entropy of the label frequencies, which is the loss of a model that always predicts the base rates.

On a dataset with a 99-to-1 class split, that baseline is about 0.056, not 0.693. A tiny loss on rare-event data can mean the model has learned nothing beyond the class ratio.

The floor is not zero either

Zero loss requires assigning probability 1 to the true label of every example. That is only possible when labels are fully determined by inputs.

Real datasets are not like that. Ambiguous examples and labeling noise leave genuine probability spread across classes, and no amount of training removes it.

The best possible model predicts that conditional distribution exactly, and still pays its entropy on every example. Below that floor, improvement is not learning; it is memorizing noise.

The Chinchilla scaling paper made this floor concrete for language. Its fitted loss curve includes a constant E of 1.69 nats that no model size or data budget removes.

The authors interpret E as the entropy of natural text on their distribution: what an ideal generative process would still pay. Chasing a language modeling loss of 0.5 is chasing a number below the floor.

The floor moves with the dataset and tokenizer, so 1.69 is not a universal constant. Label smoothing moves it too, by design: smoothed targets are not one-hot, so the minimum achievable loss becomes strictly positive.

What language model losses look like

The nanoGPT baselines put real numbers on the whole scale for OpenWebText. GPT-2 124M reaches a validation loss of 3.12, the 350M model 2.84, the 774M model 2.67, and the 1.5B model 2.54.

Every run traces the same shape: start near the ln C baseline around 10.8, descend fast, then flatten toward a data-dependent floor. Scale decides how close to the floor you land, and each size jump here buys roughly 0.1 to 0.3 nats.

Perplexity is the same information exponentiated: e raised to the loss. A loss of 3.12 is perplexity 23, meaning the model is as uncertain as choosing among 23 equally likely tokens. The untrained baseline is perplexity 50,257.

That framing explains why a 2.5 that would embarrass a 10-class classifier is a strong language modeling result. Per token, the model has cut 50,257 live options down to about 13.

Temperature does not change any of this at training time; it reshapes the softmax at sampling. Fanout's softmax temperature explainer covers that separate knob.

What is a good cross entropy loss value, in short

Compute ln C before judging anything. The baseline is one line of arithmetic and every verdict below depends on it.

Loss at or above ln C after training: the model has learned nothing. Suspect the pipeline before the architecture.
Binary loss parked near 0.69: same verdict, coin-flip edition.
Loss well under ln C and falling on validation data: learning. Track the gap to the baseline, not the absolute number.
Loss approaching zero on noisy real-world labels: suspect leakage or memorization, because the entropy floor says clean zeros should not exist.
Comparing two runs: only meaningful on the same dataset, class count, and tokenizer. A number without its baseline is noise.

For the math behind these tools, Fanout's ML math track builds the probability and log rules this post leans on.