Why averaging p99 across hosts is wrong

Use t-digest to merge latency distributions and estimate tail quantiles without retaining every observation.

TDigest Quantile Sketch

t-digest is a compact data structure for approximate quantiles such as p50, p95, p99, and p99.9.

It is designed for online accumulation and merging, which makes it useful for distributed metrics pipelines.

Why Percentiles Are Hard

Exact percentile computation needs raw values:

At billions of latency measurements, that becomes expensive for interactive dashboards.

t-digest groups sorted values into centroids:

Clusters are small near the tails and larger near the median.

That preserves accuracy where p99/p99.9 live while allowing stronger compression around common values.

The tradeoff is intentional: most observability questions care more about the slowest users than the exact median. A digest spends more detail near the extremes.

Each service, host, shard, or region can build a local digest. The metrics backend merges them to answer global percentile queries.

This is the same architectural reason HLL works for unique counts: compact summaries can be moved and combined.

Store digests by metric, tag set, and time bucket:

For a dashboard query:

Fetch all matching bucket digests. Merge the digests, not their p99 values. Query the merged digest for p50, p95, p99, and sample count.

Percentiles are not additive. Averaging p99 across hosts gives the wrong answer because hosts may have different traffic volumes and latency distributions.