The Average Is Not the Experience

An average is a compression. It takes a collection of values and returns a center: one number that can be compared, tracked and remembered. That is useful. It is also an information loss operation.

The loss is sometimes harmless. If measurements are tightly clustered, roughly symmetric and produced by the same process, the mean may describe the group well enough for the decision. When values are skewed, mixed across different populations, heavy-tailed, changing over time or attached to people who experience the extremes, the same average can be mathematically correct and practically misleading.

A center is not a typical case by default

“Average” usually means arithmetic mean: add the observations and divide by their number. The median is different: half the ordered observations fall below it and half above. The mode names the most frequent value or region. These measures can agree when a distribution is symmetric and well behaved. They can diverge sharply when it is skewed or contains extreme values.

NIST's Engineering Statistics Handbook demonstrates this with normal, exponential, lognormal and heavy-tailed distributions. In a right-skewed distribution, large values pull the mean upward while leaving the median less affected. In a heavy-tailed Cauchy distribution, the sample mean is not a stable description of location at all. This does not make the mean dishonest. It makes the choice of center conditional on the shape of the data.

Consider five response times: 2, 2, 3, 3 and 20 seconds. The mean is 6 seconds; the median is 3. Neither is false. The mean captures the cost of the long delay across the set. The median better describes the middle observation. A person who waited 20 seconds experienced neither summary.

Spread changes the claim

Two systems can have the same mean and radically different behavior. One may deliver almost every result near the center. Another may alternate between instant success and severe delay. If the decision concerns predictability, fairness, safety margin or the chance of a bad tail event, the shared average is not the relevant equivalence.

Measures of spread answer different questions. Standard deviation is sensitive to distance from the mean. The interquartile range describes the middle half. Median absolute deviation is more resistant to extreme values. A percentile answers a direct rank question: the 95th percentile is the value at or below which 95 percent of observations fall under the stated method.

No spread statistic is a universal winner. An interquartile range can hide a dangerous outer tail. A maximum can be dominated by one error. A standard deviation can be hard to interpret when the distribution is strongly non-normal. The responsible summary names the decision and chooses the statistic that preserves its failure mode.

Aggregation can reverse the story

An overall average combines groups according to their sizes. If those group sizes change, the aggregate can move even when every subgroup stays the same. If treatment or exposure is distributed differently across groups with different baselines, an overall comparison may even point in a different direction from the within-group comparisons. This family of reversals is commonly discussed under Simpson's paradox.

The practical lesson is not to split data into endless slices until a preferred answer appears. That creates its own false discoveries and tiny samples. The lesson is to identify meaningful structure before interpreting the aggregate: location, age, device class, severity, channel, time period or another variable that plausibly changes the process.

For AI evaluation, an overall accuracy score may combine easy and hard tasks, common and rare languages, or low- and high-consequence cases. The score is real, but deployment depends on the composition. The benchmark guide asks whether the tested mixture resembles the operating one.

Sequence disappears in a static average

An average over a month does not show whether results were stable, improving, deteriorating or oscillating. Ten calm days followed by one outage can share a monthly mean with eleven mediocre days. Averages also hide burstiness: the same number of arrivals distributed evenly or concentrated into one hour creates different queues and human consequences.

Plot observations over time before collapsing them. Mark process changes, outages, policy shifts and missing periods. Compare like intervals. A lower average after many people abandoned a slow queue may reflect selection, not improvement. The end-to-end delay audit treats abandonment and retries as part of the system instead of deleting them from the denominator.

The ethical problem is not solved by a correct calculation

Population summaries are essential for policy and operations. They become morally thin when they erase who bears the cost. An average increase in access can coexist with a severe loss for a smaller group. An average risk reduction can coexist with concentrated harm. Averages do not decide which distribution is acceptable.

This is where measurement becomes governance. Which groups must be visible? Which tail matters? What minimum is owed to each person? Which tradeoff requires consent or appeal? Statistics can reveal the distribution. They cannot choose the values that make one distribution preferable.

Do not respond by replacing every average with an anecdote. A vivid case also compresses reality—into one memorable person. Use both levels carefully: the distribution to see scale and variation, and concrete cases to test whether the summary still describes a recognizable experience.

The seven-field distribution audit

Add the denominator and missingness. “Average per user” changes meaning if inactive users were excluded. “Average wait” changes if abandonments vanished. “Average score” changes if failed attempts were not recorded. The claim–evidence–inference worksheet can expose the bridge between a clean summary and a broad conclusion.

What the evidence can and cannot carry

Sourced fact: different measures of location and spread respond differently to skew and extreme values, and graphical inspection reveals structure a center cannot. Inference: a decision report should preserve the feature that creates the consequence—often a percentile, subgroup or time pattern alongside the center. Value judgment: no statistic alone determines which unequal distribution is fair or acceptable. Speculation: dashboard culture may favor averages partly because one clean number is easier to govern by, even when the underlying experience is not clean.

The average is not the enemy. It is a tool with a narrow job. Use it when the center is the question. When the consequences live in the spread, the tails, the groups or the sequence, keep the distribution open.

Primary sources and further reading

• NIST/SEMATECH Engineering Statistics Handbook: Measures of Location, for mean, median, mode, skew and robustness.
• NIST/SEMATECH Engineering Statistics Handbook: Measures of Scale, for standard deviation, median absolute deviation, range and interquartile range.
• NIST/SEMATECH Engineering Statistics Handbook: Skewness and Kurtosis, for distribution shape and tail behavior.
• Stanford Encyclopedia of Philosophy: Simpson's Paradox, for the logic and interpretation of aggregation reversals.