Tail latency: what it is and how to measure it

Tail latency is the response time of the slowest few requests, usually reported as the 99th or 99.9th percentile. It matters because one user action often waits on many requests, so a delay that hits 1 request in 100 can reach most of your users. A load test that reports only the average or median will not show it.

The table assumes each call is slow with a 1% chance, independently of the others: the share is 1 - 0.99^n.

What counts as the tail?

The tail is the part of the response-time distribution above a high percentile, most often p99 and p99.9. The median describes a typical request, and the tail describes the requests people complain about.

A percentile is a cut-off: p99 = 800 ms means 99% of requests took 800 ms or less and 1% took longer. The tail sits to the right of that cut-off on a response-time histogram, often as a long, thin stretch far from the main peak. Our guide to throughput vs latency explains how latency and percentiles are measured in load testing tools.

Why does tail latency matter so much?

Because requests multiply. A page or API call that fans out to many services finishes only when the slowest of them answers, so the tail of each service becomes the typical experience of the whole.

The standard reference is “The Tail at Scale” by Jeffrey Dean and Luiz André Barroso, published in Communications of the ACM in 2013 (paper PDF, checked on 19 October 2026). Their example: a server that usually answers in 10 ms but takes one second at the 99th percentile. If one request touches one such server, 1 user request in 100 is slow. If it must collect answers from 100 of them in parallel, 63% of user requests take more than one second. Even with only 1 slow call in 10,000, a service with 2,000 servers sees almost 1 in 5 requests over a second.

They also report measurements from a real Google service: the 99th percentile for a single leaf request was 10 ms, but for all leaf requests to finish it was 140 ms, and waiting for the slowest 5% of them accounted for half of that.

What causes tail latency?

Usually interference from other work on the machine or in the network. The same request is fast most of the time and slow when something else is going on.

Dean and Barroso list the usual suspects:

• Shared resources: other work on the same machine competing for CPU cores, processor caches, memory bandwidth or network bandwidth.
• Background daemons that wake up and take a slice of the machine.
• Maintenance work such as log compaction or garbage collection in managed runtimes.
• Queueing in servers and network switches, which amplifies every other source.
• Power and thermal limits that slow a CPU after a burst of work.

In load tests, the same list shows up as GC pauses in JVM or Node.js services, connection pool exhaustion, a slow database query that only some requests trigger, cold caches after a deploy, and retries after a timeout. A tail that grows with load points at queueing; a tail that is there even at low load points at something periodic, such as GC or a scheduled job.

How do you measure tail latency in a load test?

Collect enough samples, ask your tool for the right percentiles, and never average percentiles across machines or time windows.

Enough samples. p99 is decided by the slowest 1% of requests, so a run with 500 requests puts it on 5 of them. For a stable p99 you want thousands of requests per transaction, and for p99.9 tens of thousands. Short smoke tests are fine for catching errors and poor for judging the tail.

The right percentiles. Defaults often stop short of the tail. k6’s summaryTrendStats default is avg,min,med,max,p(90),p(95), so p99 only appears if you add it (k6 options reference, checked on 19 October 2026):

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9),max" \
  script.js

JMeter’s Aggregate Report shows the 90%, 95% and 99% lines, and the aggregate_rpt_pct1 to aggregate_rpt_pct3 properties change which percentiles appear (checked on 19 October 2026).

No averaged percentiles. The p99 of a whole run is not the average of each minute’s p99, and it is not the average of each load generator’s p99 either. Merge the raw samples or a histogram, then compute the percentile once. The coordinated omission article covers HdrHistogram, the usual tool for that, and a second trap: a load generator that waits for slow responses sends fewer requests while the system is slow, which hides the tail you are looking for.

LoadFocus, which publishes this site, shows P95 and P99 tiles for each run, a response-time histogram built from the individual samples, and the slowest requests, so a long tail to the right of the peak is visible at a glance.

How do you set a pass/fail limit on the tail?

Put a limit on p99 per key transaction, next to a p95 limit and an error-rate limit. Base it on what users of that transaction need, and keep the run long enough for p99 to be stable.

A sensible set for a checkout API could read: p95 under 400 ms, p99 under 1,200 ms, errors under 0.5%. In k6 those are thresholds on http_req_duration such as p(99)<1200, and a failed threshold makes k6 exit with a non-zero code (k6 thresholds, checked on 19 October 2026). LoadFocus has a Max P99 response time field next to Max P95 for cloud, JMeter and k6 tests, and every run is then graded PASS or FAIL. Our guide to load test pass/fail criteria covers the rest of the set.

If you also report an Apdex score, remember it puts every request over four times T in one bucket, so it cannot tell a 2-second tail from a 30-second one. The p99 can.

How do you reduce tail latency?

Remove the sources of variability where you can, and design callers so one slow call does not hold up the whole response.

Common fixes, roughly from cheapest to most involved:

• Tune GC and pool sizes so pauses and waits for a free connection stay short under load.
• Keep headroom. Queueing grows sharply near saturation, so spare capacity shortens the tail; the throughput vs latency guide shows how to find the knee.
• Set timeouts and limit retries so a stuck call fails fast instead of piling up behind it.
• Use hedged requests for idempotent reads: send a second copy to another replica if the first has not answered in time. Dean and Barroso describe sending the backup only after the 95th-percentile expected latency, which limits the extra load to about 5%. In one of their tests, a hedge after 10 ms cut the 99.9th percentile from 1,800 ms to 74 ms while sending just 2% more requests.
• Fan out less. Fewer backend calls per page means fewer chances to hit the tail, as the table at the top shows.

After each change, run the same test again with the same load profile and compare p99 and p99.9 between runs.

FAQ

Is tail latency the same as p99? p99 is the most common way to report it, and the tail is the whole slow end of the distribution. Track p99.9 and the maximum as well when the run has enough requests.

Why is my p99 so much higher than my average? Because a few slow requests barely move the average and fully define the p99. A large gap is normal; a gap that grows as load rises usually means queueing.

How many requests do I need to measure p99? At the very least a few thousand per transaction, so that the p99 rests on dozens of slow requests. For p99.9, plan for tens of thousands.

Related reading

• Coordinated omission: what it is and how to avoid it
• Throughput vs latency: how they interact under load
• Apdex score: what it is and how to calculate it
• Load test pass/fail criteria: how to set them
• Little’s law in load testing: size and check your tests
• YCSB: workloads A-F, how to run it and its limits
• SLO vs SLA: the difference, SLIs and error budgets

The technical claims in this article have not been independently reviewed.