loadtest.app

Amdahl's law: formula, examples and scaling limits

Sun, 01 Nov 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Amdahl's law says that the speedup from running work in parallel is capped by the part that cannot run in parallel. If a fraction p of the work spreads across n processors, servers or threads, the speedup is S = 1 / ((1 − p) + p/n), and however large n grows it never passes 1 / (1 − p). With 95% of the work parallel, the ceiling is 20 times, even on a thousand machines.

The numbers below are arithmetic, with no test run behind them. The papers cited were checked on 1 November 2026.

Parallel fraction p	n = 2	n = 8	n = 64	n → ∞
0.50	1.33	1.78	1.97	2
0.90	1.82	4.71	8.77	10
0.95	1.90	5.93	15.42	20
0.99	1.98	7.48	39.26	100

What does the law say in simple terms?

A job can finish no faster than its serial part, so extra workers shrink only the rest. Once the parallel part is small next to the serial part, more workers barely change the total time.

Take a 10-second job where 1 second is serial (a lock, a single write) and 9 seconds split evenly, so p = 0.9. Eight workers finish it in 1 + 9/8 = 2.125 seconds (4.71 times faster); sixty-four need about 1.14 seconds. Nothing gets it under 1 second, so 10 is the ceiling.

Where does the law come from?

Gene Amdahl made the argument in 1967, at the AFIPS Spring Joint Computer Conference, in a paper defending the single-processor approach.

We have not read the full text of Validity of the single processor approach to achieving large scale computing capabilities. Mark Hill and Michael Marty's Amdahl's Law in the Multicore Era (IEEE Computer, 2008) summarizes it: one fraction infinitely parallelizable with no scheduling overhead, the rest totally sequential, and the speedup described "without presenting an equation".

What does it mean for load and capacity testing?

Throughput stops growing in step with servers, threads or pods once a shared component handles part of every request one at a time. A load test shows it as a throughput curve that flattens while you keep adding capacity.

Neil Gunther's 2008 paper, cited below, writes the same law as N / (1 + σ(N − 1)), with σ the serial fraction, and measures scalability as throughput at N divided by throughput at 1. In a web system, the serial part is whatever every request passes through in turn: a lock or a row every request updates, a single database primary taking all writes, a single queue consumer or a rate-limited third-party API.

How do you find the serial part in a load test?

Run the same test at several load levels and watch which resource saturates as throughput flattens. The component at its limit when extra capacity stops paying off is the serial part, or the queue in front of it.

Typical signs: throughput levels off while the app tier has spare CPU, waits grow in one shared place (locks, database connections, queue depth), and response time rises in step with users, which is Little's law at a ceiling. A run at one constant load cannot show this, so step the load. In LoadFocus, the Ramp Up Steps setting adds virtual users in equal batches, and the Capacity panel groups a finished run by active virtual users and, on paid plans, lists p95, requests per second and error rate per level for single-generator runs. The panel names a capacity level only when the run climbed through at least three levels and the top level was at least double the lowest. It fits no scalability model, so the fit below is yours to do.

What is Gustafson's law?

It is the counterpoint: if the problem grows with the machine, the serial share stays small and speedup grows almost linearly. Amdahl holds the work fixed; Gustafson holds the run time fixed.

John Gustafson's 1988 note Reevaluating Amdahl's law reported speedups of 1016 to 1021 on a 1024-processor hypercube at Sandia, for applications with a serial share of 0.4 to 0.8 percent, and argued that users enlarge the problem to fill the time they have. With s' as the serial share measured on the parallel run, scaled speedup, a formula the note credits to E. Barsis at Sandia, is N + (1 − N)s'.

At s' = 0.05 and 64 processors that gives 60.85, against 15.42 for Amdahl, because the two 5% figures come from different machines. In a web system, Gustafson fits when each added server works on its own share of the data; Amdahl fits when every extra user still touches the same shared row.

What does the Universal Scalability Law add?

A second term for coherency, the cost of keeping shared data consistent between workers, which grows with the number of worker pairs. With it, throughput can peak and then fall as you add capacity.

Gunther's A General Theory of Computational Scalability Based on Rational Functions (arXiv, 2008) states the Universal Scalability Law (USL) as:

C(N) = N / (1 + σ(N - 1) + κN(N - 1))
peak at N* = sqrt((1 - σ) / κ)

σ is contention (serialization or queueing) and κ is coherency (inconsistent copies of data); with κ = 0 it is Amdahl's formula. Gunther's worst-case examples include updating database records, and he calls throughput beyond N* "retrograde". With illustrative values σ = 0.05 and κ = 0.001, the peak sits near 31 workers: relative throughput is 9.03 at 32 and 7.82 at 64, where the σ term alone predicts 15.42.

How do you fit the USL to a load test series?

Measure throughput at several concurrency levels, divide each by the throughput at one user, and fit σ and κ by least squares.

Rearranged, the formula reads N / C − 1 = σ(N − 1) + κN(N − 1), which is linear in σ and κ. The values below are generated from σ = 0.05 and κ = 0.001; put your own requests per second per level in x:

import numpy as np

n = np.array([1, 2, 4, 8, 16, 32])          # virtual users
x = np.array([50, 95, 172, 284, 402, 452])  # req/s per level
c = x / x[0]                                # relative capacity
A = np.column_stack([n - 1, n * (n - 1)])
(s, k), *_ = np.linalg.lstsq(A, n / c - 1, rcond=None)
print(f"sigma={s:.3f} kappa={k:.4f}")
print(f"peak near {np.sqrt((1 - s) / k):.0f} users")

It prints sigma=0.050, kappa=0.0010 and a peak near 31 users. Change only concurrency between levels, and include a one-user level, since every point is divided by it.

Where do these models stop being useful?

They compress a throughput curve into one or two constants, so they cannot name the lock at fault or follow a system that changes under load. Use them to read a measured curve and extend it a short way: a fit on levels far below the knee says little about the peak.

Constant parameters. A warming cache, an autoscaler or a pool hitting its limit changes σ and κ mid-test.
Throughput only. p95 and p99 can degrade first: check tail latency too.
The load generator. If it saturates first, the flat curve is its own.
Closed tests. Fixed virtual users measure a closed system; real users arrive at their own rate.

FAQ

Can you explain Amdahl's law in a simple way? A job is only as fast as the part of it that cannot be shared out. If one tenth of the work must run serially, no number of extra workers makes the job more than ten times faster, because that tenth still takes the same time.

What is Gustafson's law? Gustafson's law says that when the problem grows with the number of processors, speedup grows almost linearly, as N + (1 − N)s', with s' the serial share of the parallel run. John Gustafson published it in 1988 (crediting the formula to his Sandia colleague E. Barsis) as a counterpoint to Amdahl's fixed-size assumption.

Queueing theory: what it is and why latency spikes

Sun, 01 Nov 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Queueing theory is the mathematics of waiting lines: from how fast work arrives, how long each job takes and how many servers there are, it predicts how long jobs wait and how long the queue gets. For software, its most useful result is that response time grows slowly at moderate load and then sharply as a server approaches 100% busy. In the simplest model, a server busy 90% of the time answers ten times slower on average than an idle one.

The figures below are arithmetic from standard formulas, checked with a script, with no test run behind them. Sources were checked on 1 November 2026.

Utilization ρ	Time in system W ÷ service time S	If S = 20 ms
0.5	2	40 ms
0.7	3.3	67 ms
0.8	5	100 ms
0.9	10	200 ms
0.95	20	400 ms
0.99	100	2,000 ms

These are means for the M/M/1 queue: one server, Poisson arrivals and exponential service times.

What is queueing theory?

It is the branch of probability that models work arriving, waiting and being served. A queue is described by its arrivals, its service times, its number of servers and its queue discipline, the order in which waiting jobs are picked.

Larson and Odoni's Urban Operations Research, an open MIT text, lists bank tellers, checkout counters and toll booths as classic cases. A web server fits the same frame: requests arrive, threads or cores serve them, and the rest wait in a queue.

Where did queueing theory come from?

From the telephone exchange, where A. K. Erlang worked out the first results for the Copenhagen Telephone Company.

Erlang's 1909 paper showed that calls made at random follow the Poisson distribution, and his 1917 formula for loss and waiting time was soon used by telephone companies in many countries, according to his MacTutor biography. David Kendall's 1953 paper, Stochastic Processes Occurring in the Theory of Queues and their Analysis by the Method of the Imbedded Markov Chain, introduced the notation still in use, as Geoffrey Grimmett's account of his work notes. Little's proof of L = λW and Kingman's heavy-traffic results followed in 1961.

What does Kendall's notation such as M/M/1 mean?

It names a queue in three parts, A/S/c: the distribution of gaps between arrivals, that of service times, and the number of servers. M/M/1 means random arrivals, exponential service times and one server.

M stands for memoryless (Poisson arrivals or exponential times), D for deterministic and G for general, meaning any distribution: M/D/1 is one server with a constant service time.

How does response time grow with utilization?

In an M/M/1 queue, the mean time in the system is W = 1/(μ − λ), which equals S/(1 − ρ). As utilization ρ approaches 1, W grows without limit.

lambda = arrival rate        mu = service rate (1 / S)
rho    = lambda / mu         (utilization, must stay below 1)
W      = 1 / (mu - lambda)   = S / (1 - rho)

Larson and Odoni derive this for a single-operator call centre and show there is no steady state at ρ ≥ 1: the queue keeps growing. The same source shows the time in the system is exponentially distributed, so every percentile scales by the same 1/(1 − ρ) factor, and the p95 is about three times the mean: 600 ms in the 0.9 row above.

Why does latency knee before the CPU reaches 100%?

Because a request waits whenever it finds the server busy, and that happens more often as utilization rises. Under M/M/1 assumptions, response time has already doubled at 50% utilization.

The saturating resource may also be a connection pool, a database lock or a single-threaded event loop, and the busiest one sets the curve. The early knee is why target utilization sits well below 100%. Work back from your latency budget: with S = 20 ms and a 100 ms mean target, the table allows about 80%, and a 60 ms target about 67%, before bursts take their share.

LoadFocus's Capacity panel groups a finished run by the number of active virtual users and reports the last level that held before the error rate passed 1% or the p95 rose above double the best level, provided no higher level recovered. It needs a single-generator run that ramps up through at least three load levels, with the top at least double the lowest. By the arithmetic above, if the lightest level left the bottleneck nearly idle, the doubling trips just past 50% utilization; if the lightest level already loaded it to ρ0, the doubling trips later, above (1 + ρ0)/2 under M/M/1.

Why does variability matter as much as utilization?

Queues come from randomness: bursts of arrivals and occasional slow jobs. In heavy traffic, waiting time grows with the variability of both, multiplied by a ρ/(1 − ρ) term.

Kingman's 1961 paper, The single server queue in heavy traffic, shows that when a single server's traffic intensity is near 1, the queue is long and the waiting time approximately exponential. Course notes from TU Eindhoven and the Technion give an approximation for its mean (the Technion calls it the Allen-Cunneen approximation; it is also widely known as Kingman's formula):

Wq ≈ rho / (1 - rho) × (ca² + cs²) / 2 × S
ca, cs = coefficient of variation (std dev / mean)
         of arrival gaps and of service times

Wq is the wait before service starts. Exponential times have c² = 1, giving the M/M/1 wait of 9S at 90% utilization. A constant service time halves it to 4.5S; cs² = 4, as a mix of cache hits and slow database calls can produce, raises it to 22.5S. Steady synthetic traffic is less variable than production, so a test tends to show less queueing than real bursts will.

How do more servers (M/M/c) help?

Several servers sharing one queue absorb random bursts much better than the same servers with a queue each, so each server can run busier for the same wait.

From Larson and Odoni's M/M/m formulas, with every server 90% busy, W is 10S for one server, 5.3S for two sharing a queue and 3.0S for four. Four instances behind a balancer that picks a free one behave like the last case; four that each take a random quarter of the traffic behave like separate M/M/1 queues at 10S.

How does Little's law fit in?

Little's law, L = λW, ties the average number of requests in a system to their arrival rate and their time inside, whatever the distributions. Queueing theory gives you W, and Little's law turns it into L.

In the 0.9 row, 45 requests per second at W = 200 ms means 9 requests inside on average. Our Little's law article uses the same law to size and check load tests.

Where do these models break down in load testing?

They assume an open stream of independent arrivals and a steady state. A test with a fixed number of virtual users is a closed system, where arrivals slow down as responses do.

Closed tests. With N virtual users, at most N requests are in the system, so the request rate falls where an open queue would grow, and think time lowers it further. Our open vs closed workload model article explains why that hides saturation.
LoadFocus runs k6 scripts, but arrival-rate scenarios run there with a fixed number of users, as its script validator warns. Read those runs as closed tests: past the knee, expect a flat request rate and rising response times.
Steady state. Warm-up, caches and autoscaling change the service rate mid-run, so compare levels held long enough to settle.

FAQ

Is it queueing theory or queuing theory? Both spellings are correct and name the same field. Kingman's 1961 paper writes "queueing time", while Little's proof from the same year has "Queuing Formula" in its title.

Does queueing theory need Poisson arrivals? Only for exact M/M/1 results. G/G/1 approximations such as Kingman's formula use the mean and variability of arrival gaps and service times, and Little's law needs no assumption about the distributions.

SLO vs SLA: the difference, SLIs and error budgets

Sun, 01 Nov 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

An SLA (service level agreement) is a contract with a customer that says what happens when a service misses a promised level, usually a service credit. An SLO (service level objective) is the internal target a team runs the service against; the SRE book advises setting it stricter than the SLA so that problems get fixed before the contract is breached. An SLI (service level indicator) is the measurement both are written in, such as the share of requests that succeed.

	SLI	SLO	SLA
What it is	A measurement	A target for an SLI	A contract built on SLOs
Set by	Engineering	Product owner and engineering	Business and legal teams
On a miss	Nothing by itself	Error budget spent	Service credits or penalties
Example	Checkout calls that succeed	99.95% over four weeks	99.9% monthly, or a credit

The examples are illustrations; the sources are Google's SRE book and workbook and the AWS compute SLA, checked on 1 November 2026.

What do SLI, SLO and SLA stand for?

Service level indicator, service level objective and service level agreement. The indicator is what you measure, the objective is the value you aim for, and the agreement is what you promise someone else, with consequences attached.

The SRE book chapter on Service Level Objectives defines an SLA as "an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain". The chapter's test is one question: what happens if the SLOs are not met? With no explicit consequence, you are almost certainly looking at an SLO.

The chapter adds that most people mean SLO when they say SLA, and that a real SLA violation might end in a breach-of-contract case. So when a meeting turns into an SLA vs SLO argument, first ask whether a consequence, usually money, is written into a contract.

The SRE book advises keeping a safety margin: "a tighter internal SLO than the SLO advertised to users gives you room to respond to chronic problems before they become visible externally." Its "Embracing Risk" chapter applies the same idea to a contract: a 99.9% external target backed by a stronger internal one.

What is an error budget?

It is 100% minus the SLO: the share of requests or time that may fail within the window. While budget remains, the team keeps shipping; once it is spent, reliability work comes first.

The SRE workbook chapter "Implementing SLOs" gives the arithmetic: with a 99.9% SLO, a service that receives 3 million requests over four weeks may fail 3,000 of them, so one outage with 1,500 errors spends half the budget. The book's "Embracing Risk" chapter explains the purpose: product management sets the SLO, monitoring measures uptime, and the gap is the unreliability left for the quarter. When that budget runs out, releases are halted or slowed.

How much downtime does 99.9% allow?

About 43 minutes in a 30-day window. Each extra nine divides the allowance by ten, so 99.99% leaves just over 4 minutes.

SLO	Error budget	Downtime per 30 days	Failures per million
99%	1%	7 h 12 min	10,000
99.5%	0.5%	3 h 36 min	5,000
99.9%	0.1%	43 min 12 s	1,000
99.95%	0.05%	21 min 36 s	500
99.99%	0.01%	4 min 19 s	100

Multiply the 43,200 minutes of a 30-day window by the error budget. Note the workbook's warning that each extra nine costs more while its value to users approaches zero.

What does a real SLA look like?

It names a measured percentage, a billing period, a credit schedule and what does not count. The AWS compute SLA is a short public example.

The Amazon Compute Service Level Agreement (last updated 25 May 2022; checked on 1 November 2026) commits to a Monthly Uptime Percentage of at least 99.99% per region for EC2 instances spread over two or more Availability Zones, and 99.5% for a single instance. Uptime is 100% minus the share of minutes when EC2 was unavailable. Under the Region-Level SLA, below 99.99% the credit is 10% of that month's EC2 bill in the region, below 99.0% it is 30%, and below 95.0% it is 100%; a single instance's 10% tier starts below 99.5%. Credits are applied against future payments, must be claimed through AWS Support and are the "sole and exclusive" remedy unless the customer agreement says otherwise.

Read against the downtime table, the Region-Level SLA's 10% tier spans roughly 4 minutes to 7 hours of unavailability in a 30-day month. An SLA caps what the provider owes; keeping outages from hurting your users is the job of your own objectives.

How do you choose SLIs for latency and availability?

Measure latency as a percentile and availability as good events divided by all events, as close to the user as you can. Averages and server-only numbers hide the requests people complain about.

Latency. The SRE book prefers percentiles to the mean, because an average can stay flat while a long tail gets much slower. The workbook writes latency SLIs as a ratio, such as calls completed in under 100 ms divided by all calls. So "99% of checkout calls under 400 ms" means "p99 of checkout under 400 ms" over the window, which carries straight into a test. Our tail latency article explains why p99 matters once a page calls several services.

Availability. Time-based availability counts minutes of uptime, as the AWS SLA does. Request-based availability counts successful requests: "Embracing Risk" describes a service with 2.5 million requests a day and a 99.99% daily target, which can return 250 errors and still meet it.

Where do SLOs go wrong?

Usually through too many objectives, 100% targets and indicators measured where users are not. Each one leaves a number nobody acts on, or one that looks fine while users suffer.

Too many SLOs. The SRE book advises as few as possible: if quoting an SLO can never win a conversation about priorities, it is probably not worth having.
100% targets. The workbook calls 100% the wrong target: components fail, change is the main source of outages, and a 100% SLO leaves time only to react.
Measuring at the server only. Server metrics miss problems users feel, such as a slow page script. The workbook suggests measuring at the load balancer or the client.

What does load testing have to do with SLOs?

An SLO gives a load test its pass/fail criteria, and a test at the expected peak shows whether the latency SLO holds before that traffic arrives. Production monitoring only tells you afterwards.

The SRE book notes that higher QPS often leads to larger latencies and that services commonly have a performance cliff beyond some load threshold. A load test can find that cliff if it runs at the expected peak with a realistic request mix and measures the SLI the SLO uses. Set each test limit a little tighter than the SLO, as our guide to load test pass/fail criteria describes. A run lasts minutes, so it cannot measure monthly availability.

LoadFocus, which publishes this site, grades load tests against pass/fail thresholds saved on the test: Max P95 and Max P99 response time, a maximum error rate and a minimum throughput, for cloud, JMeter and k6 tests. Its API monitoring has SLA / SLO reports that set an SLO target per monitor and show uptime, error budget consumed and an OK or Breached status.

FAQ

What does SLO stand for? Service level objective: a target value for a measured service level, such as 99.9% of requests succeeding over four weeks. It is set internally, by product and engineering, and carries no contractual penalty by itself.

Can you have an SLO without an SLA? Yes. The SRE book gives Google Search as an important service with no public SLA that is still managed with SLIs and SLOs.

Should an SLO ever be 100%? No. A 100% objective leaves no error budget, so every failure breaches it and no change is ever safe to release.

TPC-H: what the benchmark measures and how to run it

Sun, 01 Nov 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

TPC-H is the decision support benchmark of the Transaction Processing Performance Council (TPC): 22 business queries and two refresh functions run against eight tables that the TPC's dbgen program fills at a chosen scale factor, where scale factor 1 is about 1 GB. Its metric, QphH@Size, combines a single-stream power test with a multi-stream throughput test, and official results add a price per kQphH@Size. Only audited results published on tpc.org count as official; any TPC-H-style number published outside tpc.org counts as a derived workload under the TPC's fair use rules and may not be compared with official results.

We have not run the benchmark for this guide. The facts come from tpc.org, the specification revision 3.0.1 that tpc.org lists as current, and each tool's documentation, all checked on 1 November 2026. The judgements are ours.

	TPC-H	TPC-C	TPC-DS
Workload	Decision support: ad hoc queries plus data refreshes	OLTP: a wholesale supplier's orders	Decision support: queries plus data maintenance
Scope	8 tables, 22 queries, 2 refresh functions	9 table types, 5 transaction types	Snowflake schema, 7 fact tables, 99 queries
Primary metric	QphH@Size and $/kQphH@Size	tpmC	QphDS@SF and $/kQphDS@SF
Current version	3.0.1	5.11.0	4.0.0

What does TPC-H measure?

How fast a system answers complex analytical queries over a large dataset while new sales arrive and old ones are deleted. One specification defines the schema, the queries, the data and the rules.

The TPC-H page describes "a suite of business oriented ad-hoc queries and concurrent data modifications" (checked on 1 November 2026). The specification defines eight tables: PART, PARTSUPP, LINEITEM, ORDERS, CUSTOMER, SUPPLIER, NATION and REGION. All but the last two grow with the scale factor; LINEITEM, the largest, holds about 6,000,000 rows per unit. The refresh function RF1 inserts new rows into ORDERS and LINEITEM, and RF2 removes old sales (checked on 1 November 2026).

How is the TPC-H score calculated?

A run is a power test followed by a throughput test, and QphH@Size is the geometric mean of the two results. The price metric is the total system price divided by QphH@Size, times 1,000.

In the power test, a refresh stream runs RF1, a single query session runs all 22 queries once, then the refresh stream runs RF2. Power@Size is 3,600 times the scale factor divided by the geometric mean of those 24 timings. In the throughput test, several query streams run beside a refresh stream: at least 2 at scale factor 1, rising to 11 at scale factor 100,000. Throughput@Size is queries per hour across all streams, times the scale factor. A full test has two runs, and the sponsor reports the one with the lower QphH (all from the specification, checked on 1 November 2026).

The geometric mean weighs every query equally, so halving one query's time raises Power by only about 3%. An official result uses a scale factor from a fixed list: 1, 10, 30, 100 and so on up to 100,000.

Who publishes TPC-H results, and which ones count?

Vendors that sponsor a test publish audited results on tpc.org, and only those count as official. Any other run is a derived workload with its own rules.

The specification requires an independent audit by a TPC-certified auditor. The TPC's fair use quick reference allows the TPC name next to results only for official published results. Anything else needs the prefix "Derived from", may not use the TPC metric, must carry a disclaimer that it is not comparable to published results, should disclose its environment and deviations, and may not be compared with official results (checked on 1 November 2026).

How do you run TPC-H yourself?

Generate the data and query text, load the tables, then run the power test followed by the throughput test. You can use the TPC's own tools, a kit built for one database or an engine with a built-in generator.

The TPC's current specifications page offers a free tools package: you register, accept a licence and get the link by email. The specification says DBGen must produce the data and QGen the query text (checked on 1 November 2026).

For PostgreSQL, the wiki's TPC-H page gives no step-by-step procedure and points to DBT-3, a full kit that the wiki's DBT-3 page calls a fair usage implementation of the specification. HammerDB ships TPROC-H, which its documentation calls an open source workload derived from TPC-H and not comparable to published results. It runs on PostgreSQL, MySQL, Oracle and other engines, and the docs recommend comparing the geometric mean of query times instead of calculating QphH (all checked on 1 November 2026).

DuckDB has a tpch extension that, per its documentation, implements the data generator and the queries. These lines generate scale factor 1, run query 4 and list the 22 queries:

CALL dbgen(sf = 1);
PRAGMA tpch(4);
FROM tpch_queries();

The same page warns that the data differs slightly from the specification, that each query runs with fixed parameters and that dbgen does not drop existing tables. For fully compliant data it points to tpchgen-cli (checked on 1 November 2026).

How do you read TPC-H results honestly?

Compare results only at the same scale factor and benchmark version, and read each metric with its size suffix. A chart of single query times shows neither the power test nor the throughput test.

The results page groups results by database size and carries three notes: version 2 and version 3 results are not comparable, price/performance in different currencies should not be compared, and comparisons across database sizes are misleading (checked on 1 November 2026). The specification adds that results with different stream counts are comparable at the same database size.

For an informal run, look for what the fair use rules ask for: hardware and memory, changes to schema or query text, and whether refresh functions ran. A post that times the 22 queries once on a warm cache, with no refresh stream, approximates only the query half of a power test, though such a run still helps with regression tracking. On query rate against response time, see throughput vs latency.

When should you use something other than TPC-H?

When the workload is transactional, when you want a newer analytics benchmark, or when the question is what users feel. The benchmark models one analytical workload, without short transactions or an application layer.

Choose TPC-DS for a broader decision support test: its version 4.0.0 specification uses a snowflake schema with seven fact tables and 99 queries, and its overview page says version 4 results are not comparable with earlier versions. Choose TPC-C, or HammerDB's TPROC-C, for order-entry OLTP: the TPC-C page describes five transaction types measured in tpmC (all checked on 1 November 2026). For a quick check of one database, pgbench covers PostgreSQL and sysbench covers MySQL.

LoadFocus, which publishes this site, does not run TPC-H; its tests are aimed at web and API endpoints. It runs JMeter test plans and k6 scripts from 25 cloud regions, with no JDBC driver and no SQL module, so it cannot connect to the database. Choose a database benchmark tool such as DuckDB's extension, DBT-3 or HammerDB instead of LoadFocus for this job, and use LoadFocus for the web or API layer in front of the database, starting from its template for the endpoints behind heavy queries.

FAQ

What is the difference between TPC-H and TPC-DS? Both are TPC decision support benchmarks. TPC-H uses 8 tables and 22 queries and reports QphH@Size, while TPC-DS uses a snowflake schema with seven fact tables and 99 queries and reports QphDS@SF.

What does QphH@Size mean in TPC-H? It is the composite queries-per-hour metric: the geometric mean of the power test and throughput test results at one database size. The suffix names that size, for example QphH@1000GB, and results at different sizes should not be compared.

Can LoadFocus run TPC-H? No. LoadFocus runs JMeter test plans and k6 scripts aimed at web and API endpoints and has no SQL connection, so use a database benchmark tool for TPC-H and LoadFocus for the application layer in front of the database.

redis-benchmark: how to use it and read its numbers

Sat, 31 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

redis-benchmark is the benchmark utility that ships with Redis: it opens a number of client connections (50 by default), sends a fixed number of commands (100,000 by default) and reports requests per second and latency for each test. Run redis-benchmark -q -n 100000 against a running server, narrow it to the commands you care about with -t, and read the result as the speed of one Redis instance under synthetic commands. Your application adds its own network path, client library and command mix, so expect its figures to differ.

We ran it on 31 October 2026 on an Apple M1 Max laptop, Redis 7.2.6 server and client on one machine over loopback, persistence off. Each row is one run of 100,000 requests, with requests per second rounded to the nearest thousand; p50 is as printed.

Run	What changed	Requests per second	p50 (ms)
`-t set,get`	Nothing: 50 clients, 3-byte values	SET 113,000, GET 118,000	0.367, 0.351
`-t set,get -P 16`	16 commands per round trip	SET 1,010,000, GET 1,190,000	0.711, 0.583
`-t set,get -c 200`	200 clients	SET 117,000, GET 123,000	1.135, 1.071
`-t set -d 1024`	1,024-byte values	SET 114,000	0.367
`-t set -r 100000`	Random keys out of 100,000	SET 116,000	0.359
`script load "..."`	A custom command	106,000	0.415

How do you run redis-benchmark?

Point the tool at a running server with -h and -p; with no options it runs its default test suite against 127.0.0.1:6379. Add -t to pick tests and -q to print one line per test.

redis-benchmark -h 10.0.0.12 -p 6379 -c 50 -n 100000 \
  -t set,get -q

The official Redis benchmark page and the --help output of version 7.2.6 list these options, among others (checked on 31 October 2026):

-c parallel connections (default 50) and -n total requests (default 100,000).
-d size of SET and GET values in bytes (default 3).
-P commands per pipeline (default 1, no pipelining).
-t a comma-separated list of tests; -q quiet output; --csv CSV output.
-r random keys from a key space of that size. Without it, every command uses the same key.
-k 1 keeps connections alive (the default), -k 0 reconnects.
--threads turns on multi-thread mode; --cluster turns on cluster mode, where -c must be at least the number of nodes.

How do you read redis-benchmark output?

With -q you get one line per test: requests per second and the median latency. Without it, each test prints its settings, a latency distribution by percentile and a summary with the average, minimum, p50, p95, p99 and maximum latency.

The header also prints the server's save and appendonly settings, which tell you whether persistence was part of the test. Our full GET run read 119,617 requests per second, an average of 0.352 ms, p95 0.559 ms, p99 0.623 ms and a maximum of 0.759 ms. A request that makes many Redis calls is likelier to meet a slow one, so read p99 too; see our tail latency article.

--csv prints the summary as one row per test. Repeat runs before you trust a small difference: two runs of the same SET test gave us about 113,000 and 125,000 requests per second.

Why does pipelining change the numbers so much?

Without -P, each client waits for a reply before sending its next command, so every command pays a full round trip. With -P 16 each client sends 16 at once, and in our run SET rose from about 113,000 to 1,010,000 requests per second and GET from 118,000 to 1,190,000.

The official page calls pipelining realistic, since many Redis applications use it, and advises a pipeline size close to the average your application can use. Our median latency rose at the same time, to 0.711 and 0.583 ms.

What happens when you add more clients?

Once the bottleneck is saturated, more clients only add waiting. On one laptop that can be the server or the single-threaded benchmark client.

Going from 50 to 200 clients left throughput almost flat (about 117,000 SET and 123,000 GET requests per second) while the median latency roughly tripled, to 1.135 and 1.071 ms. That is Little's law at work. Each client keeps at most one command in flight, so -c fixes how many are in the system, and if throughput cannot grow, the time each spends there must. In our full GET run, 50 clients ÷ 119,617 requests per second gives 0.42 ms per client loop, a little above the measured average of 0.352 ms. With 200 clients the same calculation gives 1.63 ms.

This also makes the tool's load a closed workload: when Redis stalls, the clients wait, so the commands your application would have sent meanwhile never appear. Neither the official page nor --help lists a fixed-rate option, so read its percentiles with coordinated omission in mind.

How do you benchmark your own command or a Lua script?

Put the command after the options. The tool then runs only that command and ignores -t.

redis-benchmark -n 100000 -q \
  script load "redis.call('set','foo','bar')"
redis-benchmark -r 10000 -n 10000 \
  eval 'return redis.call("ping")' 0

The first line is the official page's example, which ran at about 106,000 requests per second on our laptop. SCRIPT LOAD only loads the script, so use eval, as in the second line from --help, to time running one. With -r, the help says __rand_int__ in an argument becomes a random 12-digit number in that range, and in cluster mode the key must contain a {tag} to reach the right node.

Why does your application see different numbers?

Your application measures its whole path to Redis: network, client library, serialization and its own command mix. The official page lists pitfalls to check before you compare.

A loop of synchronous commands mostly measures the network and the client library.
With client and server on one box, the CPU limits the benchmark. Unix domain sockets can reach around 50% more throughput than TCP loopback, on Linux for instance.
4 KB strings at 100,000 queries per second need about 3.2 Gbit/s, more than a 1 Gbit/s link carries.
To compare Redis with a transactional database, turn on AOF and choose a suitable fsync policy.
Laptops vary their CPU frequency, which hurts repeatability.
INFO at intervals is probably fine during a run, while MONITOR slows Redis significantly. Our publisher's roundup of Redis monitoring tools covers the metrics worth watching.

Are these laptop numbers Redis's real capacity?

No. They show one laptop talking to itself, with no network, 3-byte values in most runs, no persistence and one run per row.

The table shows how each option moves the figures. Capacity depends on your server, network, value sizes and persistence settings.

When is redis-benchmark the wrong tool?

When you want to know what users feel. It sends synthetic commands to Redis alone, while a page's response time also includes the application code and its other calls.

For that, load the application or API that uses Redis and watch Redis while the test runs. LoadFocus, which publishes this site, works at that layer: it runs JMeter test plans and k6 scripts from 25 cloud regions against your web app or API. It does not run redis-benchmark.

For more control over Redis traffic, the memtier_benchmark repository lists multi-threaded clients, a read:write ratio, random or sequential key patterns, Redis Cluster and TLS (checked on 31 October 2026).

FAQ

Is redis-benchmark designed to make Redis look fast? No. The official Redis page calls that a common misconception: by default the tool keeps at most one query pending per connection, with no pipelining, and a client that pipelines can generate more throughput.

How do you fill a Redis database with redis-benchmark? Run only the SET test with random keys, as the tool's help shows: -t set -n 1000000 -r 100000000 fills about one million keys. Random keys can repeat, so the final count ends a little lower.

Can LoadFocus run redis-benchmark? No. LoadFocus does not run redis-benchmark. Its JMeter test plans and k6 scripts are aimed at the web app or API in front of Redis.

How to use sysbench for CPU, disk and MySQL benchmarks

Sat, 31 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

sysbench is an open source, multi-threaded benchmark tool built on LuaJIT that ships with CPU, memory, file I/O, thread and mutex tests, plus OLTP scripts for MySQL and PostgreSQL. Tests that need data follow the same lifecycle: prepare creates the test files or tables, run executes the benchmark for a set time or number of events, and cleanup removes the data; the CPU and memory tests need only run. It measures one machine or one database server, so it cannot tell you how your application behaves under real user traffic.

We have not run sysbench for this guide. The facts come from its README and source code on GitHub's master branch, checked on 31 October 2026, and the advice is ours.

Topic	sysbench
Licence	GPLv2
Written in	C, with LuaJIT for scripts
Latest release	1.0.20 (24 April 2020)
Latest commit	9 March 2025
Built-in tests	`cpu`, `memory`, `fileio`, `threads`, `mutex`
Database scripts	Eight `oltp_` scripts, `bulk_insert`, two `select_random_`
Database drivers	MySQL (default), PostgreSQL
Platforms	Linux, macOS; Windows through WSL

Is sysbench still maintained?

Only lightly. The latest release, 1.0.20, came out in April 2020; the master branch has had occasional commits since, none of them released, the most recent in March 2025.

The 1.0.20 release is dated 24 April 2020. The commit history shows changes such as a uniform default for --rand-type (2020), an upstream LuaJIT (2023) and MariaDB client support (2023), then four build commits on 9 March 2025 (checked on 31 October 2026). The README's package repositories are updated on each release, so those changes need a build from source.

How does a sysbench run work?

You name a test and a command: sysbench [options] [testname] [command]. The test is a built-in, a bundled Lua script or your own script, and the command is usually prepare, run, cleanup or help.

The README says prepare creates files for fileio or fills the database for the OLTP scripts, cleanup removes that data, and sysbench <testname> help lists a test's options. A disk test:

sysbench fileio --file-total-size=8G prepare
sysbench fileio --file-total-size=8G \
  --file-test-mode=rndrw --time=300 run
sysbench fileio --file-total-size=8G cleanup

The fileio source lists the modes seqwr, seqrewr, seqrd, rndrd, rndwr and rndrw, with 2 GB of files by default. Files that fit in memory can be served from the page cache, so make the set larger than RAM to measure the disk.

How do you benchmark CPU and memory with sysbench?

Use sysbench cpu run and sysbench memory run. The CPU test reports events per second, where one event checks every number up to a limit for primality, and the memory test reports operations and MiB transferred per second.

sysbench cpu --cpu-max-prime=20000 \
  --threads=4 --time=60 run

The CPU test sets --cpu-max-prime to 10000 by default, and the memory test takes a block size, a total size, read or write operations and sequential or random access (checked on 31 October 2026). Compare only runs with identical settings.

How do you benchmark MySQL or PostgreSQL with sysbench?

Pick a bundled OLTP script, fill the tables with prepare, run transactions from several threads with run, then drop the tables with cleanup. MySQL is the default driver; PostgreSQL needs --db-driver=pgsql and a build with PostgreSQL support.

sysbench oltp_read_write --mysql-host=10.0.0.5 \
  --mysql-user=sbtest --mysql-password=secret \
  --tables=10 --table-size=1000000 prepare
sysbench oltp_read_write --mysql-host=10.0.0.5 \
  --mysql-user=sbtest --mysql-password=secret \
  --tables=10 --table-size=1000000 \
  --threads=16 --time=300 --report-interval=10 run

The shared OLTP script creates one table of 10,000 rows unless you pass --tables and --table-size. In oltp_read_write, each transaction runs 10 point selects, four range queries, two updates and a delete plus insert inside BEGIN and COMMIT (checked on 31 October 2026). Make the data larger than the server's buffer pool if you want disk reads in the result.

Which sysbench options matter most?

--threads and --time set the size and length of a run, --rate sets a target event rate, and --report-interval, --percentile and --histogram decide what you see.

--threads: worker threads, 1 by default.
--time: seconds, 10 by default, 0 for no limit. --events caps the event count, and both limits apply, so pair it with --time=0.
--warmup-time: runs with statistics off first, so cold caches do not skew the result. Master only: the 1.0.20 release does not have it.
--rate: average events per second across all threads; 0, the default, means as fast as possible.
--report-interval: per-interval figures every N seconds, to expose stalls.
--percentile: the latency percentile to report, 95 by default.
--histogram: a latency histogram at the end, defined in sb_logger.c.

Does sysbench use a closed or an open workload model?

Closed by default: a fixed number of threads each start the next event when the previous one ends, so a slow server receives fewer events. With --rate, events arrive on their own schedule, which is closer to an open model.

Our article on the open vs closed workload model explains why that changes what a benchmark measures. In the default mode, a stall also delays the events that would have started during it, and those delays never reach the latency figures: coordinated omission. The main source file shows that --rate queues events at exponentially distributed intervals, adds each event's queue wait to its latency, and stops with an "event queue is full" error when the threads cannot keep up (checked on 31 October 2026).

How do you read sysbench output?

Start with throughput: events per second, or transactions and queries per second for the OLTP scripts. Then read the latency block in milliseconds: minimum, average, maximum, the chosen percentile (95th by default) and the sum.

The 1.0.20 release prints a "General statistics" block with the total time and total number of events in place of "Throughput". The field names below come from the database report code and the main source file on master, with the values left out:

SQL statistics:
    queries performed: read, write, other, total
    transactions, queries (N per sec.)
    ignored errors, reconnects (N per sec.)
Throughput:
    events/s (eps), time elapsed, total number of events
Latency (ms):
    min, avg, max, 95th percentile, sum
Threads fairness:
    events, execution time (avg/stddev)

A 95th percentile ignores the slowest one in twenty events. Set --percentile=99 or add --histogram when the tail matters; our guide to tail latency explains why.

When is sysbench the wrong tool?

When your question is about your application. A CPU score shows how fast one machine checks primes, and an OLTP run measures the database server with sysbench's own tables and queries, without your schema, connection pool or network path.

Choose pgbench for PostgreSQL-native tests: the PostgreSQL documentation describes a default scenario loosely based on TPC-B, plus custom script files (checked on 31 October 2026).
Choose HammerDB for TPC-C-style tests: its documentation covers the TPROC-C workload, derived from TPC-C, for SQL Server, Db2, Oracle, PostgreSQL, MariaDB and MySQL (checked on 31 October 2026).
Choose JMeter with its JDBC Request sampler to mix SQL and HTTP in one test plan; the database test plan guide says the driver .jar goes into JMeter's lib folder.

LoadFocus, which publishes this site, does not run sysbench and cannot connect to MySQL or PostgreSQL itself. It runs JMeter test plans and k6 scripts from 25 cloud regions against the web or API layer in front of the database. A useful pairing is a sysbench run to size the database server, then a load test of the application to see whether real requests reach that limit.

FAQ

Is sysbench still maintained? Lightly. The latest release is 1.0.20 from 24 April 2020. The master branch has received occasional commits since, none of them in a release, the most recent on 9 March 2025.

What does the 95th percentile in sysbench output mean? It is the latency, in milliseconds, within which 95% of events finished. 95 is the default of the --percentile option, and you can set another value such as 99.

Can sysbench benchmark PostgreSQL? Yes. The bundled OLTP scripts run against PostgreSQL with --db-driver=pgsql when sysbench was built with PostgreSQL support.

Can LoadFocus run sysbench? No. LoadFocus runs JMeter test plans and k6 scripts against websites and APIs from 25 cloud regions, and it does not connect to MySQL or PostgreSQL directly.

YCSB: workloads A-F, how to run it and its limits

Sat, 31 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

YCSB, the Yahoo! Cloud Serving Benchmark, is an open source Java tool that loads synthetic records into a datastore and then runs a fixed mix of reads, updates, inserts and scans against them while it records throughput and latency per operation. Its six core workloads, A to F, are standard mixes, so the same test can run against Cassandra, MongoDB, Redis, HBase and dozens of other stores through bindings. It measures the datastore on its own, so it says nothing about your application's real queries or the latency your users see.

We have not run YCSB for this article. The facts come from its GitHub repository, wiki and source code, checked on 31 October 2026, and the judgements are ours.

Workload	Operation mix	Key distribution	Example given in the wiki
A: update heavy	50% read, 50% update	zipfian	Session store recording recent actions
B: read mostly	95% read, 5% update	zipfian	Photo tagging
C: read only	100% read	zipfian	User profile cache
D: read latest	95% read, 5% insert	latest	User status updates
E: short ranges	95% scan, 5% insert	zipfian (scans of up to 100 records)	Threaded conversations
F: read-modify-write	50% read, 50% read-modify-write	zipfian	User database

The mixes come from the files in the workloads directory and the examples from the Core Workloads page (checked on 31 October 2026). Each file ships with only 1,000 records and 1,000 operations, enough to debug a setup and far too few for a benchmark.

What is YCSB, and is it still maintained?

It is a workload generator plus a set of standard workloads for comparing key-value and "cloud serving" stores, released as open source by Yahoo! in 2010 under the Apache 2.0 licence. The last release dates from October 2019, and the code still receives small commits.

The papers page lists the original paper, "Benchmarking Cloud Serving Systems with YCSB" by Cooper, Silberstein, Tam, Ramakrishnan and Sears (ACM SoCC 2010). The 0.17.0 release is dated 6 October 2019 and no later tag exists, while the commit history shows 2026 work such as an option to keep going when an insert fails and a Cloud Spanner change on 20 October (checked on 31 October 2026). Those changes ship only in a build from source, which needs Maven 3.

How do you run a YCSB benchmark?

In two phases that share the same properties: load inserts recordcount records, then run executes operationcount operations from the workload's mix. You name the binding for your database and pass the workload file with -P.

bin/ycsb load mongodb -s -P workloads/workloada \
  -p recordcount=1000000 \
  -p mongodb.url=mongodb://db1:27017/ycsb
bin/ycsb run mongodb -s -P workloads/workloada \
  -p recordcount=1000000 -p operationcount=5000000 \
  -p mongodb.url=mongodb://db1:27017/ycsb \
  -threads 32 -target 20000

The Running a Workload page covers the details that trip people up (checked on 31 October 2026). The client never creates tables, so you first map usertable to whatever your store needs, such as a table in MySQL or a keyspace in Cassandra. Later -P files override earlier ones, and -p overrides them all. The run phase needs the same recordcount as the load, because it picks keys from the full dataset. -s prints a status line every 10 seconds. Because D and E insert records, the Core Workloads page suggests loading once with workload A's file, running A, B, C, F and D, then deleting the data and reloading for E.

What do -threads and -target control?

-threads sets how many client threads send operations (one by default), and -target caps the total operations per second. Without a target each thread sends its next operation as soon as the last one returns, so a slower database simply receives less load.

That is a closed loop, and our article on the open vs closed workload model explains what it hides. The wiki's example of 10 threads and a target of 100 operations per second holds only while average latency stays under 100 ms; with too few threads the achieved rate falls below the target. That is Little's law at work: threads divided by average latency caps the throughput. Running several targets and noting the latency at each gives you the curve from our guide to throughput vs latency.

How do you read YCSB output?

At the end of each phase the client prints an [OVERALL] block with run time and throughput, then one block per operation type with the count, average, minimum, maximum and percentile latencies, plus a count per return code.

These field names come from the 0.17.0 source code; the values are placeholders:

[OVERALL], RunTime(ms), <total>
[OVERALL], Throughput(ops/sec), <rate>
[READ], Operations, <count>
[READ], AverageLatency(us), <value>
[READ], 95thPercentileLatency(us), <value>
[READ], 99thPercentileLatency(us), <value>
[READ], Return=OK, <count>

The wiki's sample output shows milliseconds, but the default HdrHistogram measurement in both 0.17.0 and the current source reports microseconds, (us). The Core Properties page and release 0.17.0 set the default percentiles to 95 and 99, changeable with hdrhistogram.percentiles; since April 2025 the master branch adds 50, so a source build also prints 50thPercentileLatency(us) (checked on 31 October 2026). The README points you at P99 and beyond, warns that percentiles cannot be averaged, and suggests saving HDR histogram files with hdrhistogram.fileoutput=true and merging them when several clients run.

Does YCSB account for coordinated omission?

Only when you ask for it, and only with -target set. By default each latency is timed from the moment the operation was actually sent, so a stall that delays the operations queued behind it barely shows.

The measurements source has a measurement.interval property that takes op (the default), intended or both (checked on 31 October 2026). Under a target, each thread records the scheduled start of every operation; intended times latency from that point, and both adds Intended- blocks such as [Intended-READ] next to the usual ones. The wiki does not list measurement.interval, so check the source you build. Our guide to coordinated omission explains why the scheduled start gives the honest number.

Which databases can YCSB benchmark?

Any store with a binding. The build file on the master branch lists 46 binding modules, among them Cassandra, MongoDB, Redis, HBase, DynamoDB, Cloud Spanner, RocksDB and a generic JDBC binding.

The list is in the pom.xml, and each binding directory has its own README with connection properties, such as mongodb.url for the MongoDB binding (checked on 31 October 2026). The Running a Workload page says one client machine suits clusters of about 10 machines; beyond that you start several clients, and the Running a Workload in Parallel page splits the load between them with insertstart and insertcount.

When is YCSB the wrong tool?

When your question is about your application. The core workload writes synthetic records, 10 fields with a length of 100 each by default, through simple key operations, so it cannot show how your own queries, indexes or API code behave under load.

For relational OLTP, pgbench runs a TPC-B-like script against PostgreSQL and accepts your own scripts, and sysbench bundles OLTP-style Lua benchmarks (both checked on 31 October 2026).
For latency as users see it, load the HTTP or API layer in front of the datastore. LoadFocus, which publishes this site, runs JMeter test plans and k6 scripts from 25 cloud regions against web and API endpoints. It does not run YCSB.

FAQ

What are the YCSB core workloads? Six standard mixes: A is 50% reads and 50% updates, B is 95% reads, C is read only, D favours the most recently inserted records, E runs short scans, and F does read-modify-write.

What is the difference between ycsb load and ycsb run? The load phase inserts the number of records set by recordcount. The run phase then executes operationcount operations from the workload's mix against those records and reports throughput and latency.

Is YCSB still maintained? The last release, 0.17.0, came out on 6 October 2019. The repository still receives occasional commits, the most recent on 20 October 2026, so newer fixes require a build from source.

Can LoadFocus run YCSB benchmarks? No. LoadFocus does not run YCSB. It runs JMeter test plans and k6 scripts aimed at web and API endpoints, the layer in front of the datastore.

pgbench: how to benchmark and load test PostgreSQL

Sat, 31 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

pgbench is the benchmark program that ships with PostgreSQL: it runs the same SQL transaction over and over in several concurrent database sessions and reports transactions per second and latency. You create its test tables with pgbench -i -s 50 mydb, then run something like pgbench -c 10 -j 2 -T 300 mydb for ten clients on two threads for five minutes. It measures the database alone, so it says nothing about what users feel through the application in front of it.

We have not run pgbench for this guide. The facts come from its page in the PostgreSQL 18 documentation, the version postgresql.org marks as current, checked on 31 October 2026, and the judgements are ours.

Option	What it does
`-i` / `-s`	Initialize the test tables / scale factor (100,000 accounts per unit)
`-c`	Clients, that is concurrent database sessions (default 1)
`-j`	Worker threads inside pgbench (default 1)
`-T` / `-t`	Run for N seconds, or N transactions per client (default 10)
`-b` / `-f`	A built-in script or your own file, with an optional `@weight`
`-P`	Progress report every N seconds
`-R` / `-L`	Target rate in tps / latency limit in ms
`-r` / `-l`	Per-statement latencies / a log line per transaction

What does pgbench measure?

It runs a transaction script again and again in a set number of database sessions and reports the average transaction rate and latency. The default script is loosely based on the TPC-B banking benchmark.

The pgbench documentation says the default scenario is inspired by TPC-B and is "not actually TPC-B, hence the name": the output labels it TPC-B (sort of) (checked on 31 October 2026).

How do you set up a pgbench database?

Run pgbench -i against an existing database, with -s for the scale factor. It creates four tables and drops any existing tables with the same names, so give it a database of its own.

At scale 1 the tables hold 1 branch, 10 tellers, 100,000 accounts and an empty history table, and -s multiplies those rows: -s 100 gives 10,000,000 rows in pgbench_accounts. The documentation gives no size on disk, so measure it after initializing (checked on 31 October 2026).

Which built-in scripts does pgbench include?

Three: tpcb-like (the default), simple-update and select-only. You pick one with -b, or with the shortcuts -N and -S.

The default runs seven commands per transaction: BEGIN, an update and a select on a random account, teller and branch updates, a history insert, and END. simple-update drops the teller and branch updates, which avoids contention on those small tables and, in the documentation's words, makes the test "even less like TPC-B". select-only issues only the select.

How do you write a custom pgbench script?

Put your SQL in a file, set variables with \set and the random functions, and pass the file with -f. Each run of the file counts as one transaction.

\set aid random(1, 100000 * :scale)
\set delta random(-5000, 5000)
BEGIN;
UPDATE pgbench_accounts SET abalance = abalance + :delta
  WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
END;

A weight after @ sets how often each script is picked: -f update.sql@9 -b select-only@1 picks the file about nine times in ten. The documentation also lists random_exponential, random_gaussian and random_zipfian, which can make a few rows hot. \sleep adds a pause, the database version of think time. When you run only your own files, pass the database's scale with -s, or :scale stays 1 and the script above touches only the first 100,000 accounts. With your own tables, leave out -i and add -n so pgbench does not vacuum the standard ones (checked on 31 October 2026).

Which options make pgbench behave like a load test?

Concurrency comes from -c and -j, duration from -T, and the shape of the load from -R. Without -R, each client starts its next transaction as soon as the last one ends.

That default is a closed model, explained in our article on the open vs closed workload model. With -R, pgbench starts transactions on a Poisson-distributed schedule that moves forward from the client's start time, whatever happened to earlier transactions. Reported latency then counts from each scheduled start, so it includes the schedule lag, and the average and maximum lag are shown separately: the correction our coordinated omission article describes. The client count stays fixed, so a lag that keeps growing means the system cannot process that rate with the chosen clients and threads: lower -R or add clients.

--latency-limit (-L) counts transactions slower than a limit in milliseconds as late; with -R it also skips transactions already too far behind schedule and reports them as skipped. -M prepared uses prepared statements, which the documentation says runs faster than the other modes (checked on 31 October 2026).

How do you read pgbench output?

The summary lists the settings, transactions processed and failed, average latency, initial connection time and tps. The shape below comes from the documentation, numbers left out:

transaction type: <builtin: TPC-B (sort of)>
number of transactions actually processed: ...
number of failed transactions: ... (...%)
latency average = ... ms
latency stddev = ... ms   (only with -P, -R or -L)
initial connection time = ... ms
tps = ... (without initial connection time)

Failed transactions are serialization or deadlock errors, which --max-tries can retry. Other SQL errors abort that client and make the exit status 2. The summary has no percentiles, so for tail latency log each transaction with -l, which records its time in microseconds, and compute p95 or p99 yourself.

Unless you pass -P, -R or -L, the summary's average latency is not measured: pgbench derives it from run time and client count, Little's law run backwards, failures included. Add -P when latency matters.

How do you get pgbench numbers you can trust?

Run for minutes, repeat, keep the scale factor at least as large as the client count, and watch vacuum. If pgbench becomes the bottleneck, move it to another machine.

The first guideline in the documentation's "Good Practices" section is "never believe any test that runs for only a few seconds" and adds that reproducible numbers can take hours (checked on 31 October 2026). With the default script, a scale below the client count mostly measures update contention, because pgbench_branches has only -s rows and every transaction updates one. Dead rows and autovacuum shift results between runs. With many sessions pgbench can saturate its own host: run it on a separate machine with low network latency, or spread several instances over several machines.

When is pgbench the wrong tool?

When the question is about users, another database or a standard OLTP mix. pgbench measures PostgreSQL alone, without your connection pool, ORM, cache or HTTP layer.

To see what users feel, load the HTTP or API layer in front of it. LoadFocus, which publishes this site, does not run pgbench and cannot connect to PostgreSQL directly: it runs JMeter test plans and k6 scripts from 25 cloud regions against web and API endpoints, and its template for the endpoints behind heavy queries is a starting point.

Choose sysbench instead for MySQL: its repository describes a scriptable multi-threaded benchmark tool with OLTP-like database tests. Choose HammerDB for a TPC-C style mix: its documentation describes TPROC-C, a workload derived from TPC-C, which it runs on PostgreSQL and other engines. Choose JMeter with its JDBC Request sampler to mix SQL and HTTP in one test plan on your own machines (all checked on 31 October 2026).

FAQ

What does the -s option do in pgbench? It sets the scale factor at initialization. Each unit adds 100,000 rows to pgbench_accounts, so -s 100 creates 10,000,000 rows. With the default script, keep it at least as large as your client count.

What is the difference between -c and -j in pgbench? The -c option sets the number of simulated clients, each a concurrent database session. The -j option sets how many worker threads inside pgbench share those clients.

Can LoadFocus run pgbench? No. LoadFocus runs JMeter test plans and k6 scripts and has no direct PostgreSQL connection, so with it you load the web or API layer that sits in front of the database.

NBomber load testing: a review of the .NET framework

Wed, 21 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

NBomber is a load testing framework for .NET: you write each test as a C# or F# scenario, add it to an ordinary project and run it like any other program. NBomber load testing suits teams that already build in .NET and want their load tests in the same language, the same repository and the same test runner. NBomber is free for personal and academic use only; a company needs a paid licence, and that one fact decides more evaluations than any feature.

We have not run NBomber for this review. The facts come from NBomber's documentation, pricing page, GitHub repository and NuGet package, checked on 21 October 2026, and the judgements are ours.

Topic	NBomber
Licence	Free for personal and academic use; paid for organisations
Price for a company	Business $1,500 a year, Enterprise $2,500 a year
Current release	6.7.1 (16 October 2026)
Runtime	.NET Standard 2.1, so .NET Core 3.0 or later
Tests written as	C# or F# scenarios
Load model	Inject (arrival rate) or constant copies
Pass/fail checks	Thresholds during the run, asserts on final stats
Scaling out	NBomber Cluster, Enterprise licence

Who makes NBomber, and is it free?

NBomber LLC develops it, with the source on GitHub. It is free for personal, hobby and academic use, and the licence page states plainly that NBomber is not free for organisational use.

The licence page counts benchmarks, tutorials and review content as personal use. Inside a company you need a Business or Enterprise licence, issued per organisation rather than per developer seat, and with no limit on instances or parallel runs. The pricing page lists Business at $1,500 a year and Enterprise at $2,500 a year, both with a 30-day money-back guarantee. The licence key goes into the runner with .WithLicense(...) or a --license argument, and NBomber needs internet access to validate it, with offline validation available on request by email (checked on 21 October 2026).

The latest release is v6.7.1, published on 16 October 2026, after 6.7.0 four days earlier (checked on 21 October 2026).

How does an NBomber scenario work?

A scenario is a name plus an async function that NBomber times on every call. You attach load simulations and thresholds to it, register it with NBomberRunner and run the program.

This test skips the warm-up, ramps the arrival rate to 50 new scenario copies a second over two minutes, holds it for ten minutes, and exits with code 1 if 1% or more of requests failed. It uses the NBomber.Http plugin for the request:

var http = Http.CreateDefaultClient();

var scenario = Scenario.Create("browse", async context =>
{
    var request = Http.CreateRequest("GET",
        "https://staging.example.com/products");
    return await Http.Send(http, request);
})
.WithoutWarmUp()
.WithLoadSimulations(
    Simulation.RampingInject(rate: 50,
        interval: TimeSpan.FromSeconds(1),
        during: TimeSpan.FromMinutes(2)),
    Simulation.Inject(rate: 50,
        interval: TimeSpan.FromSeconds(1),
        during: TimeSpan.FromMinutes(10)))
.WithThresholds(
    Threshold.Create(s => s.Fail.Request.Percent < 1));

var result = NBomberRunner
    .RegisterScenarios(scenario).Run();
return result.Thresholds.Any(t => t.IsFailed) ? 1 : 0;

The HTTP client page recommends Http.CreateDefaultClient() over a plain new HttpClient() and says the plugin tracks status codes and data transfer (checked on 21 October 2026). A threshold on its own only records the failure in the report, so the last line turns it into an exit code a pipeline can read. Without .WithoutWarmUp(), NBomber first runs a 30-second warm-up with the scenario's own simulations, per the scenario page (checked on 21 October 2026). Because the scenario is plain C#, anything a .NET library can call can be load tested the same way.

What load models does NBomber offer?

Both kinds. The inject simulations start new copies at a set rate whatever earlier copies are doing, which is an open model; the constant simulations keep a fixed number of copies running, which is a closed one.

The load simulation page lists them (checked on 21 October 2026):

Open: Inject, RampingInject, InjectRandom and IterationsForInject.
Closed: KeepConstant, RampingConstant and IterationsForConstant.
Timing: Pause, which delays a scenario's start or pauses it for a set time.

Our article on the open vs closed workload model explains which one matches your traffic. A public website usually behaves like the open model; a fixed pool of internal clients, like the closed one.

How do thresholds and pass/fail checks work?

Runtime thresholds check the running statistics, every 5 seconds by default, and appear in the HTML report. Asserts, written in xUnit or NUnit, check the final statistics after the run, and a failed assert ends the run with a failed status.

The thresholds page shows checks on error percentage, status codes, latency percentiles (p50, p75, p95, p99 and max), requests per second and bytes transferred. A threshold does not stop the test by default; abortWhenErrorCount makes it stop after a set number of failed checks (checked on 21 October 2026). The page does not state an exit code, which is why the sample above returns one from the result. Our guide to load test pass/fail criteria covers which limits to set.

Which protocols can NBomber test?

Any protocol a .NET library can speak, because NBomber only times the code you give it. Ready-made plugins cover HTTP, WebSockets, MQTT and AMQP, plus a WebBrowser option.

The overview page describes the framework as having no dependency on a protocol, and the pricing page lists those plugins in every plan, including the free one. That also covers databases and message queues you call through their own .NET clients (checked on 21 October 2026). The trade-off is that outside the plugins you write and maintain the client code yourself.

How do you scale out an NBomber test?

With NBomber Cluster: one Coordinator orchestrates the run and Agents execute the scenarios, all talking through a NATS message broker. Cluster is part of the Enterprise licence.

The cluster overview explains the roles, and the licence page says developers can try it locally with a Local Dev Cluster before buying; the Local Dev Cluster page limits it to 1 Coordinator and 2 Agents and stops each run after one minute (checked on 21 October 2026). NBomber Studio's Kubernetes integration, also Enterprise-only, provisions NATS and the Agent pods for you. Without Cluster, one NBomber process runs on one machine; the system requirements page suggests 4 to 8 CPU cores for high load.

LoadFocus, which publishes this site, does not run NBomber tests. It runs JMeter test plans and k6 scripts from 25 cloud regions.

What does NBomber do well, and where are its limits?

It is strong for .NET teams: tests are C# or F# code, any .NET client becomes a load test, and both load models are built in. Its limits are the company licence, a single-language ecosystem and paid distributed runs.

Tests live next to the application code, with the same IDE, packages and reviews.
Thresholds are built in, and asserts run inside the xUnit or NUnit tests you already have.
Commercial use needs a licence, even for a single developer.
Distributed runs need Enterprise and a NATS broker.
Non-.NET testers cannot pick it up without learning C# or F#.

When should you choose k6, Gatling or JMeter instead?

Choose another tool when your team does not write .NET, or when the licence cost does not fit.

Choose k6 instead when you want JavaScript scripts and an open source tool with no licence fee for company use: its repository is AGPL-3.0 (checked on 21 October 2026). LoadFocus runs k6 scripts from 25 cloud regions, though it runs arrival-rate scenarios with a fixed number of users. Choose Gatling instead for a code-first JVM tool; our Gatling review covers it. Choose JMeter instead when testers prefer a GUI or you need JDBC, JMS or LDAP samplers out of the box. The open source load testing tools overview sets those side by side.

FAQ

Is NBomber free for commercial use? No. NBomber is free for personal and academic use only. A company needs a Business licence at $1,500 a year or an Enterprise licence at $2,500 a year.

Which .NET versions does NBomber support? It targets .NET Standard 2.1, so it runs on .NET Core 3.0 and every later .NET version. The documentation recommends .NET 10.

Can LoadFocus run NBomber tests? No. LoadFocus runs JMeter test plans and k6 scripts. To run NBomber on more than one machine, use NBomber Cluster, which needs an Enterprise licence.

Tsung load testing: a review of the Erlang tool

Wed, 21 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Tsung is an open source load testing tool written in Erlang that describes each test in an XML file and can spread the load over many client machines. Tsung load testing still makes sense for protocols few tools cover, such as XMPP, LDAP or PostgreSQL at the wire level, and for very large user counts on modest hardware. Tsung's last release came out in March 2023, so for new HTTP or API work most teams are better served by k6, Gatling or JMeter.

We have not run Tsung for this review. The facts come from Tsung's documentation and GitHub repository, checked on 21 October 2026, and the judgements are ours.

Topic	Tsung
Licence	GPL-2.0
Written in	Erlang (needs Erlang/OTP R16B03 or later)
Latest release	1.8.0 (2 March 2023)
Tests written as	XML configuration files
Load model	New users per arrival phase (open)
Protocols	HTTP, WebDAV, XMPP, PostgreSQL, MySQL, LDAP, WebSocket, AMQP, MQTT, raw TCP/UDP (some experimental)
Scaling out	Client machines over passwordless SSH
Reports	Live dashboard on port 8091, `tsung_stats.pl` HTML report

Who maintains Tsung, and is it still active?

ProcessOne hosts the project on GitHub under GPL-2.0. It is maintained but quiet: the last release, 1.8.0, is from March 2023, and changes since then are occasional merged pull requests.

The repository shows its most recent commit on 2 March 2026, a pull request that replaces deprecated calls, and the v1.8.0 tag is dated 2 March 2023 (checked on 21 October 2026). The documentation still carries the 1.7.0 title, so check the repository when a detail looks out of date. A quiet project also means fewer answers when something goes wrong.

How does a Tsung test work?

A test is one XML file with four main parts: the client machines, the target servers, the load phases and the user sessions. Each new user picks a session by its probability and runs its requests in order.

This fragment starts five new users a second for ten minutes, and each user loads a product list, pauses about two seconds and posts to a cart:

<load>
  <arrivalphase phase="1" duration="10" unit="minute">
    <users arrivalrate="5" unit="second"/>
  </arrivalphase>
</load>
<sessions>
  <session name="browse" probability="100"
           type="ts_http">
    <request><http url="/products" method="GET"/></request>
    <thinktime value="2" random="true"/>
    <request><http url="/cart" method="POST"
                   contents="sku=A-100"/></request>
  </session>
</sessions>

The sessions page says the session probabilities must add up to 100 (or, since 1.5.0, you can use weights), and that a random think time follows an exponential distribution around its value unless you give a min and max range (checked on 21 October 2026). For HTTP and PostgreSQL there is a proxy recorder that turns a browser or client session into this XML.

What load model does Tsung use?

An open model. Each arrival phase starts new users at a set interval or rate, whatever the users already running are doing, and a phase ends when its time is up even if its sessions have not finished.

The load page lists the controls: interarrival or arrivalrate per phase, maxnumber to cap the users a phase starts, a duration on the whole load to stop the test at a fixed time, and wait_all_sessions_end to hold the next phase until the current users finish (checked on 21 October 2026). Our articles on the open vs closed workload model and on think time explain how those choices change what a test measures.

Which protocols can Tsung test?

More than most load testing tools: HTTP (including SOAP), WebDAV, XMPP, PostgreSQL, MySQL, LDAP, WebSocket, AMQP, MQTT and raw TCP, UDP or SSL. The documentation marks the MySQL, WebSocket, AMQP and MQTT plugins as experimental, with AMQP 0.9.1 and MQTT 3.1 only.

The features page describes each plugin (checked on 21 October 2026). XMPP and PostgreSQL are where Tsung has long had few rivals: it speaks the protocol itself, so you test the server rather than a client library. For real-time apps, our roundup of WebSocket load testing tools puts Tsung next to newer options.

How do you scale a Tsung test across machines?

You list the client machines in the XML, and the controller starts the load on them over SSH without a password, or rsh. Each client can take a weight, a user limit and a CPU count.

The clients and server page explains that Tsung runs one Erlang virtual machine per CPU you declare with cpu on each client (one by default) and starts a new one when a VM reaches maxusers. It can also take client machines from a batch scheduler such as PBS, LSF or OAR. The installation page lists Erlang/OTP R16B03 or later, plus optional Perl 5 and gnuplot, or Python and matplotlib, for the graphs (checked on 21 October 2026).

LoadFocus, which publishes this site, does not run Tsung files. It runs JMeter test plans and k6 scripts from 25 cloud regions.

How do you read Tsung results?

Two ways: a live dashboard served by the controller on port 8091, and an HTML report built after the run with the tsung_stats.pl script.

The reports page says the dashboard has existed since 1.6.0, shows the current status and builds graphs while the test runs, and stops when the test ends unless you start Tsung with -k (checked on 21 October 2026). The report covers response and connection times, transactions, requests per second and error codes, and can include server CPU, memory and network figures collected through Erlang agents, SNMP or munin. The documentation describes no pass/fail threshold, so a CI pipeline has to parse the results itself; our guide to load test pass/fail criteria covers what to check.

What does Tsung do well, and where are its limits?

It is strong on protocol breadth, open-model arrival phases and many users per machine, which the documentation puts at thousands per CPU. Its limits are a quiet release history, XML-only scripting and no built-in pass/fail checks.

XMPP, LDAP and wire-level database tests without writing client code.
Distribution over plain SSH, with no broker or extra service.
Server monitoring in the same report.
Logic beyond the XML means Erlang, which few teams write.
Erlang must be installed on the controller and every client.

When should you choose k6, Gatling or JMeter instead?

Choose another tool when you test HTTP or APIs and want an active project with scripts in a mainstream language.

Choose k6 instead for JavaScript scripts with thresholds built into the script and an active project (checked on 21 October 2026). Choose Gatling instead for code-first simulations on the JVM; our Gatling review covers it. Choose JMeter instead when testers want a GUI or you need JDBC, JMS or LDAP samplers with a large plugin ecosystem. LoadFocus runs JMeter and k6 from 25 cloud regions, and the open source load testing tools overview sets these tools side by side.

FAQ

Is Tsung still maintained? Yes, but quietly. The latest release, 1.8.0, came out on 2 March 2023, and the repository still receives occasional merged changes, most recently in March 2026.

Which protocols does Tsung support? HTTP and SOAP, WebDAV, XMPP, PostgreSQL, LDAP and raw TCP, UDP or SSL, plus MySQL, WebSocket, AMQP and MQTT, which the documentation marks as experimental.

Can LoadFocus run Tsung tests? No. LoadFocus runs JMeter test plans and k6 scripts. To spread a Tsung test over several machines, list them as clients in the XML file.

Azure Load Testing: a review of Microsoft's service

Mon, 19 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Azure Load Testing is Microsoft's managed service for running JMeter and Locust tests at scale from Azure, billed per virtual user hour. It is a strong fit when the system you test runs on Azure, because it can pull server metrics from Azure Monitor and reach private endpoints inside a virtual network. Teams that write k6 scripts, have no Azure subscription or test from outside Azure regions will find other services simpler.

We have not run Azure Load Testing for this review. The facts come from Microsoft's documentation and pricing page, checked on 19 October 2026, and the judgements are ours.

Topic	Azure Load Testing
Type	Managed service in the Azure App Testing hub
Test frameworks	Apache JMeter and Locust only
No-code option	URL-based quick test, up to five HTTP requests
Scale per run	Up to 400 engine instances
Longest test	24 hours
Load origin	The resource's region, or up to 5 regions
Private endpoints	Yes, through virtual network injection
Price	$0.15 per virtual user hour, $0.06 after 10,000
Results kept	6 months; client metrics 365 days

What is Azure Load Testing?

It is a fully managed service that runs your load test on Microsoft's test engines, so you never build or patch load generators. You create a load testing resource in a chosen Azure region, add tests to it and start runs from the portal, the Azure CLI or a pipeline.

The overview page says the application under test can live anywhere: in Azure, on-premises or in another cloud. Microsoft now lists the service inside the Azure App Testing hub, next to Playwright Workspaces; the Locust quickstart creates the resource from that hub (checked on 19 October 2026).

Which test types and tools does it support?

Three ways in: a URL-based quick test, an uploaded JMeter script, or an uploaded Locust script. The docs state plainly that it does not support any other testing framework.

The quick test takes up to five HTTP requests and a target number of virtual users or requests per second. Each engine handles up to 250 virtual users, so 400 users run on two engines with 200 each.

With JMeter, you can use plugins from jmeter-plugins.org or upload your own, and the overview says every protocol JMeter supports works, such as a database connection or a message queue. With Locust, you can add Locust plugins and other Python libraries. Locust's own documentation points to the service as its hosted option, and our Locust review covers the tool itself. k6, Gatling and Artillery scripts do not run here (checked on 19 October 2026).

How do engines, virtual users and limits work?

You scale a test by adding engine instances. For JMeter the total load is the users in your test plan multiplied by the number of engines; for Locust you set the total users and pick how many engines generate them.

The concepts page gives both formulas, and the service limits page sets the ceilings per region and subscription (checked on 19 October 2026):

Engine instances per test run: 1 to 400 by default depending on the offer, 400 at most.
Concurrent engine instances: 5 to 400 by default, up to 1,000 on request.
Test duration: 3 to 24 hours by default, 24 hours at most.
Fail criteria: up to 50 per test.
Retention: server-side metrics 90 days, client-side metrics 365 days, test results and logs 6 months.

Raising a default limit means a support request; nothing goes past the stated maximums.

Can it test private endpoints and generate load from several regions?

Yes to both, with one catch: they do not combine. Multi-region load only works against public endpoints.

By default the engines run in the region of your load testing resource. The multi-region guide lets you pick up to 5 regions per test and split the load between them, and it switches the test to public traffic when you do. For services without a public endpoint, virtual network injection places the engines inside the network that holds the application (checked on 19 October 2026).

What do the results and server metrics show?

The dashboard shows client-side metrics live during the run, such as virtual users, response time and requests per second. For Azure-hosted apps it adds server-side metrics from Azure Monitor, Application Insights and Container insights on the same timeline.

That pairing is the service's main advantage: you can see the database or container that saturates at the moment response times rise, without wiring up a separate dashboard. The server-side metrics guide lists the steps. Past runs stay comparable side by side, and you can download results for other tools.

How do pass/fail criteria and CI work?

You set fail criteria on client or server metrics, and the run is marked failed when any of them is breached. Pipelines in Azure DevOps, GitHub Actions or any CI tool that can call the Azure CLI can start a test and gate on that result.

Per the fail criteria guide, a criterion is an aggregate of a client metric, such as average response time or error percentage, compared with a threshold, either for the whole test or for one named request. Server metrics need the resource's managed identity to read them. Auto-stop ends a run when the error rate passes a limit, so a wrong URL does not keep billing. The CI/CD guide covers GitHub Actions and Azure Pipelines (checked on 19 October 2026). Our guide to load test pass/fail criteria explains how to choose the limits.

How much does Azure Load Testing cost?

You pay per virtual user hour (VUH): $0.15 per VUH for the first 10,000 VUH and $0.06 per VUH after that, the same in every region the pricing page lists. Since 1 March 2026 there is also a minimum charge per test run.

The pricing page defines a VUH as virtual users multiplied by minutes, divided by 60, counted every 10 seconds; 1,000 users for 30 minutes is 500 VUH, or $75 at the first-tier rate. The minimum is 10 virtual users per engine for the run's duration, and at least 10 minutes. A 5-user, 3-hour run on one engine is therefore billed as 30 VUH, not 15 (checked on 19 October 2026).

When should you choose something else?

Choose another option when your scripts, your cloud or your budget point elsewhere.

Our overview of open source load testing tools covers the tools you can run yourself.

Choose LoadFocus, which publishes this site, instead when you want to run JMeter test plans and k6 scripts from 25 cloud regions without an Azure subscription. It does not run Locust, and it runs k6 arrival-rate scenarios with a fixed number of users.

Choose Grafana Cloud k6 or the k6-operator instead when your tests are written for k6, which Azure Load Testing cannot run (checked on 19 October 2026).

Choose self-hosted Locust instead when you already run Kubernetes or spare machines: its distributed mode is free, and you only pay for the hardware.

FAQ

Does Azure Load Testing support k6? No. It runs Apache JMeter and Locust tests, plus URL-based quick tests that it turns into a script for you. k6 scripts need another service or your own infrastructure.

Can I use Azure Load Testing for an app that is not hosted on Azure? Yes. The test engines can target any public endpoint. Server-side metrics are only collected for Azure resources, so for other hosting you read server metrics from your own monitoring.

Is there a free tier? The pricing page lists only per-hour prices for load testing, with a minimum per test run. New Azure accounts get a $200 credit for their first 30 days, according to the same page.

Apdex score: what it is and how to calculate it

Mon, 19 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

An Apdex score is a number between 0 and 1 that tells you what share of requests were fast enough for the people waiting on them. You choose a target time T: responses at or under T count in full, responses up to four times T count as half, and slower ones count as zero. A result of 1 means every request met the target, and 0 means every request was too slow.

Zone	Response time	Weight in the score
Satisfied	up to T	1
Tolerating	above T, up to 4T	0.5
Frustrated	above 4T	0

The facts below come from the tools' own documentation and source code, checked on 19 October 2026.

How do you calculate an Apdex score?

Add the satisfied requests to half the tolerating ones and divide by all requests. The result always lands between 0 and 1.

Apdex(T) = (satisfied + tolerating / 2) / total

New Relic's Apdex documentation works through an example with T set to 500 ms: out of 200 requests, 170 finished within 500 ms, 20 took between 500 ms and 2 seconds, and 10 failed or took longer. That gives (170 + 20 / 2) / 200 = 0.90 (checked on 19 October 2026).

A second example shows why the score is useful in a load test. Say a run sends 10,000 requests with T at 300 ms: 8,200 finish within 300 ms, 1,300 between 300 ms and 1.2 seconds, and 500 take longer. The Apdex is (8,200 + 650) / 10,000 = 0.885. The same run could easily report an average of 280 ms, which looks fine on its own and hides the 500 requests that took over a second.

Where does Apdex come from?

Apdex, short for Application Performance Index, is an open standard from a group of companies called the Apdex Alliance. Peter Sevcik of NetForecast founded the Alliance in 2004.

The Wikipedia article on Apdex cites Sevcik's 2005 paper and gives the 4T rule. The Apdex Users Group now says the rules for the T and F thresholds "have become flexible" (checked on 19 October 2026), and tools use that freedom.

How do JMeter, Artillery and LoadFocus compute it?

All three start from T = 500 ms. They differ on where the tolerating zone ends and on what they do with failed requests, so the same run can give three slightly different numbers.

Tool	Default T	Tolerating up to	Failed requests
JMeter HTML dashboard	500 ms	1,500 ms	counted as frustrated
Artillery `apdex` plugin	500 ms	4T (2,000 ms)	only responses are scored
LoadFocus result page	500 ms, fixed	4T (2,000 ms)	scored on response time
New Relic APM	set per app	4T	server errors are frustrated

JMeter. The dashboard report has an APDEX table per transaction. Its defaults are jmeter.reportgenerator.apdex_satisfied_threshold=500 and apdex_tolerated_threshold=1500, so the tolerating zone ends at 3T unless you change it, and apdex_per_transaction sets different thresholds per sampler. In ApdexSummaryConsumer a failed sample adds to the total and never to the satisfied or tolerating counts, so every failure pulls the score down (checked on 19 October 2026).

Artillery. The apdex plugin works with the HTTP engine only and reports apdex.satisfied, apdex.tolerated and apdex.frustrated counters. Its source in artillery-plugin-apdex 1.24.0 scores each response on its total time against T and 4T, then prints the score with a label: excellent from 0.94, good from 0.85, fair from 0.70, poor from 0.49, unacceptable below that. A request that never gets a response is never scored (checked on 19 October 2026). Our Artillery review covers the tool itself.

LoadFocus, which publishes this site, shows the Apdex score in the Insights tab of a load test result, next to a response-time histogram. T is fixed at 500 ms and the tolerating zone ends at 2,000 ms; you cannot change T. Each sample is classified by its response time, with the satisfied, tolerating and frustrated shares printed beside the score.

How do you choose the T threshold?

Pick T from what users of that page or endpoint expect, and set it per transaction when your tool allows it. A single T for a whole application blurs a fast search endpoint and a slow report export into one number.

Some practical rules:

Start from a real target. If you already have a latency objective such as "95% of checkout calls under 400 ms", T = 400 ms keeps the Apdex and the objective telling the same story.
Keep T the same between runs. An Apdex is only comparable with another Apdex measured at the same T. Report it as Apdex(0.4) or "Apdex at T = 400 ms" so nobody compares across thresholds.
Match the tolerating zone too. JMeter's default stops at 3T and the others at 4T. When you compare a JMeter report with another tool, set apdex_tolerated_threshold to four times T first.
Do not tune T to make a release pass. Moving T after a bad run changes the ruler, and the trend you were watching is gone.

What is a good Apdex score?

Artillery's plugin calls 0.94 and above excellent and 0.85 to 0.94 good, and anything under 0.70 poor or worse. Those labels only mean something if T was chosen honestly.

An Apdex of 0.95 at T = 2 seconds can describe a slower service than 0.80 at T = 200 ms. When you set a release gate, write the threshold and the score together ("Apdex(0.5) of at least 0.9") and check it next to a percentile limit, because the two catch different problems.

What does Apdex hide?

It squeezes the whole response-time distribution into three buckets, so it cannot tell you how slow the frustrated requests were. A request at 2.1 seconds and one at 30 seconds lower the score by exactly the same amount.

That is why it works best next to other numbers:

Percentiles show how slow the slow requests are. Our guide to tail latency explains why the 99th percentile matters as soon as one page calls several services.
Error rate catches failures your tool may not score. Artillery never scores requests that got no response, and New Relic counts server errors as frustrated, so read the error count separately.
Sample size matters: New Relic notes that below about 100 requests per minute the score is unstable.
Measurement bias applies too. A load generator that waits for slow responses sends fewer requests during slow periods, so the Apdex looks better than reality; see coordinated omission.

How do you use Apdex in a load test?

Treat it as a summary for people who will not read a histogram, and keep the pass/fail decision on explicit limits.

A workable routine:

Set T per key transaction from your latency objective.
Run the test and record the Apdex at that T, along with the 95th and 99th percentiles and the error rate.
Compare the Apdex with the previous run at the same T. A drop from 0.93 to 0.86 is a clear signal even to someone outside the team.
Fail the build on percentile and error limits, as described in our guide to load test pass/fail criteria, and report the Apdex alongside them.

FAQ

Is a higher Apdex score better? Yes. 1.0 means every request finished within T, and 0 means every request took longer than the tolerating limit or was counted as frustrated.

What is the default Apdex threshold in JMeter? The HTML dashboard uses 500 ms for satisfied and 1,500 ms for tolerated. Change them with jmeter.reportgenerator.apdex_satisfied_threshold and apdex_tolerated_threshold.

Does Apdex include errors? It depends on the tool. JMeter counts failed samples as frustrated and New Relic treats server errors the same way, while Artillery's plugin only scores requests that received a response.

Can I compare Apdex scores from two tools? Only when T, the tolerating limit and the handling of errors match. Otherwise compare the underlying percentiles and error rates instead.

Artillery load testing: a review of the Node.js tool

Mon, 19 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Artillery is an open source load testing tool that runs on Node.js and describes each test as a YAML, TypeScript or JavaScript file. Artillery load testing suits teams that live in the JavaScript ecosystem, want new users to arrive at a set rate, or want browser tests with Playwright in the same tool. Teams that want a single binary with no runtime are better served by k6, and teams that need many protocols out of the box by JMeter.

We have not run Artillery for this review. The facts come from Artillery's documentation, repository, npm package and pricing page, checked on 19 October 2026, and the judgements are ours.

Topic	Artillery
Licence	MPL-2.0; some Azure modules under BSL
Runtime	Node.js, latest LTS (package needs 22.18.0 or later)
Current release	2.0.34 (14 August 2026)
Tests written as	YAML, TypeScript or JavaScript
Load model	New virtual users per second, in phases
Built-in engines	HTTP, WebSocket, Socket.IO, Playwright
Pass/fail checks	`ensure` plugin, non-zero exit code
Scaling out	`run-lambda`, `run-fargate`, `run-aci`
Hosted reports	Artillery Cloud, opt-in, free tier

Who makes Artillery, and how is it licensed?

Artillery Software Inc. maintains it on GitHub. Most of the code is MPL-2.0, which allows commercial use; some Azure-specific modules are under the Business Source License.

The repository README says you may use Artillery on Azure for evaluation and proof-of-concept work, and that commercial or production use there needs a commercial licence. The latest release is 2.0.34, published to npm on 14 August 2026, and its package metadata asks for Node.js 22.18.0 or later; the install guide says to use the most recent LTS release (checked on 19 October 2026).

How does an Artillery test script work?

A script has two parts: config, which sets the target, the load phases and plugins, and scenarios, which list what each virtual user does. You can write it in YAML, or export config and scenarios from a TypeScript or JavaScript file.

This test ramps from 5 to 50 new users per second over two minutes, holds 50 per second for ten minutes, and fails if the 99th percentile response time reaches 800 ms:

config:
  target: https://staging.example.com
  phases:
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: ramp
    - duration: 600
      arrivalRate: 50
      name: sustain
  plugins:
    ensure:
      thresholds:
        - http.response_time.p99: 800
scenarios:
  - name: browse and add to cart
    flow:
      - get:
          url: /products
      - think: 2
      - post:
          url: /cart
          json:
            sku: A-100

Per the test script reference, the four phase types are a constant arrival rate, a linear ramp with rampTo, a fixed arrivalCount spread over a period, and a pause. Custom code loaded through config.processor can be a TypeScript module locally and on Fargate, but not on Lambda (checked on 19 October 2026).

What load model does Artillery use?

Artillery starts new virtual users at a rate you set, whether or not earlier users have finished. That is an open workload model, which keeps arrivals steady even when the system slows down.

Each new user picks one scenario, runs it and leaves, so a slow system collects users the way real traffic would. maxVusers caps the number of concurrent users in a phase if you need to protect the load generator. Our article on the open vs closed workload model explains the difference, and Little's law shows how arrival rate and response time set the number of concurrent users.

Which protocols and browsers can it test?

HTTP is the default, and WebSocket, Socket.IO and Playwright come built in. gRPC and AWS Kinesis need separately installed engines.

The engines page describes the model: each scenario runs on one engine, and you can write your own. On npm, artillery-engine-grpc is at 0.0.9 and artillery-engine-kinesis at 1.1.1, both MPL-2.0. The Playwright engine runs real headless browsers from your Playwright code and reports Web Vitals such as TTFB, FCP and LCP per page. The docs warn that a separate browser per user, instead of a browser context, needs a lot more CPU and memory (checked on 19 October 2026). Our roundup of WebSocket load testing tools puts Artillery next to other options for real-time apps.

How does it fit into CI?

The ensure plugin turns metrics into pass/fail checks, and a failed check makes Artillery exit with a non-zero code. That is enough for any CI system to fail the build.

The ensure reference offers two kinds of check: thresholds, where a metric must stay below a value, and conditions, which combine metrics in an expression such as http.response_time.p95 < 250 and http.request_rate > 1000. A condition marked strict: false reports without failing. The apdex plugin adds an Apdex score for HTTP tests, with a default threshold of 500 ms (checked on 19 October 2026); our article on the Apdex score explains how to read it. For a workflow file, see our guide to load testing in GitHub Actions.

How do you scale out an Artillery test?

From the command line, on your own cloud account: run-lambda and run-fargate on AWS, run-aci on Azure Container Instances. Artillery creates the resources it needs for the run and removes the temporary ones afterwards.

The Fargate guide lists the resources and IAM permissions involved. The Lambda guide states its limits: a test cannot run longer than 15 minutes, a running test cannot be stopped, and before and after hooks run once per worker. The Azure guide needs Artillery 2.0.19 or later and, outside evaluation, a subscription (checked on 19 October 2026).

LoadFocus, which publishes this site, does not run Artillery scripts. It runs JMeter test plans and k6 scripts from 25 cloud regions, so for Artillery the commands above are the way to scale.

What does Artillery Cloud cost?

The dashboard is optional: results go to it only when you run with --record and an API key. Plans run from a free tier to $499 a month.

The pricing page lists (checked on 19 October 2026):

Free: 30 reports a month, 1 month of retention, 2 team members, distributed tests up to 5 workers and 30 minutes.
Team, $199 a month: 1,000 reports, 6 months, 8 members, up to 25 workers and 2 hours.
Business, $499 a month: 2,500 reports, 18 months, no limit on members, workers or duration.

Enterprise SSO, audit logs and support SLAs start at $1,199 a month, and annual billing saves 20%.

What does Artillery do well, and where are its limits?

It is strong on arrival-rate load, short readable scripts and browser tests with Playwright. It is weaker where you need a single binary, many protocols without add-ons, or Azure use without a licence.

As a Node.js tool it can use any npm module in custom functions, at the cost of one more runtime on every test machine. Limits the docs state themselves:

Lambda runs stop at 15 minutes and cannot be cancelled.
TypeScript custom code (config.processor) does not run on Lambda.
gRPC and Kinesis are separate engines.
Production use on Azure requires a commercial licence.

When should you choose k6, JMeter or Locust instead?

Choose another tool when its runtime, protocols or language fit your team better.

Choose k6 instead when you want one Go binary with no Node.js to manage and thresholds in the script. Our k6 vs Artillery comparison covers the two side by side. LoadFocus runs k6 scripts from 25 cloud regions, though it runs arrival-rate scenarios with a fixed number of users, so for a held arrival rate run them with k6 itself.

Choose JMeter instead when you need JDBC, JMS, LDAP or FTP out of the box, or testers who build tests in a GUI.

Choose Locust instead when your team writes Python; our Locust review covers it. The open source load testing tools overview sets all of them side by side.

FAQ

Is Artillery free for commercial use? Most of the CLI is MPL-2.0, which permits commercial use. The exception is Azure: commercial or production use of its Azure modules needs a commercial licence.

Can Artillery run browser tests? Yes. Its built-in Playwright engine runs your Playwright code in headless browsers and reports Web Vitals for each page.

Can LoadFocus run Artillery scripts? No. LoadFocus runs JMeter test plans and k6 scripts. To scale an Artillery test, use run-lambda, run-fargate or run-aci.

Tail latency: what it is and how to measure it

Mon, 19 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Tail latency is the response time of the slowest few requests, usually reported as the 99th or 99.9th percentile. It matters because one user action often waits on many requests, so a delay that hits 1 request in 100 can reach most of your users. A load test that reports only the average or median will not show it.

Backend calls per page	Share of pages hitting at least one 1-in-100 slow call
1	1.0%
5	4.9%
10	9.6%
20	18.2%
50	39.5%
100	63.4%

The table assumes each call is slow with a 1% chance, independently of the others: the share is 1 - 0.99^n.

What counts as the tail?

The tail is the part of the response-time distribution above a high percentile, most often p99 and p99.9. The median describes a typical request, and the tail describes the requests people complain about.

A percentile is a cut-off: p99 = 800 ms means 99% of requests took 800 ms or less and 1% took longer. The tail sits to the right of that cut-off on a response-time histogram, often as a long, thin stretch far from the main peak. Our guide to throughput vs latency explains how latency and percentiles are measured in load testing tools.

Why does tail latency matter so much?

Because requests multiply. A page or API call that fans out to many services finishes only when the slowest of them answers, so the tail of each service becomes the typical experience of the whole.

The standard reference is "The Tail at Scale" by Jeffrey Dean and Luiz André Barroso, published in Communications of the ACM in 2013 (paper PDF, checked on 19 October 2026). Their example: a server that usually answers in 10 ms but takes one second at the 99th percentile. If one request touches one such server, 1 user request in 100 is slow. If it must collect answers from 100 of them in parallel, 63% of user requests take more than one second. Even with only 1 slow call in 10,000, a service with 2,000 servers sees almost 1 in 5 requests over a second.

They also report measurements from a real Google service: the 99th percentile for a single leaf request was 10 ms, but for all leaf requests to finish it was 140 ms, and waiting for the slowest 5% of them accounted for half of that.

What causes tail latency?

Usually interference from other work on the machine or in the network. The same request is fast most of the time and slow when something else is going on.

Dean and Barroso list the usual suspects:

Shared resources: other work on the same machine competing for CPU cores, processor caches, memory bandwidth or network bandwidth.
Background daemons that wake up and take a slice of the machine.
Maintenance work such as log compaction or garbage collection in managed runtimes.
Queueing in servers and network switches, which amplifies every other source.
Power and thermal limits that slow a CPU after a burst of work.

In load tests, the same list shows up as GC pauses in JVM or Node.js services, connection pool exhaustion, a slow database query that only some requests trigger, cold caches after a deploy, and retries after a timeout. A tail that grows with load points at queueing; a tail that is there even at low load points at something periodic, such as GC or a scheduled job.

How do you measure tail latency in a load test?

Collect enough samples, ask your tool for the right percentiles, and never average percentiles across machines or time windows.

Enough samples. p99 is decided by the slowest 1% of requests, so a run with 500 requests puts it on 5 of them. For a stable p99 you want thousands of requests per transaction, and for p99.9 tens of thousands. Short smoke tests are fine for catching errors and poor for judging the tail.

The right percentiles. Defaults often stop short of the tail. k6's summaryTrendStats default is avg,min,med,max,p(90),p(95), so p99 only appears if you add it (k6 options reference, checked on 19 October 2026):

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9),max" \
  script.js

JMeter's Aggregate Report shows the 90%, 95% and 99% lines, and the aggregate_rpt_pct1 to aggregate_rpt_pct3 properties change which percentiles appear (checked on 19 October 2026).

No averaged percentiles. The p99 of a whole run is not the average of each minute's p99, and it is not the average of each load generator's p99 either. Merge the raw samples or a histogram, then compute the percentile once. The coordinated omission article covers HdrHistogram, the usual tool for that, and a second trap: a load generator that waits for slow responses sends fewer requests while the system is slow, which hides the tail you are looking for.

LoadFocus, which publishes this site, shows P95 and P99 tiles for each run, a response-time histogram built from the individual samples, and the slowest requests, so a long tail to the right of the peak is visible at a glance.

How do you set a pass/fail limit on the tail?

Put a limit on p99 per key transaction, next to a p95 limit and an error-rate limit. Base it on what users of that transaction need, and keep the run long enough for p99 to be stable.

A sensible set for a checkout API could read: p95 under 400 ms, p99 under 1,200 ms, errors under 0.5%. In k6 those are thresholds on http_req_duration such as p(99)<1200, and a failed threshold makes k6 exit with a non-zero code (k6 thresholds, checked on 19 October 2026). LoadFocus has a Max P99 response time field next to Max P95 for cloud, JMeter and k6 tests, and every run is then graded PASS or FAIL. Our guide to load test pass/fail criteria covers the rest of the set.

If you also report an Apdex score, remember it puts every request over four times T in one bucket, so it cannot tell a 2-second tail from a 30-second one. The p99 can.

How do you reduce tail latency?

Remove the sources of variability where you can, and design callers so one slow call does not hold up the whole response.

Common fixes, roughly from cheapest to most involved:

Tune GC and pool sizes so pauses and waits for a free connection stay short under load.
Keep headroom. Queueing grows sharply near saturation, so spare capacity shortens the tail; the throughput vs latency guide shows how to find the knee.
Set timeouts and limit retries so a stuck call fails fast instead of piling up behind it.
Use hedged requests for idempotent reads: send a second copy to another replica if the first has not answered in time. Dean and Barroso describe sending the backup only after the 95th-percentile expected latency, which limits the extra load to about 5%. In one of their tests, a hedge after 10 ms cut the 99.9th percentile from 1,800 ms to 74 ms while sending just 2% more requests.
Fan out less. Fewer backend calls per page means fewer chances to hit the tail, as the table at the top shows.

After each change, run the same test again with the same load profile and compare p99 and p99.9 between runs.

FAQ

Is tail latency the same as p99? p99 is the most common way to report it, and the tail is the whole slow end of the distribution. Track p99.9 and the maximum as well when the run has enough requests.

Why is my p99 so much higher than my average? Because a few slow requests barely move the average and fully define the p99. A large gap is normal; a gap that grows as load rises usually means queueing.

How many requests do I need to measure p99? At the very least a few thousand per transaction, so that the p99 rests on dozens of slow requests. For p99.9, plan for tens of thousands.

gRPC load testing: tools, setup and what to watch

Fri, 16 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

gRPC load testing needs a tool that speaks HTTP/2 and Protocol Buffers: k6 and ghz do it out of the box, JMeter needs a plugin, Gatling's gRPC support is capped in its free edition, and Locust needs you to wrap a gRPC client yourself. Count results by gRPC status, not HTTP status, because failed calls still arrive as HTTP 200, and check that the load reaches every backend.

No gRPC test was run for this article. Tool facts come from each project's documentation and repository, checked on 16 October 2026; LoadFocus facts come from its code.

Tool	gRPC support	Loads the API from	Rate control
k6	Built in (`k6/net/grpc`), unary calls and streams	`.proto` files or server reflection	Arrival-rate executors
ghz	Purpose-built gRPC CLI	`.proto`, protoset or server reflection	`--rps` and load schedules
JMeter	Third-party plugin (jmeter-grpc-request)	A folder of `.proto` files	Thread groups and timers
Gatling	Separate gRPC component; free edition capped at 5 users, 5 minutes	Code generated from `.proto` with protoc	Injection profiles
Locust	No built-in client; wrap one and report calls	Your Python gRPC client	Wait times, `constant_throughput`

Why does gRPC need a different load test from REST?

gRPC runs over HTTP/2, sends binary Protocol Buffers instead of JSON, puts the call's result in a trailer rather than the HTTP status, and multiplexes many calls on one connection. Each of those changes how you script the test or read its results.

The gRPC load balancing post lists the HTTP/2 traits that matter here: a binary protocol, many requests multiplexed on one connection and header compression, with strongly typed Protobuf messages on top (checked on 16 October 2026). A tool that only sends JSON over HTTP/1.1 cannot call a gRPC method, so check protocol support first.

Which tool should you use?

Use k6 if you want scripted scenarios, checks and thresholds in code; use ghz for a quick benchmark of one method from the command line. Reach for JMeter, Gatling or Locust when your team already runs everything else in them.

k6. The k6/net/grpc module loads Protobuf definitions, connects, makes unary calls with invoke and opens streams. It reports grpc_req_duration and stream message counts in its metrics.
ghz. A gRPC-only benchmarking CLI. Its options include --rps, --concurrency, --total and --duration, and if you pass neither --proto nor --protoset it tries server reflection. The latest release is v0.121.0 (checked on 16 October 2026).
JMeter. Core JMeter has no gRPC sampler; the jmeter-grpc-request plugin adds one that reads a folder of .proto files, with no code generation (latest tag v1.2.6, checked on 16 October 2026).
Gatling. The Gatling README puts gRPC, MQTT and GraphQL under the Gatling Enterprise Component License, capped at 5 concurrent users and 5-minute tests in Community Edition (checked on 16 October 2026). Its gRPC setup guide generates code from your .proto files with protoc.
Locust. Its docs on testing other systems include a gRPC example: you wrap a gRPC client so each call is reported to Locust, and constant_throughput paces a task to at most a set number of runs per second.

Choose ghz instead of k6 when you only need to know how one method behaves at a given rate and do not want to write a script. Choose Locust instead when your team writes Python and the test needs logic that is easier there.

How do you write a gRPC load test in k6?

Load the .proto file in the init code, then connect, invoke the method and check the gRPC status in each iteration. The structure follows the k6 documentation's own example.

import grpc from 'k6/net/grpc';
import { check } from 'k6';

const client = new grpc.Client();
client.load(['protos'], 'hello.proto');

export const options = { vus: 20, duration: '1m' };

export default () => {
  client.connect('grpc.example.com:443');
  const res = client.invoke('hello.Greeter/SayHello', {
    name: 'load test',
  });
  check(res, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });
  client.close();
};

The invoke page uses the same load, connect, invoke, check and close sequence. connect uses TLS unless you pass plaintext: true, and with reflect: true it asks the server for its API instead of reading a .proto file (connect parameters). To hold a request rate rather than a number of users, run the function in an arrival-rate scenario.

How do you run a quick benchmark with ghz?

Point ghz at the method, give it a request body and a rate or concurrency, and it prints a latency histogram, percentiles and a count for each status code. It suits a first look at one method.

ghz --insecure --proto ./hello.proto \
  --call hello.Greeter.SayHello \
  -d '{"name":"load test"}' \
  --rps 200 -c 50 -z 1m \
  localhost:50051

--rps sets the target request rate, shared among the workers, -c sets how many workers there are, and -z sets the duration. Drop --proto to let ghz use server reflection, if the server enables it. The ghz output page shows a sample report.

What should you measure in a gRPC load test?

Measure call latency percentiles, throughput and the count of each gRPC status code. An HTTP-level success rate can read 100% while calls fail, because the result travels in the grpc-status trailer.

The gRPC HTTP/2 protocol spec sets the HTTP status to :status 200 and carries the call's result in trailers as grpc-status and grpc-message. The status code list has the codes that matter most under load:

DEADLINE_EXCEEDED (4): the deadline expired before the call completed.
RESOURCE_EXHAUSTED (8): a quota or resource ran out.
INTERNAL (13): an internal invariant broke, a code reserved for serious errors.
UNAVAILABLE (14): usually transient; retrying with backoff can clear it, though not always safely for non-idempotent calls.

Count each one per load step. Deadline errors can climb while the calls that succeed still look fast, because the slow calls are cut off at the deadline instead of being recorded as slow. Our guide to load test pass/fail criteria shows how to turn those counts into thresholds.

Why does all the load land on one server?

Because gRPC multiplexes many calls on one long-lived connection, a load balancer that works per connection sends every call on it to the same backend. A load generator with few connections can then saturate one server while the others sit idle.

The gRPC load balancing post explains the mechanism. A transport-level (L3/L4) balancer terminates the TCP connection and opens another to one chosen backend, copying the HTTP/2 and gRPC frames between them; an application-level (L7) balancer parses HTTP/2 and can spread the streams of one client connection across backends. If your production clients go through an L7 balancer or balance on the client side, test through the same path, and check per-backend CPU and request counts during the run.

Can you run gRPC load tests on LoadFocus?

Not in a way its reports would show. LoadFocus runs the stock k6 v0.57.0 release, which includes the gRPC module, but its k6 agent forwards only HTTP request samples to the results, so gRPC calls would generate load without appearing in its charts or error counts.

Two more limits apply. The k6 upload accepts .js, .ts, .csv, .txt, .json, .env and .properties files, not .proto files, so a script would have to connect with reflect: true. And the JMeter route is closed: LoadFocus's JMeter agents ship JMeter 5.6.2 with plugins that do not include a gRPC sampler, and the JMeter upload does not accept plugin JAR files. For gRPC, run k6 or ghz on your own machines, and use LoadFocus's k6 scripts for the HTTP side of the same system, where it generates load from 25 regions.

FAQ

Can you load test gRPC streaming calls? Yes, with k6 and ghz. The k6 gRPC module has a Stream class with write, end and event handlers, and ghz has options for streaming intervals and durations. Measure messages sent and received as well as call latency.

Do you need the .proto file to load test a gRPC service? Not if the server enables reflection. k6 can connect with reflect: true, and ghz falls back to reflection when you pass neither --proto nor --protoset. Without reflection you need the .proto file or a compiled protoset.

Why do my gRPC errors not show up as failed requests? Because the HTTP status of a gRPC call is 200 even when the call fails. The result is in the grpc-status trailer, so check that value in your script and count each code separately.

Throughput vs latency: how they interact under load

Fri, 16 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Throughput is how much work a system finishes per unit of time, such as requests per second. Latency is how long one piece of work takes, such as 120 milliseconds for one request. Under light load the two barely affect each other; as load approaches the system's capacity, throughput stops rising and latency starts to climb, and finding that point is what a load test is for.

The worked example below is arithmetic with no test run behind it. Tool definitions come from each project's documentation, checked on 16 October 2026.

	Throughput	Latency	Bandwidth
Question it answers	How much finishes per second?	How long does one take?	How much could the link carry?
Typical unit	requests or transactions per second	milliseconds	megabits per second
In k6	`http_reqs` (count and rate)	`http_req_duration`	not a test target
In JMeter	Throughput	Elapsed time (and a separate "Latency")	not a test target

What is the difference between throughput and latency?

Throughput counts completed work over time; latency measures the time one unit of work takes. A system can have low latency and low throughput, or high throughput and high latency, so you need both numbers to describe it.

JMeter's glossary calculates throughput as the number of requests divided by the total time, from the start of the first sample to the end of the last, including the gaps between samples, "as it is supposed to represent the load on the server" (checked on 16 October 2026). That makes throughput a property of the whole run, while latency belongs to each request and is best read as a distribution: median, p95, p99.

Is bandwidth the same as throughput?

No. Bandwidth is the most data a network link could carry; throughput is what actually gets through. In a load test, throughput usually means completed requests, and bandwidth matters only when large payloads fill the link before the application runs out of capacity.

AWS's comparison page puts the network view simply: latency is the delay in network communication, throughput is the average volume of data that actually passes over a period, and bandwidth is the theoretical maximum (checked on 16 October 2026). Much of what you find online uses that network sense. Load testing uses the same words for requests rather than packets, which is why it pays to say which one a report means.

What does "latency" mean in a load testing tool?

It depends on the tool. In JMeter, Latency is the time to the first part of the response, while the full response time is called Elapsed time. In k6, http_req_duration covers sending, waiting and receiving, and the time to first byte is a separate metric.

The JMeter glossary measures elapsed time "from just before sending the request to just after the last response has been received" and latency to "just after the first response has been received". The k6 metrics reference defines http_req_duration as http_req_sending + http_req_waiting + http_req_receiving, without DNS lookup or connection time, and calls http_req_waiting the time to first byte (both checked on 16 October 2026). So JMeter's Latency lines up with k6's http_req_waiting, not with http_req_duration. Compare a JMeter "latency" with a k6 "duration" and the JMeter number will look better without being better.

How do throughput and latency change as load grows?

At first, more load means more throughput and latency stays roughly flat. Once the system reaches its capacity, throughput stops growing and every extra user only adds waiting, so latency rises with the load.

A worked example, arithmetic only. A service handles at most 200 requests per second and answers in 0.1 seconds when it is not busy. Each virtual user sends one request, then pauses for 1 second. With a fixed number of users, the closed-model form of Little's law ties them together: users = requests per second × (latency + think time), as our Little's law article shows step by step.

Virtual users	Throughput (requests/s)	Latency (s)
50	45	0.1
150	136	0.1
220	200	0.1
300	200	0.5
400	200	1.0
600	200	2.0

Up to 220 users, throughput grows in step with them. After that it is pinned at 200, so latency has to absorb the rest: 300 users need 300 ÷ 200 = 1.5 seconds per loop, which leaves 0.5 seconds of latency. Real systems typically bend more gradually, because queues start to form before the exact ceiling. With an arrival-rate test the picture is harsher: if requests arrive faster than 200 per second, the backlog keeps growing for as long as the excess lasts, which is the difference the open vs closed workload model article explains.

How do you find the knee in your own results?

Raise the load in steps, hold each step long enough to settle, and record throughput and a latency percentile for each one. The knee is the first step where throughput stops rising with the load while latency starts to climb.

A few things make that reading reliable:

Hold each step for a few minutes so throughput is measured on a steady stretch, not a ramp.
Read a percentile, not only the average; an average can stay calm while the slowest few percent of requests get much slower.
Check errors at every step. Fast failures count as completed requests with short times, so a failing system can show rising throughput and falling latency.
Watch the load generator. If its CPU is saturated, the ceiling you found may be its own.

LoadFocus's response time vs virtual users chart plots average and maximum response time against active virtual users, one point per user level, which is where the knee shows up in a stepped test.

Can you raise throughput without raising latency?

Sometimes. Adding capacity moves the ceiling, so latency stays flat for longer. Batching usually trades the other way: it raises throughput by making individual requests wait.

TCP's Nagle algorithm is the classic example of that trade. RFC 896 holds back new segments while earlier data on the connection is still unacknowledged, so the network carries fewer, fuller packets and each small write may wait (checked on 16 October 2026). Request batching and write buffering work the same way: more done per second, more time per item. When you change one of them, rerun the stepped test and compare both curves, not just one number.

Which one belongs in your pass/fail criteria?

Both, tied together. A latency target means little without the throughput it was measured at, and a throughput target means little if latency doubled to reach it. State them as a pair, such as p95 under 300 milliseconds at 200 requests per second with under 1% errors.

Our guide to load test pass/fail criteria shows how to write and automate that pair. If your tool waits for slow responses before sending the next request, its latency figures can understate the problem; coordinated omission explains how that happens and how to avoid it. LoadFocus's glossary entry on latency breaks down where latency comes from and why percentiles matter.

FAQ

Does low latency mean high throughput? Not necessarily. A system can answer each request quickly but handle few at once, or handle many at once while each one waits. You only know both after measuring both under the same load, ideally across several load levels.

What is the difference between latency and response time? In everyday use they often mean the same thing. In JMeter, Latency is the time to the first part of the response and Elapsed time is the full response time, so check which one a report shows before comparing tools.

Why does throughput stop increasing in a load test? Something has reached its limit: CPU, a database, a connection pool, a rate limit, or the load generator itself. From that point extra users only queue, which shows up as rising latency at flat throughput.

Gatling vs Locust: JVM simulations or Python locustfiles?

Tue, 13 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Gatling and Locust are both free, open source tools that keep load tests as code in your repository. Pick Gatling if your team works on the JVM or you want open and closed injection profiles, assertions and an HTML report in one package. Pick Locust if your team writes Python, wants a web UI to start and watch a run, and needs distributed mode without a commercial edition.

The facts in this Gatling vs Locust comparison come from each project's own documentation and repository, checked on 13 October 2026.

	Gatling	Locust
Licence	Apache 2.0; gRPC, MQTT and GraphQL under a separate component licence (repository)	MIT (repository)
Runtime	JVM, OpenJDK LTS 11 to 25 (release 3.16.0)	Python 3.11 or later (release 2.46.7)
Test code	Java, Kotlin, Scala, JavaScript or TypeScript	Python
Built-in protocols	HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events, JMS	HTTP and HTTPS; others through a Python client
Load model	Open or closed injection profiles	A pool of users, so a closed model
Pass/fail	Assertions, evaluated after the run	Exit code 1 on failed requests, or your own listener
Reports	HTML report after each run	Live web UI charts, CSV files, HTML report
Scaling out	Gatling Enterprise	Built-in master and workers, Kubernetes operator

What is the difference between Gatling and Locust?

Gatling runs simulations on the JVM, and Locust runs plain Python programs. Gatling gives you a load model and a report out of the box, and Locust gives you test logic you write yourself.

Gatling's README describes a non-blocking engine built on Netty and an SDK in five languages: Java, Kotlin and Scala run through Maven, Gradle or sbt, and JavaScript and TypeScript through a separate SDK. Its installation page asks for a 64-bit OpenJDK LTS from 11 to 25, and for Node.js 24 or later with the JavaScript SDK.

Locust's README calls it a way to write scalable load tests in plain Python, and its package metadata requires Python 3.11 or later. The What is Locust page says it is event-based on gevent, so one process can handle many thousands of concurrent users (checked on 13 October 2026).

How do you write the same test in each tool?

In Gatling you chain requests into a scenario and give it an injection profile. In Locust you write a user class with tasks, and a wait time between them.

Fifty concurrent users who list users, then pause one to three seconds, for five minutes. In Gatling's Java SDK (imports left out):

public class ApiSimulation extends Simulation {
  HttpProtocolBuilder api =
      http.baseUrl("https://staging.example.com");

  ScenarioBuilder users = scenario("API users")
      .exec(http("list users").get("/api/users"))
      .pause(1, 3);

  {
    setUp(users.injectClosed(
        constantConcurrentUsers(50).during(300)))
      .protocols(api);
  }
}

The same test in Locust:

from locust import HttpUser, task, between


class ApiUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def list_users(self):
        self.client.get("/api/users")

Gatling's injection reference lists constantConcurrentUsers among the closed-model steps. Locust takes the user count and run time as flags (checked on 13 October 2026):

locust -f locustfile.py --headless -u 50 -r 5 -t 5m \
  --host https://staging.example.com

How do Gatling and Locust model load?

Gatling offers both open and closed models, chosen by the method you call. Locust runs a pool of users, which is a closed model.

Gatling's workload models page defines closed systems as ones that cap concurrent users and open systems as ones where users keep arriving however slowly the application responds, and says most websites are open. injectOpen takes arrival rates and injectClosed takes concurrency targets.

In Locust, the locustfile guide says wait time can only constrain throughput and cannot launch new users to reach a target rate. If an iteration takes longer than the wait allows, the rate drops below the target (checked on 13 October 2026). Our guide to the open vs closed workload model explains why that matters.

How do you run a bigger test than one machine can generate?

Locust has distributed mode built in. Gatling's open source edition generates load from one machine, and distributed runs need Gatling Enterprise.

Locust starts one process with --master and any number of --worker processes. Its distributed docs say a Python process cannot use more than one core, so you run one worker per core, and --processes starts several on one machine. A Kubernetes operator manages distributed runs inside a cluster.

Gatling's FAQ says the Community Edition does not support distributed testing, and the README describes Enterprise as running on managed load generators or private infrastructure (checked on 13 October 2026).

There is a way out of both. LoadFocus, which publishes this site, does not run Gatling simulations or Locust tests. If you would rather not run either at scale, it runs k6 scripts and JMeter test plans from 25 cloud regions. Its k6 validator warns that arrival-rate scenarios run there with a fixed number of users, so they do not hold a request rate. Moving from either tool means rewriting the test.

How do they decide whether a test passed, and what reports do you get?

Gatling evaluates assertions after the run and writes an HTML report every time. Locust exits with code 1 if any request failed, and you add your own checks as a listener.

Gatling's assertions are evaluated after the simulation, and one failed assertion fails the simulation. Its HTML report charts response time percentiles, users started per second, concurrent users and requests per second.

Locust's headless docs set exit code 1 when any sample failed, and show a listener that fails the run on failure ratio, average response time or 95th percentile. The configuration reference lists --csv for statistics files and --html for a report (checked on 13 October 2026).

Which protocols can each tool test?

Gatling ships more protocols, with limits on the newer ones. Locust ships HTTP and HTTPS and expects a Python client for the rest.

The Gatling README lists HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events and JMS under Apache 2.0 with no limits. gRPC, MQTT and GraphQL sit under the Gatling Enterprise Component License, and Community Edition tests with them are capped at 5 concurrent users and 5 minutes.

Locust's page on other systems shows how to wrap a protocol library and report each call, with gRPC and MQTT examples. The library has to work with gevent, which cannot patch I/O done in compiled C code, and locust-plugins adds Kafka, Selenium and Playwright users (checked on 13 October 2026).

Which one should you choose?

Choose by language and by how you load the system. Gatling suits JVM teams and open-model tests, and Locust suits Python teams and tests with custom logic.

Choose Gatling instead of Locust when your team lives on the JVM and wants Java, Kotlin or Scala tests built with Maven, Gradle or sbt, when you need an open model such as a fixed arrival rate, when you need JMS or Server-Sent Events, or when you want assertions and a report with no extra code.

Choose Locust instead of Gatling when your team writes Python and wants tests that reuse an internal Python client, when you want a web UI to start a run and change the load while it runs, when you need distributed mode in the free tool, or when the system is mostly waiting users and your protocol has a gevent-friendly Python library.

For a GUI test builder look at JMeter, and for a single binary with no JVM or Python look at k6.

FAQ

Is Gatling faster than Locust? We have not measured this ourselves, so we will not name a winner. The fair test is the same scenario with both tools on your own hardware.

Can Locust run Gatling simulations? Not according to Locust's documentation, which describes tests only as Python code. A Gatling simulation has to be rewritten as a locustfile, and the reverse is true as well.

Can LoadFocus run Gatling or Locust tests? No. LoadFocus runs JMeter test plans and k6 scripts. To spread a Gatling simulation across machines use Gatling Enterprise, and for Locust use its master and worker mode or its Kubernetes operator.

JMeter vs Postman: which tool for load testing?

Tue, 13 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Postman is an API client built for functional checks, collections and team collaboration, and it has added a performance test mode on top. Apache JMeter is a load generator from the start. Use Postman when you want to reuse the collection you already have for a quick load check or a CI smoke test. Use JMeter when load testing is the job: many protocols, large user counts, detailed reports and distributed generation.

The facts about Postman come from its documentation and pricing page, and the facts about JMeter from its own site, all checked on 13 October 2026.

	Postman	Apache JMeter
Main purpose	API client, functional and collection testing, with a performance mode	Load and performance testing
Test definition	A collection of requests with scripts and `pm.test()` assertions	A test plan built in a GUI and saved as a `.jmx` file
Virtual users	Each runs the collection in a loop; local runs are limited by your machine	One Java thread per user in a thread group
Load profiles	Fixed, ramp up, spike and peak	Thread groups with a ramp-up period
Where load comes from	Your machine or CI runner, or Postman's cloud on paid plans	Your machine, or remote JMeter servers you run
Cost of load	Local runs on all plans; cloud runs at $0.04 per virtual user hour	Free, Apache 2.0 licence
Pass or fail in CI	`postman performance run` with `--pass-if`	CLI mode (`jmeter -n`)

What does Postman's performance testing do?

It runs the requests of a collection with several virtual users in parallel, for a set number of minutes. Each user repeats the requests in order, and your existing scripts and assertions run under load.

Postman's performance testing docs say the collection you use for functional tests is also your load test, with no separate script or language to maintain (checked on 13 October 2026). You pick a duration and a number of virtual users, then a fixed, ramp up, spike or peak profile, as the configuration page describes.

What are the limits of Postman for load testing?

The main limit is where the load comes from. A local run is capped by the machine that runs it, and anything beyond that needs a paid plan.

For local runs, Postman's docs say the number of virtual users depends on available system resources and the collection used, that pre-request and post-response scripts reduce how many users you can simulate, and that the run stops if the machine passes its resource threshold. They suggest a first test of 10 to 20 virtual users (checked on 13 October 2026). Past that, the load generator itself can distort your numbers, which is why Postman also reports the peak CPU and memory of the test machine.

Data is the next limit. CSV and JSON data files work for local runs only, while datasets work for local and cloud runs, according to the same configuration page.

How much does Postman load testing cost?

Local performance runs are included in every plan. Cloud runs are for paid plans and are billed by usage in virtual user hours.

Postman's pricing page lists unlimited local performance testing and cloud performance testing at $0.04 per virtual user hour (checked on 13 October 2026). At that listed rate, 500 virtual users for one hour is 500 virtual user hours, or $20, on top of the plan fee. Static IP addresses for reaching targets behind a firewall are an Enterprise feature. We did not price the plans themselves, so check the page for the current per-user fees.

Which protocols can you load test?

JMeter has a long, documented list. For Postman, we found no list of protocols supported in performance runs, so we make no claim about them.

JMeter's home page lists HTTP and HTTPS, SOAP and REST, FTP, databases through JDBC, LDAP, JMS messaging, SMTP, POP3 and IMAP mail, native commands and shell scripts, TCP and Java objects (checked on 13 October 2026). Postman handles other request types in the app, such as GraphQL, gRPC and WebSocket, but that does not tell you which of them run under load. Test one before you rely on it.

How do they scale beyond one machine?

Postman scales through its own cloud, JMeter through servers you run yourself. The first is easier to start, the second gives you control.

On a paid plan, Postman's cloud runs split the load across managed workers, so one machine no longer limits the test. JMeter's remote testing mode lets one client drive many JMeter servers. Every node needs the same JMeter version, a keystore for RMI over SSL unless you disable it, and its own copy of any data files (checked on 13 October 2026).

There is a third route for JMeter test plans. LoadFocus, which publishes this site, runs existing JMeter test plans from 25 cloud regions. You upload the .jmx file and set virtual users, duration and ramp-up, with no remote servers to configure. LoadFocus runs JMeter and k6 only, so it does not run a Postman collection.

Which one should you choose?

Choose Postman instead of JMeter for functional and contract checks, for a team that shares collections in a workspace, and for a quick smoke run of an API you already test. Choose JMeter when load testing is the main goal, or when you need several protocols, large user counts, or tests that stay in your own repository.

Postman fits a short check in CI: postman performance run exits with a non-zero code when a threshold such as the 95th percentile is missed, which fails the build. JMeter is the better choice for database or messaging load, for long soak tests, and for the HTML dashboard with APDEX and percentile tables that its docs describe.

Many teams use both: Postman to build and verify the API checks, JMeter to put real volume behind the critical flows.

FAQ

Can Postman replace JMeter for load testing? For small checks on HTTP APIs it can, because the collection you already have is the test. For high concurrency, other protocols or long runs, the documented limits on local runs and the usage pricing of cloud runs make JMeter the safer base.

Can Postman run a JMeter .jmx file? We found nothing in its documentation that imports a JMeter test plan. Postman performance tests are built from collections, so a .jmx file would have to be rebuilt.

Is JMeter free? Yes. The current release on the download page is 5.6.3, which needs Java 8 or later (checked on 13 October 2026). You pay for the machines that generate load, or for a cloud service that runs your plans.

k6 vs Locust: JavaScript scripts or Python users?

Tue, 13 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

k6 and Locust are both open source tools that keep a load test as code in your repository, and either one will load test an HTTP API well. Pick Locust if your team writes Python and wants plain Python for test logic. Pick k6 if you write JavaScript, want pass/fail thresholds built in, or need a test that holds an arrival rate.

The facts in this k6 vs Locust comparison come from each project's own documentation and repository, checked on 13 October 2026.

	k6	Locust
Licence	AGPL-3.0 (repository)	MIT (repository)
Runtime	One Go binary with an embedded JavaScript engine (release v2.3.0)	Python 3.11 or later (release 2.46.7)
Test scripts	JavaScript, or TypeScript with the types stripped	Python classes in a locustfile
Built-in protocols	HTTP/1.1, HTTP/2, WebSockets, gRPC	HTTP and HTTPS
Load model	Six executors, two of them arrival-rate (open model)	A pool of users, each pausing for a wait time (closed model)
Pass/fail	Thresholds, which can abort a run early	Exit code 1 on failed requests; other limits via a listener
Browser testing	Built-in browser module	Browser users via the separate locust-plugins project
Scaling out	k6-operator on Kubernetes, or Grafana Cloud k6	Master and workers, a Kubernetes operator, or Azure Load Testing

What is the difference between k6 and Locust?

The main difference is the language you write tests in. k6 runs JavaScript in a compiled Go program, while Locust runs your Python code and builds on gevent for many concurrent users.

k6 installs as a package or a standalone binary with no other runtime. It also runs .ts files, though its TypeScript support strips the types without checking them. Locust installs with pip install locust, and its What is Locust page says tests are plain Python that you keep in version control like any other code (checked on 13 October 2026).

How do you write the same test in each tool?

Both express the test as a short program. Here is a test with 50 users for five minutes that fails if the 95th percentile tops 500 ms or more than 1% of requests fail.

In k6 the goals are part of the script, as thresholds:

import http from "k6/http";
import { sleep } from "k6";

export const options = {
  vus: 50,
  duration: "5m",
  thresholds: {
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  http.get("https://staging.example.com/api/users");
  sleep(1);
}

In Locust the user is a class, and the same goals go in a listener, following the pattern in the headless mode docs:

import logging
from locust import HttpUser, between, events, task


class ApiUser(HttpUser):
    wait_time = between(1, 2)

    @task
    def list_users(self):
        self.client.get("/api/users")


@events.quitting.add_listener
def _(environment, **kw):
    stats = environment.stats.total
    if stats.fail_ratio > 0.01:
        environment.process_exit_code = 1
    elif stats.get_response_time_percentile(0.95) > 500:
        environment.process_exit_code = 1

Run them with k6 run script.js and locust -f locustfile.py --headless -u 50 -r 10 -t 5m --host https://staging.example.com (checked on 13 October 2026).

How do k6 and Locust model load?

k6 supports both workload models, while Locust describes load as a fixed number of users. In k6 you choose the model through the executor, and in Locust through the user count and wait time.

k6 has six executors. Four are closed-model, where a virtual user starts its next iteration only when the last one finishes, and the two arrival-rate executors are open-model, where iterations start independently of how long earlier ones take. Locust's locustfile guide says wait time can only constrain throughput and cannot launch new users to reach a target rate (checked on 13 October 2026).

If your goal is "300 requests arrive every second whatever the response time", k6 can express that directly. Our guide to the open vs closed workload model explains when each model fits.

How do k6 and Locust decide whether a test passed?

k6 has pass/fail criteria as a core feature, and Locust leaves most of it to your code. A failed k6 threshold ends the run with a non-zero exit code, and abortOnFail stops the test the moment it fails.

Locust exits with code 1 if any request failed. For limits on the failure ratio, the average or the 95th percentile, its docs show a listener like the one above that sets the exit code yourself (checked on 13 October 2026). Our guide to pass/fail criteria for load tests covers how to pick the limits.

Which protocols does each tool cover?

k6 ships four protocols and a browser module, while Locust ships HTTP and HTTPS and lets you wrap any Python client for the rest.

k6 lists HTTP/1.1, HTTP/2, WebSockets and gRPC as built-in protocols, with more through xk6 extensions, and a browser module that drives a Chromium-based browser. Locust's page on other systems says it comes with built-in support only for HTTP/HTTPS but can be extended to test almost any system, by wrapping a protocol library and reporting each call as a request. The library has to cooperate with gevent (checked on 13 October 2026).

How do you run a bigger test than one machine can generate?

Locust scales with a master and workers, and k6 with a Kubernetes operator or a hosted service. Both can split one test across many machines.

The Locust distributed docs say one process can make more than a thousand requests per second for a simple test plan, so larger tests run one worker per processor core across as many machines as needed. The k6-operator splits one test across several k6 instances in a Kubernetes cluster, and Grafana Cloud k6 is the hosted route (checked on 13 October 2026).

There is a third route for k6 scripts. LoadFocus, which publishes this site, runs k6 scripts from 25 cloud regions, with the users and duration taken from the test's settings. Its script validator warns that arrival-rate scenarios run there with a fixed number of users, so an open-model script does not hold its arrival rate on LoadFocus. LoadFocus does not run Locust: it runs k6 scripts and JMeter test plans only.

What reports do you get?

k6 prints an end-of-test summary and can export an HTML report from its web dashboard. Locust shows live charts in its web UI and writes files on request.

k6's end-of-test summary goes to the terminal, and its web dashboard exports a self-contained HTML report when you set K6_WEB_DASHBOARD_EXPORT. Locust serves a web UI on port 8089 with live charts, and its configuration reference lists --csv, --html and --json output (checked on 13 October 2026).

Which one should you choose?

Locust fits Python teams, and k6 fits teams that want built-in thresholds and an open workload model. Protocols and scaling settle most other cases.

Choose Locust instead of k6 when your team writes Python and wants loops, branches, generated data and its own API client in plain Python, when you want to reuse Python libraries for a protocol, or when you want a web UI with live charts to explore a system. Choose k6 instead of Locust when your team writes JavaScript or TypeScript, when thresholds should fail a CI build and stop a bad run early, when you need an arrival rate held constant, or when you want gRPC and browser tests in the same tool.

For a wider view, see our review of Locust load testing.

FAQ

Is Locust faster than k6? We have not measured this ourselves, so we will not name a winner. The fair test is the same scenario with both tools on your own hardware.

Can Locust run k6 scripts? Not according to the Locust docs, which describe tests as Python code in a locustfile. A k6 script is JavaScript, so it has to be rewritten as a Python user class, and the thresholds become a listener.

Can LoadFocus run Locust tests? No. LoadFocus runs k6 scripts and JMeter test plans. To scale a Locust test, use its master and worker mode, the Kubernetes operator or Azure Load Testing.

Gatling load testing: a review of the JVM tool

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Gatling is a free, Apache 2.0 load testing tool in which every test is a simulation written as code and run on the JVM, with an HTML report after each run. Gatling load testing suits developer teams who want tests in the repository, open or closed workload models and assertions that fail the build. Teams that want a GUI test builder or many protocols are better served by JMeter, and teams that want one binary with no JVM by k6.

We have not run Gatling for this review. The facts come from Gatling's own documentation, repository and release pages, checked on 9 October 2026, and the judgements are ours.

Topic	Gatling
Licence	Apache 2.0, except the gRPC, MQTT and GraphQL plugins
Runtime	JVM, 64-bit OpenJDK LTS 11 to 25
Current release	3.16.0 (29 September 2026)
Tests written in	Java, Kotlin, Scala, JavaScript, TypeScript
Built-in protocols	HTTP/1.1, HTTP/2, WebSocket, SSE, JMS
Results	HTML report per run
Build and CI	Maven, Gradle, sbt, npm plugins
Scaling out	Gatling Enterprise (commercial)

Who makes Gatling, and is it still maintained?

Gatling Corp, a French company, develops Gatling and owns its copyright. It is actively maintained, with three JVM releases so far in 2026.

The licences page gives Gatling Corp the copyright. Maven Central lists 3.15.0 in February, 3.15.1 in May and 3.16.0 on 29 September 2026, and the JavaScript SDK reached 3.16.1 on 6 October. The 3.16 release notes add GraphQL and shared queues between virtual users. The README calls the repository the Community Edition, free to use including commercially (checked on 9 October 2026).

What do you need to run Gatling?

A 64-bit JVM for Java, Kotlin or Scala, or Node.js for JavaScript and TypeScript. Either way, the simulation itself runs on a JVM.

The installation page supports OpenJDK LTS versions 11 to 25 and excludes 32-bit JVMs and OpenJ9. The JavaScript CLI needs Node.js 24 or later and downloads a Gatling runtime bundle and, per the gatling-js README, GraalVM (checked on 9 October 2026).

How is a Gatling simulation structured?

A simulation defines scenarios, gives each one an injection profile and attaches a protocol configuration, all registered in one setUp call.

In the simulation reference, a scenario chains requests with pauses, loops and conditions. This Java simulation saves a product id from a search, pauses one to three seconds and opens the product, while arrivals ramp to 20 users per second and hold for ten minutes:

public class ShopSimulation extends Simulation {

  HttpProtocolBuilder api =
      http.baseUrl("https://staging.example.com");

  ScenarioBuilder shoppers = scenario("Shoppers")
      .exec(http("search").get("/api/products?q=lamp")
          .check(jsonPath("$[0].id").saveAs("id")))
      .pause(1, 3)
      .exec(http("product").get("/api/products/#{id}"));

  {
    setUp(shoppers.injectOpen(
        rampUsersPerSec(1).to(20).during(120),
        constantUsersPerSec(20).during(600)))
      .protocols(api)
      .assertions(
          global().responseTime().percentile(95.0).lt(800));
  }
}

Imports are left out, and #{id} is Gatling's Expression Language reading the saved value. Gatling vs k6 shows a JavaScript simulation.

Should you inject users open or closed?

Open, for most websites and APIs, where new users keep arriving however slow the system gets. Closed suits systems that cap concurrency, such as a ticket queue.

The injection reference splits the API: injectOpen takes arrival rates such as rampUsersPerSec, and injectClosed takes concurrency targets such as constantConcurrentUsers. The workload models page says most websites are open systems and one profile cannot mix the two (checked on 9 October 2026). Our open vs closed workload model guide explains the choice.

How do checks, assertions and the report work?

Checks verify each response and can save values from it, while assertions judge the whole run once it ends. Every run then writes an HTML report.

Without an explicit status check, Gatling checks that each HTTP response is 2XX or 304. Assertions test response times, request counts or requests per second, globally or per request, and one failure fails the simulation. The HTML report charts response time ranges and percentiles, users started per second, concurrent users and requests per second; its percentiles over time count successful requests only (checked on 9 October 2026).

How does Gatling fit into a build and CI?

Through its Maven, Gradle, sbt or npm plugin, which runs in any CI system. A failed assertion fails the build.

The Maven plugin binds gatling:test to the integration-test phase, so mvn verify runs it, and with CI=true it skips the simulation prompt (the run then fails if the project has more than one simulation and none is named). The Gradle plugin uses gatlingRun, and the sbt plugin takes Scala simulations only. Gatling's CI integrations, such as its GitHub Action, start Gatling Enterprise runs (checked on 9 October 2026). Our GitHub Actions guide covers the workflow.

Which protocols does Gatling support?

HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events and JMS ship without limits. gRPC, MQTT and GraphQL are separate plugins capped at 5 users and 5 minutes outside Gatling Enterprise.

The README puts the first five under Apache 2.0. The gRPC, MQTT and GraphQL setup pages put each plugin under the Gatling Enterprise Component License: unlimited on Gatling Enterprise, while a Community Edition test stops at 5 users or 5 minutes. That licence grants free use for up to one year, only for internal testing of your own web applications; it ends as soon as a limit is reached, and Gatling may change the limits. GraphQL needs 3.16.0 or later. Its release notes and the README say the JavaScript SDK does not support it yet, although the setup page already lists an npm package, so check before planning a JavaScript GraphQL test (checked on 9 October 2026). Our guide on how to load test a GraphQL API works around the cap with plain HTTP requests.

What does Gatling Enterprise add?

Gatling Enterprise is the commercial edition on the same engine. It generates load from managed or private infrastructure, which the free edition does not.

Per the README, Enterprise runs tests on managed load generators or in your own AWS, Azure, GCP or Kubernetes accounts, and adds live dashboards, trends, teams with SSO and CI/CD integrations. The FAQ says the Community Edition does not support distributed testing and that simulations move to Enterprise unchanged (checked on 9 October 2026).

LoadFocus, which publishes this site, does not run Gatling simulations. It runs JMeter test plans and k6 scripts from 25 cloud regions, so for Gatling the route to more machines is Gatling Enterprise.

What does Gatling do well, and where are its limits?

It is strong on tests as reviewable code, on load models and on reports with no setup. It is weaker on protocols beyond the web and messaging, and on free scaling past one machine.

The README credits its engine: non-blocking, built on Netty, with each virtual user a lightweight message. Limits the docs state themselves (checked on 9 October 2026):

Gatling is not a browser: per the HTTP protocol page, it runs no JavaScript.
The JavaScript SDK has no JMS and cannot use npm libraries that need native binaries or Node.js APIs.
HTTP caching is on by default, which can lower request counts.
The simulation.log format is undocumented and can change at any time.

When should you choose k6, JMeter or Locust instead?

Choose another tool when its language, protocols or free scaling route fit your team better.

Our open source load testing tools overview covers five others.

Choose k6 instead when you want one binary in CI with no JVM, thresholds that can stop a failing run early, or gRPC among its built-in protocols (checked on 9 October 2026). Gatling vs k6 compares the two.

Choose JMeter instead when testers build tests in a GUI, or you need JDBC, LDAP or FTP from its core (checked on 9 October 2026). LoadFocus runs existing JMeter test plans from 25 cloud regions, and Gatling vs JMeter goes deeper.

Choose Locust instead when your team writes Python and wants a distributed mode in the free tool (checked on 9 October 2026). Our Locust review covers it.

FAQ

Is Gatling free for commercial use? Yes. The Community Edition is Apache 2.0. The gRPC, MQTT and GraphQL plugins have their own licence and stop at 5 users and 5 minutes outside Gatling Enterprise.

Do you need Java to use Gatling? For Java, Kotlin or Scala, yes: a 64-bit OpenJDK LTS from 11 to 25. The JavaScript SDK needs Node.js 24 or later, and its CLI downloads the JVM runtime itself.

Can LoadFocus run Gatling tests? No. LoadFocus runs JMeter test plans and k6 scripts. To spread a Gatling simulation across several machines, use Gatling Enterprise.

Gatling vs JMeter: tests as code or a GUI test plan?

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Gatling and Apache JMeter are both free, open source load testing tools that run on the JVM, and the choice comes down mostly to how your team wants to build tests. Pick Gatling if you want tests as code in Java, Kotlin, Scala, JavaScript or TypeScript, open injection profiles and an HTML report after every run. Pick JMeter if you need protocols such as JDBC, LDAP, FTP or mail out of the box, prefer to build tests in a GUI, or already own .jmx test plans.

The facts in this Gatling vs JMeter comparison come from each project's own documentation and repository, checked on 9 October 2026.

	Gatling	Apache JMeter
Licence	Apache 2.0; gRPC, MQTT and GraphQL under a separate component licence (repository)	Apache 2.0 (repository)
Runtime	JVM, OpenJDK LTS 11 to 25 (release 3.16.0)	JVM, Java 8 or later (release 5.6.3)
Tests written as	Code in Java, Kotlin, Scala, JavaScript or TypeScript	Test plans built in a GUI, saved as XML `.jmx` files
Built-in protocols	HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events, JMS	HTTP(S), SOAP/REST, FTP, JDBC, LDAP, JMS, mail, TCP, shell commands, Java objects
Recorder	HTTP proxy or HAR file converter	HTTP(S) Test Script Recorder (proxy)
Load model	Open or closed injection profiles	Thread groups; an experimental Open Model Thread Group
Scaling out	One machine; distributed runs need Gatling Enterprise	Built-in remote mode, one client driving many servers
CI and headless runs	Maven, Gradle, sbt and npm plugins; a failed assertion fails the run	CLI mode (`jmeter -n`); third-party Maven, Gradle and Jenkins plugins
Reports	HTML report after each run	HTML dashboard; live results through a Backend Listener

What is the difference between Gatling and JMeter?

In Gatling a test is a program in a build project. In JMeter a test is a tree of elements you assemble in a desktop GUI and save as an XML file, and most other differences follow from that.

Gatling's README calls the tests simulations and keeps them in your repository like application code, run through a Maven, Gradle, sbt or npm plugin. The JavaScript and TypeScript SDK installs through npm and downloads its own GraalVM runtime.

A JMeter plan holds a thread group for the users, samplers for the requests, and assertions and listeners for the results. JMeter 5.6.3, the current release on the download page, requires Java 8 or later. The README on JMeter's development branch already calls for Java 17 (checked on 10 October 2026). Underneath, JMeter runs each virtual user as a Java thread, while Gatling's README describes them as lightweight messages on a non-blocking Netty engine.

How do you write a test in each tool?

In Gatling you write a simulation class that sets up a scenario, an injection profile and assertions. In JMeter you add a thread group and samplers in the GUI, save the plan, then run it from the command line.

A Java simulation modelled on Gatling's first simulation tutorial: new users arrive at one per second, rising to ten per second over a minute and holding there for five minutes. It fails if the 95th percentile tops 500 ms or more than 1% of requests fail:

import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;

import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;

public class ApiSimulation extends Simulation {

  HttpProtocolBuilder httpProtocol =
      http.baseUrl("https://staging.example.com");

  ScenarioBuilder users = scenario("API users")
      .exec(http("list users").get("/api/users"))
      .pause(1);

  {
    setUp(users.injectOpen(
            rampUsersPerSec(1).to(10).during(60),
            constantUsersPerSec(10).during(300)))
        .protocols(httpProtocol)
        .assertions(
            global().responseTime().percentile(95.0).lt(500),
            global().failedRequests().percent().lt(1.0));
  }
}

Run it with ./mvnw gatling:test; a JavaScript or TypeScript simulation runs with npx gatling run.

In JMeter the same test is a thread group holding an HTTP Request sampler, and a Duration Assertion can mark each slow response as failed. The getting started guide builds the plan in GUI mode and runs it in CLI mode, with -e -o adding the HTML report:

jmeter -n -t api-users.jmx -l results.jtl -e -o report/

Both tools can record a first draft from a browser. Gatling's Recorder works as an HTTP proxy or converts a HAR file into a simulation, and JMeter's HTTP(S) Test Script Recorder captures browser traffic into a plan (checked on 9 October 2026). Gatling's docs now list its Recorder as legacy, and the newer Gatling Studio needs an Enterprise account to log in (checked on 10 October 2026).

Which protocols can each tool test?

JMeter covers more protocols out of the box. Gatling ships with the main web protocols plus JMS, and leaves most of the rest to separate components or community plugins.

JMeter's README lists every protocol in the table above. JDBC and JMS need driver jars from your database or JMS provider, and the Plugins Manager adds community plugins from inside the GUI.

Gatling's README says gRPC, MQTT and GraphQL are separate components under the Gatling Enterprise Component License, capped at five concurrent users and five-minute tests without Enterprise. Kafka, AMQP, SFTP, FTP and JDBC come from third-party plugins for Gatling 3.13 that Gatling does not maintain (checked on 9 October 2026).

How do Gatling and JMeter model load?

Gatling makes you choose between an open and a closed model in the script itself. JMeter's classic thread group is a closed pool of users, and arrival-based load needs the newer Open Model Thread Group.

In Gatling, injectOpen takes arrival rates such as constantUsersPerSec and rampUsersPerSec, and injectClosed takes concurrency targets such as constantConcurrentUsers. JMeter 5.5 added the Open Model Thread Group, which the reference still marks as experimental (checked on 9 October 2026). Our guide to the open vs closed workload model explains why that choice matters.

How do you run a test across several machines?

JMeter has a distributed mode built in. Gatling's open source edition generates load from one machine, and its docs list a Gatling Enterprise account as a requirement for distributed runs.

JMeter's remote testing mode lets one client control any number of servers. Each server runs the whole plan, so 1,000 threads on six servers means 6,000 threads. All nodes should run the same JMeter and Java versions, RMI needs a keystore unless SSL is disabled, and data files must be copied to each server.

There is a third route for JMeter test plans. LoadFocus, which publishes this site, runs existing JMeter test plans unchanged from 25 cloud regions, with no remote servers to set up. LoadFocus does not run Gatling simulations.

How do they fit into CI, and what reports do you get?

Both run headless. Gatling's assertions can fail the build directly, while JMeter's assertions flag individual samples and CI support comes from third-party libraries.

Gatling assertions are checked after the run, and one failed assertion fails the simulation and the build task. The HTML report shows response time percentiles, users started per second, concurrent users and requests per second over time.

JMeter's CLI mode has no pass/fail option, so a missed target fails the build only through the Maven, Gradle and Jenkins libraries its README lists, or your own check. JMeter's dashboard adds an APDEX table, a statistics table with three configurable percentiles and error summaries, and a Backend Listener streams live results to InfluxDB or Graphite.

Which one should you choose?

Choose Gatling when your team writes Java, Kotlin, Scala or TypeScript and wants load tests reviewed like the rest of its code. Choose JMeter instead when you need protocols beyond the web, when your testers prefer a GUI, or when you already have .jmx plans.

Gatling suits teams that describe traffic as arrival rates, want pass/fail rules next to the scenario and a readable report with no extra setup.

JMeter is the better choice for JDBC, LDAP, FTP or mail testing, for testers who refine a recorded flow visually, and for distributed runs on your own servers without a commercial licence. A company that already runs JMeter also keeps its plans, plugins and know-how.

FAQ

Is Gatling faster than JMeter? We have not measured this ourselves, so we will not name a winner. The fair test is the same scenario with both tools on your own hardware.

Can Gatling run JMeter .jmx files? Not directly. Gatling runs simulations written with its SDKs, so a JMeter plan has to be rewritten. Gatling's README lists AI extensions that convert JMeter test plans and require a Gatling Enterprise API token.

Can LoadFocus run Gatling tests? No. LoadFocus runs JMeter test plans and k6 scripts. To spread a Gatling simulation across several machines, use Gatling Enterprise.

Coordinated omission: what it is and how to avoid it

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Coordinated omission is a measurement error in which a load generator waits for each slow response before sending its next request, so it sends fewer requests just when the system is slow and records too few of the bad moments. The percentiles you read afterwards then look better than what users lived through. You avoid it by sending requests on a schedule that ignores response times, and by timing each request from the moment it was due to be sent.

The tool facts below come from each project's own documentation or repository, checked on 9 October 2026.

Tool	What it offers	What to check
wrk2	A constant rate (`-R`), latency timed from the planned send	`--u_latency` adds the uncorrected view
HdrHistogram	Two methods that back-fill missing samples	Apply one of them, never both
k6	Arrival-rate executors	The `dropped_iterations` counter
Gatling	`injectOpen` steps such as `constantUsersPerSec`	The steps sit inside `injectOpen`
Locust	`constant_throughput` paces each user	The achieved rate against the target
JMeter	Open Model Thread Group (5.5+, experimental)	A plain Thread Group is a closed pool

What is coordinated omission?

It is the error you get when the measuring side stops taking samples because the system is slow, so the samples that go missing are exactly the slow ones. Losing samples at random barely moves a percentile; losing them in step with a slowdown skews all of them.

The name comes from Gil Tene, whose 2013 mailing list post on the problem draws this same line. He is the original author of HdrHistogram, wrk2 lives in his GitHub account, and both projects' docs explain the effect (checked on 9 October 2026). The k6 docs use the term for a closed workload model that slows its own load down, a common cause; our open vs closed workload model article covers that choice, so this one stays with the measurement.

How does a load test end up hiding slow responses?

A closed-loop generator sends the next request on a connection only after the previous response arrives. When one response takes a second, that connection sends nothing for a second, and the requests it would have sent never get a response time.

The wrk2 README says this of wrk's model, which times each request from first byte sent to full response: since a connection only sends after a response, the generator coordinates with the server and avoids measuring during high-latency periods (checked on 9 October 2026).

HdrHistogram's README offers a thought experiment. A system sampled every 10 ms answers in 1 ms for 100 seconds, then stalls for 100 seconds, yielding one 100-second sample. The raw histogram shows about 99.99% of results at 1 ms or below; corrected for the blocked samples, only about half are.

How can you tell if your results have coordinated omission?

Compare the rate you sent with the rate you planned. If the request rate fell while response times rose, the generator backed off and the slow period is under-sampled.

The load was set as users or connections with no rate. Any closed loop can do this.
The request rate dips where response times climb. A dip in the rate during a spike means requests went unsent.
The maximum dwarfs the 99th percentile. A few huge samples among thousands of fast ones is the example above in miniature. Read the whole response time distribution, knowing it only counts requests that were sent.
The load generator was short of CPU. LoadFocus documents that an engine near 100% CPU cannot send at the configured rate, and its results show a load generator saturation warning when an engine peaked at or above 80% CPU.

How does wrk2 avoid coordinated omission?

It sends at a constant rate and times each request from when the plan said it should go out. A request that leaves late because its connection was busy is charged for the wait.

wrk2 adds a rate argument (-R, total requests per second) to wrk and records latencies in an HdrHistogram. --u_latency adds the uncorrected histogram, labelled CO, to the report, as in the README's own example:

wrk -t2 -c100 -d30s -R2000 --u_latency http://127.0.0.1:80/index.html

In the README's second example, a 1.4-second server pause during a 30-second run made the corrected 99th percentile 200 times the uncorrected one. The README also calls the project experimental and warns that runs under 10 to 20 seconds may not be useful (checked on 9 October 2026). Our HTTP benchmarking tools review compares wrk2 with hey, oha and others.

How does HdrHistogram correct recorded latencies?

If you know the interval at which requests were meant to go out, it adds the samples that a long response blocked. You can correct while recording or afterwards, on a copy.

recordValueWithExpectedInterval records a value and, when it exceeds the interval, adds values stepping down by that interval. copyCorrectedForCoordinatedOmission returns a corrected copy of a finished histogram. The JavaDoc calls the two mutually exclusive (checked on 9 October 2026).

// planned interval: 10 ms, values in microseconds
Histogram h = new Histogram(3);
h.recordValueWithExpectedInterval(latencyUs, 10_000);

// or, after the run, on a raw histogram:
Histogram fixed = raw.copyCorrectedForCoordinatedOmission(10_000);

The correction is only as good as the interval you pass in, so it suits a generator that was meant to send at a steady pace.

How do k6, Gatling, Locust and JMeter handle it?

k6, Gatling and JMeter can start work on a schedule that does not wait for responses. Locust paces each user but cannot start extra users to hold a rate.

k6. In the arrival-rate executors, iterations start independently of system response while a virtual user is free. When none is, k6 skips the iteration and counts it in dropped_iterations, which late in a test may mean the system is degrading (checked on 9 October 2026). A dropped iteration has no response time, so read that counter next to the percentiles. On LoadFocus, k6 scripts run arrival-rate scenarios with a fixed number of users, a closed model, so run k6 yourself when the arrival rate is the point.

Gatling. Steps such as constantUsersPerSec inside injectOpen start new virtual users at a set rate, whether or not earlier ones have finished (checked on 9 October 2026).

Locust. constant_throughput adapts each user's wait so it runs a task at most a set number of times per second. The docs say wait time can only limit the rate and cannot launch new users, and the API reference adds that a task which overruns its interval gets no wait afterwards, so a slow response still delays that user (checked on 9 October 2026).

JMeter. A Thread Group is a pool of looping threads, a closed loop. JMeter 5.5 added the experimental Open Model Thread Group, which follows a schedule such as rate(1/sec) random_arrivals(2 min) and, per the release notes, spawns threads as needed (checked on 9 October 2026).

Does an open workload model remove coordinated omission?

It removes the main cause, a generator that waits before sending. Two gaps remain: a generator short of users or CPU still falls behind its schedule, and each user in a multi-step journey still waits for its own responses.

The first gap shows up as dropped iterations or a saturated engine; either is a reason to rerun with more headroom. The second is built into most HTTP testing: the wrk2 README notes that sending without waiting only suits non-blocking protocols or one request per connection, which is why wrk2 corrects the timing instead.

FAQ

Who named coordinated omission? Gil Tene, the original author of HdrHistogram, whose wrk2 fork of wrk is built around avoiding it. His 2013 post to the mechanical-sympathy mailing list describes it as a problem he calls by that name.

Is coordinated omission the same as a closed workload model? They are related. A closed model is a common cause, since each virtual user waits for its response before sending again, and the wrk2 README also finds the effect in monitoring code.

Should you correct every latency histogram for it? Only when you know the interval at which requests were planned, because HdrHistogram needs it to fill in the missing samples. Apply one of its two correction methods to a data set, never both.

How to load test a GraphQL API: a practical guide

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

To load test a GraphQL API, script the operations your clients really send, give each one its own name in the test tool and mix them in the proportions you see in production, with varied variables. Check every response body for an errors array, because a GraphQL server usually answers with HTTP 200 even when part of an operation failed. Then ramp up the load and judge each operation by its own percentiles.

Sources: graphql.org, the draft GraphQL over HTTP specification and each tool's own documentation, all checked on 9 October 2026.

Tool	Name each operation with	Fail a response that has `errors`	Feed variables from
k6	`tags: { name: "..." }`	A `check()` on the parsed body	`SharedArray`
JMeter	The sampler name	JSON Assertion on `$.errors`, inverted	CSV Data Set Config
Gatling	`http("...")` or the GraphQL plugin	`jsonPath("$.errors").notExists()`	Feeders
Artillery	A request `name`	`notHasProperty: "errors"`	`config.payload`

Why does a GraphQL API need a different load test?

Because every operation goes to one URL, and the work behind it depends on the query and its variables. A report grouped by URL puts a cheap lookup and an expensive search in the same bucket.

graphql.org's Serving over HTTP guide says a GraphQL server operates on a single endpoint, usually /graphql. Operations arrive there by POST with a JSON body holding query, an optional operationName and variables.

How do you get results per operation?

Name every request after its GraphQL operation and make the tool group its statistics by that name. Each tool has its own switch for this (checked on 9 October 2026).

k6: the tags and groups guide says to set tags.name to give a request a meaningful name, and thresholds can target one tag, such as http_req_duration{name:GetProduct}.
JMeter: the Aggregate Report creates one row per differently named request. Since JMeter 5.4 the GraphQL HTTP Request sampler has Query, Variables and Operation Name fields.
Gatling: the request name is the key for its statistics, and the GraphQL plugin names requests after the operation, such as query GetUser.
Artillery: the metrics-by-endpoint plugin reports per URL and adds a request's name to its entry, so named operations are reported separately; useOnlyRequestNames: true drops the URL and keeps the name alone.

Which operations should the test send?

The ones production receives, in the same proportions. Take the operation counts from your server logs or GraphQL monitoring and turn them into a weighted mix of reads and mutations.

Use the documents your clients ship, at their real depth, rather than one simplified query. Feed variables from a data file, so the server looks up many different records instead of returning one cached result. Keep the number of distinct operations close to reality too: Apollo's technote on load testing GraphQL, a vendor document, explains that each new operation is parsed and validated before it runs, and warns against generating far more unique operations than real clients send.

Mutations change data, so run them where the data can be reset. The step-by-step load testing guide covers goals and ramp-up.

Why can an HTTP 200 response still be a failure?

Because GraphQL reports errors inside the response body. A request can return status 200 with an errors array and partial data, and a status check passes it.

graphql.org's Response page explains that when a resolver fails, execution continues and the response carries data alongside errors. Its HTTP guide adds that such a response should still get a 2xx status, since HTTP has no code for partial success, and that some legacy servers answer 2xx even for validation errors. The GraphQL over HTTP draft proposes a custom 294 status for partial success with the newer application/graphql-response+json media type.

In a load test this hurts twice. Errors go uncounted, and requests that fail fast, such as operations rejected by a depth limit, pull the response times down.

How do you check the errors array in each tool?

Parse the JSON body and fail the request when errors is present. Each tool has a built-in way.

In k6, a check on the parsed body does it, and a threshold on the checks metric fails the run, because checks alone do not change k6's exit code:

import http from "k6/http";
import { check, sleep } from "k6";
import { SharedArray } from "k6/data";

const URL = "https://staging.example.com/graphql";
const ids = new SharedArray("ids", () =>
  JSON.parse(open("./product-ids.json")));
const QUERY = `query GetProduct($id: ID!) {
  product(id: $id) { id name price }
}`;

export const options = {
  vus: 20,
  duration: "5m",
  thresholds: {
    checks: ["rate>0.99"],
    "http_req_duration{name:GetProduct}": ["p(95)<400"],
  },
};

function parse(res) {
  try { return res.json(); } catch (e) { return null; }
}

export default function () {
  const id = ids[Math.floor(Math.random() * ids.length)];
  const payload = JSON.stringify({
    query: QUERY,
    operationName: "GetProduct",
    variables: { id: id },
  });
  const res = http.post(URL, payload, {
    headers: { "Content-Type": "application/json" },
    tags: { name: "GetProduct" },
  });
  const body = parse(res);
  check(res, {
    "status is 2xx": (r) => r.status >= 200 && r.status < 300,
    "no GraphQL errors": () => body !== null && !body.errors,
  });
  sleep(1);
}

SharedArray loads the IDs once for all virtual users. The try stops a body that is not JSON, such as a proxy's error page, from ending the iteration.

In JMeter, add a JSON Assertion with the path $.errors and tick "Invert assertion", so the sample fails when the path exists. A JSON Extractor copies IDs into variables for the next operation.

In Gatling, add .check(jsonPath("$.errors").notExists()), both from its checks reference. The GraphQL plugin checks the status and the errors by default, but its setup page limits Community Edition runs to 5 users and 5 minutes (checked on 9 October 2026). For bigger tests without Gatling Enterprise, use plain HTTP requests with the check.

In Artillery, the expect plugin offers notHasProperty: "errors", and reportFailuresAsErrors: true puts failed expectations in the report. A capture with a json path carries a value into the next request.

What should you watch besides response times?

Caching, persisted queries and database load per request. Each can make a test faster or slower than production for reasons unrelated to capacity.

graphql.org's performance guide notes that queries sent by GET can use HTTP and CDN caching, and that persisted queries let a client send a hash instead of the full document. If your clients do that, the test should too. If they do not, a test repeating one query with the same variables measures a cache that real traffic misses. Apollo's technote adds that Apollo Server, Gateway and the GraphOS Router enable automatic persisted queries by default.

The same guide describes the N+1 problem: a list whose resolver loads each item separately turns one request into many database queries, usually fixed by batching with a tool like DataLoader. Watch database queries per second next to requests per second; if they climb much faster as the load rises, suspect a resolver. Depth limits, complexity limits and rate limiting can also trip during a test.

Where should the load come from?

From one machine while you build and smoke test the script, then from the cloud when you need more load or traffic from several regions.

LoadFocus runs k6 scripts from 25 cloud regions. There, the number of users and the duration come from the test settings rather than the script's options, and arrival-rate scenarios run with a fixed number of users instead of a fixed request rate, the difference covered in open vs closed workload models. LoadFocus also publishes a template for GraphQL endpoints with complex queries.

FAQ

Should a GraphQL load test send queries by GET or POST? Send them the way your clients do. graphql.org allows GET for queries only, and mutations must use POST. If production sends persisted queries by GET through a CDN, test that path too.

Can the same script load test GraphQL subscriptions? Usually not. graphql.org notes that subscriptions are long-lived and often run over WebSockets or server-sent events, so they need their own scenario in a tool that supports that transport.

Should a partial response count as a failed request? In most tests, yes, because the client got an error for part of its request. If some errors are expected, such as not-found results for deleted records, match those error codes and fail on the rest.

Gatling vs k6: JVM simulations or a single Go binary?

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Gatling and k6 are both open source tools that keep load tests as code in your repository, and either one will load test an HTTP API well. Pick Gatling if your team works in Java, Kotlin or Scala, or wants an HTML report after every run with no extra setup. Pick k6 if you want one Go binary with no JVM, gRPC and browser tests out of the box, and thresholds that can stop a failing run early.

The facts in this Gatling vs k6 comparison come from each project's own documentation and repository, checked on 9 October 2026.

	Gatling	k6
Licence	Apache 2.0; gRPC, MQTT and GraphQL under a separate component licence (repository)	AGPL-3.0 (repository)
Runtime	JVM; the Java SDK needs OpenJDK LTS 11 to 25 (release 3.16.0)	One Go binary with an embedded JavaScript engine (release v2.3.0)
Test scripts	Java, Kotlin, Scala, JavaScript or TypeScript	JavaScript, or TypeScript with the types stripped
Core protocols	HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events, JMS	HTTP/1.1, HTTP/2, WebSockets, gRPC
Load model	Open or closed injection profiles	Six executors, two of them arrival-rate (open model)
Pass/fail	Assertions, checked after the run	Thresholds, which can also abort a run early
Reports	HTML report after each run	End-of-test summary; web dashboard with HTML export
Browser testing	None built in; the Recorder turns browser traffic into code	Built-in browser module (drives a Chromium-based browser)
Scaling out	Gatling Enterprise	k6-operator on Kubernetes, or Grafana Cloud k6

What is the difference between Gatling and k6?

The biggest difference is the runtime. Gatling runs on the JVM and you write tests with one of its SDKs, while k6 is a single compiled Go program that runs JavaScript.

Gatling's README lists a Java SDK for Java, Kotlin and Scala, run through a Maven, Gradle or sbt plugin. Its JavaScript and TypeScript SDK installs through npm, and its command line tool downloads GraalVM, so even a JavaScript simulation runs on a JVM.

k6 installs as a package or a standalone binary with no other runtime. It runs .ts files directly too, though its TypeScript support strips the types without checking them (checked on 9 October 2026).

How do you write the same test in each tool?

Both tools can express the same test in JavaScript. Gatling calls it a simulation and describes load as an injection profile, while k6 calls it a script and describes load as a scenario with an executor.

Here is a test where ten new users arrive every second for five minutes, and the run fails if the 95th percentile tops 500 ms or more than 1% of requests fail. In Gatling's JavaScript SDK, modelled on its first simulation tutorial:

import {
  constantUsersPerSec, global, scenario, simulation,
} from "@gatling.io/core";
import { http } from "@gatling.io/http";

export default simulation((setUp) => {
  const api = http.baseUrl("https://staging.example.com");

  const users = scenario("API users")
    .exec(http("list users").get("/api/users"));

  setUp(users.injectOpen(constantUsersPerSec(10).during(300)))
    .protocols(api)
    .assertions(
      global().responseTime().percentile(95).lt(500),
      global().failedRequests().percent().lt(1),
    );
});

The same test in k6, with the constant-arrival-rate executor:

import http from "k6/http";

export const options = {
  scenarios: {
    api_users: {
      executor: "constant-arrival-rate",
      rate: 10,
      timeUnit: "1s",
      duration: "5m",
      preAllocatedVUs: 50,
    },
  },
  thresholds: {
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  http.get("https://staging.example.com/api/users");
}

Run them with npx gatling run and k6 run script.js.

How do Gatling and k6 model load?

Both support open and closed workload models. Gatling makes the choice through the injection method you call, and k6 through the executor you pick.

In Gatling, injectOpen takes arrival rates such as constantUsersPerSec, and injectClosed takes concurrency targets such as constantConcurrentUsers. k6 has six executors: the two arrival-rate ones follow the open model, and the other four follow the closed model, where a virtual user starts its next iteration only when the last one finishes.

A k6 arrival-rate scenario starts iterations from a pool of pre-allocated virtual users, and when the pool runs dry, k6 skips iterations and counts them as dropped iterations (checked on 9 October 2026). Our guide to the open vs closed workload model explains when each model fits.

How do Gatling and k6 decide whether a test passed?

Both turn performance goals into a pass or fail result that a CI pipeline can act on. Gatling checks its goals after the run ends, while k6 can also stop a run as soon as a goal is missed.

Gatling assertions are evaluated after the simulation, and one failed assertion fails the simulation and the build. k6 thresholds end a failed run with a non-zero exit code, and a threshold set with abortOnFail stops the test the moment it fails (checked on 9 October 2026).

Which protocols and browser tests does each tool cover?

Both cover HTTP and WebSockets out of the box. Gatling adds Server-Sent Events and JMS, and k6 adds gRPC and real browser sessions.

Gatling's README says gRPC, MQTT and GraphQL are separate components under the Gatling Enterprise Component License, capped at five concurrent users and five-minute tests without Enterprise. k6 lists HTTP/1.1, HTTP/2, WebSockets and gRPC as built-in protocols, with more through xk6 extensions such as Kafka and Redis.

For browser tests, k6 has a browser module that drives a Chromium-based browser and collects frontend metrics. Gatling's docs describe no browser engine: the Recorder turns a browser session or HAR file into an HTTP-level simulation (checked on 9 October 2026).

How do you run a bigger test than one machine can generate?

Gatling's open source edition generates load from one machine, and distributed runs need Gatling Enterprise. k6 can spread a test over a Kubernetes cluster with the open source k6-operator, or run it on Grafana Cloud k6.

Gatling's installation guide lists an Enterprise account for distributed runs, which use managed load generators or private locations in your own AWS, Azure, GCP or Kubernetes account. The k6-operator splits one test across several k6 instances in a cluster (checked on 9 October 2026).

There is a third route for k6 scripts. LoadFocus, which publishes this site, runs k6 scripts on its own cloud infrastructure from 25 regions, with the users and duration taken from the test's settings. Its script validator warns that arrival-rate scenarios run there with a fixed number of users, so an open-model script like the one above does not hold its arrival rate on LoadFocus. LoadFocus does not run Gatling simulations, so for Gatling the route to more machines is Gatling Enterprise.

What reports do you get?

Gatling writes an HTML report after every run with no setup. k6 prints a summary in the terminal and can produce an HTML report from its built-in web dashboard.

Gatling's HTML report charts response time percentiles, users started per second and requests per second over time. k6 prints an end-of-test summary, and its web dashboard shows live charts and exports a self-contained HTML report when you set K6_WEB_DASHBOARD_EXPORT.

Which one should you choose?

Gatling fits teams on the JVM, and k6 fits teams that want one binary and JavaScript. Protocols, browser tests and scaling out settle most other cases.

Choose Gatling instead of k6 when your team lives on the JVM and wants Java, Kotlin or Scala tests built with Maven, Gradle or sbt next to the application code, when you need JMS or Server-Sent Events, or when you want an HTML report after every run with no setup.

Choose k6 instead of Gatling when you want a single binary in CI with no JVM to install, when your tests need gRPC or real browser sessions, when thresholds should stop a failing run early, or when you plan to scale out on Kubernetes with an open source operator.

FAQ

Is Gatling faster than k6? We have not measured this ourselves, so we will not name a winner. The fair test is the same scenario with both tools on your own hardware.

Can Gatling run k6 scripts? Not according to Gatling's docs, which describe no way to run k6 scripts. The two SDKs use different APIs, and Gatling's converters cover JMeter and LoadRunner, so a k6 script has to be rewritten as a Gatling simulation, even with the JavaScript SDK.

Can LoadFocus run Gatling tests? No. LoadFocus runs k6 scripts and JMeter test plans. To spread a Gatling simulation across several machines, use Gatling Enterprise.

HTTP benchmarking tools: wrk, hey, oha, ab and more

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

The command-line HTTP benchmarking tools most teams reach for are wrk, wrk2, hey, oha, ab (Apache Bench), bombardier, autocannon and h2load. Use wrk2 when you need a constant request rate and honest tail latency, hey or oha for a quick check that also speaks HTTP/2, h2load when HTTP/2 or HTTP/3 is the thing under test, and autocannon when your team works in Node.js.

We have not run these tools against each other, so there are no speed numbers or "fastest" verdicts here. Facts come from each project's own repository, manual or releases page (checked on 9 October 2026); the "choose" notes are our judgement.

Tool	Written in	Licence	Rate control	HTTP/2	Latest release
wrk	C, LuaJIT scripts	Modified Apache 2.0	None	Not documented	Tag 4.2.0, Feb 2021
wrk2	C	Apache 2.0	Constant (`-R`)	Not documented	None, last commit Sep 2019
hey	Go	Apache 2.0	Limit per worker (`-q`)	Yes	v0.1.5, Jan 2026
oha	Rust	MIT	Overall limit (`-q`)	Yes, plus HTTP/3	v1.16.0, Aug 2026
ab	C (Apache httpd)	Apache 2.0	None	No	httpd 2.4.69, Oct 2026
bombardier	Go	MIT	Limit (`-r`)	Yes	v2.0.2, Mar 2025
autocannon	Node.js	MIT	Limit (`-R`)	No	v8.0.0, Oct 2024
h2load	C++ (nghttp2)	MIT	Per client (`--rps`)	Yes, plus HTTP/3	nghttp2 v1.70.0, Jul 2026
Vegeta	Go	MIT	Constant (`-rate`)	Yes	v12.13.0, Oct 2025

Why does it matter whether a tool fixes the rate?

A tool that holds a fixed number of connections sends less when the server slows down, so its latency figures leave out the time requests spent waiting. A fixed-rate tool keeps sending on schedule, and the slowdown shows up in the percentiles.

Most tools here run a set number of connections or workers (-c), each waiting for a response before sending more. The wrk2 README calls the resulting blind spot Coordinated Omission and times each request from when it was supposed to be sent. Vegeta also holds a constant rate; see our open source load testing tools roundup. oha and autocannon can correct their latency for the same effect when a rate is set, while hey and bombardier document their rate options as limits only.

Is wrk still worth using?

Yes, for a quick look at how much load one machine can push at an endpoint, read with the caveat above. Its newest tag dates from February 2021.

wrk is written in C and keeps a fixed number of connections open across several threads. Optional LuaJIT scripts can build requests, process responses and write custom reports, and make builds it together with the bundled LuaJIT and OpenSSL. It prints average, standard deviation and maximum latency; --latency adds the 50th, 75th, 90th and 99th percentiles.

Choose wrk2 instead when you need a fixed request rate or tail latency you can trust during a slowdown.

When do you need wrk2 instead of wrk?

When the question is how latency behaves at a given request rate. wrk2 holds that rate constant and records every latency in an HdrHistogram.

wrk2 adds a throughput argument (-R, requests per second) to wrk. --latency prints the full percentile spectrum and --u_latency adds uncorrected figures for comparison. Its README warns that runs under 10 to 20 seconds may not be useful because of a 10-second calibration period, and calls the project experimental. It has no releases, and its Makefile builds a binary also named wrk, so keep the two apart.

Choose oha or autocannon instead when you want a rate setting with latency correction from a project that is still maintained.

Are hey and oha interchangeable?

For quick checks, largely: both run a pool of workers for a request count or a duration, and both speak HTTP/2. They differ in rate limiting and in what you see during a run.

hey is a small Go program with prebuilt binaries for Linux, macOS and Windows and a Homebrew formula. -q limits queries per second for each worker, -h2 turns on HTTP/2, and the summary includes a response time histogram and a latency distribution, or CSV with -o csv.

oha is written in Rust, was inspired by hey and shows a live terminal UI. Its -q sets one overall rate, --latency-correction corrects for coordinated omission, and --http-version accepts 1.1, 2 and, in an experimental build, 3. It installs with cargo, Homebrew, winget or pacman and can write JSON or CSV.

Choose hey instead when a per-worker limit fits how you think about load; choose oha for HTTP/3, JSON or latency correction.

Should you still use ab?

For a quick check on a machine where it is already installed, yes. ab ships with the Apache HTTP Server, and its own manual lists known limits.

ab takes a request count (-n) and a concurrency level (-c), with no rate or HTTP/2 option, and uses KeepAlive only with -k. Its output ends with the percentage of requests served within a given time, and it can write those percentages to CSV (-e) or every measurement to a gnuplot file (-g). The manual's Bugs section says ab does not implement HTTP/1.x fully and warns that its heavy string parsing can make you measure ab's own performance.

Choose hey or oha instead when you need HTTP/2, a rate limit or keep-alive by default.

What are bombardier and autocannon good for?

bombardier is a Go tool with optional HTTP/2 and a rate limit. autocannon suits Node.js teams, because it also runs as a library inside a Node script.

bombardier uses the fasthttp client by default and switches to Go's net/http client with --http1 or --http2. -r sets a rate limit, -l prints the 50th, 75th, 90th and 99th percentiles, and -o json makes the result machine-readable.

autocannon describes itself as an HTTP/1.1 tool with pipelining and HTTPS, installed with npm i autocannon -g. -R caps the overall rate and -r the rate per connection, with latency corrected for coordinated omission when either is set. Its table shows the 2.5th, 50th, 97.5th and 99th percentiles, -l prints them all and -j writes JSON.

Choose autocannon instead of bombardier when your benchmarks live in a Node.js test suite, and bombardier when you need HTTP/2.

Which tool is built for HTTP/2 and HTTP/3?

h2load. It comes from the nghttp2 project, prefers HTTP/2, and can force HTTP/1.1 or, when built with support, HTTP/3.

h2load is compiled with nghttp2 by passing --enable-app to ./configure. Besides clients (-c), it sets concurrent streams per client (-m) to load-test multiplexing, and --rps sets requests per second for each client. The manual reports min, max, mean, median, p95 and p99 for request time, connect time and time to first byte. HTTP/3 support is marked experimental.

Choose oha instead when you want HTTP/2 with fewer options and a live view.

When should you move from a benchmark to a real load test?

When the question involves user journeys, more load than one machine can generate, or traffic from where your users are.

Every tool here fires simple requests from one machine, and the h2load guide notes that the client itself can become the bottleneck. Move to a load testing tool when you need logins, multi-step journeys or think time, when traffic must come from several regions, or when a test should pass or fail in CI. Our guide on how to run a load test covers that step by step, and k6 vs Artillery compares two scriptable tools.

LoadFocus, the publisher of this site, runs load tests for websites and APIs from 25 cloud regions, with JMeter, k6 or no-code tests. If your benchmark is a request you already send with curl, its free cURL to k6 converter turns that command into a k6 script in your browser.

FAQ

What is the difference between wrk and wrk2? wrk2 is a fork of wrk that sends requests at a constant rate set with -R and times each request from when it should have been sent. wrk keeps a fixed number of connections busy and has no rate setting.

Which HTTP benchmarking tools support HTTP/2? hey, oha, bombardier, h2load and Vegeta, according to their documentation. wrk and wrk2 do not document it, autocannon describes itself as HTTP/1.1 and ab has no HTTP/2 option.

Load test pass/fail criteria: how to set them

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Load test pass/fail criteria are the few limits a run must meet to count as a pass: a latency percentile for the requests users wait on, an error rate and a minimum request rate, all at a stated load. Take the limits from a service-level objective (SLO), add regression limits from earlier runs inside them, and let the tool's exit code fail the run only on those hard limits. Everything else is a warning that a person reads.

Tool facts were checked in each project's own docs on 9 October 2026, and every limit shown is an example.

Criterion	Example limit	Source of the number	On a miss
p95 latency, checkout	Under 400 ms	Latency SLO minus a margin	Fail
p99 latency, checkout	Under 1,200 ms	Earlier runs of this test	Warning
Error rate	Under 0.5%	Availability SLO	Fail
Requests per second	At least 95% of plan	The planned load	Fail
Load generator CPU	Under 80%	Tool or platform	Run invalid

What makes a load test pass or fail?

A run passes when every hard limit holds at the planned load and fails when one is missed. It is invalid when the test itself did not do its job, whatever the numbers say.

A saturated load generator, a load that never reached its target or a failing script all produce numbers about the test setup. LoadFocus, for example, shows a load generator saturation warning when an engine peaked at or above 80% CPU. k6's exit codes separate the cases for CI: 99 means thresholds failed, 107 a script exception (checked on 9 October 2026).

How do you turn an SLO into load test criteria?

Restate each SLO that load can break, usually latency and availability, as a limit on what the test measures, and make it stricter than the SLO so a pass leaves room.

Google's SRE book defines an SLO as a target value or range for a service level measured by an indicator (SLI), with examples such as 99% of Get calls completing in under 100 ms. It wants an SLO to state how it is measured and when it is valid, and recommends a safety margin: an internal target tighter than the one users see.

For a load test, "when it is valid" means the load and request mix: "95% of checkout calls under 500 ms" might become "p95 of checkout under 400 ms at 300 requests per second for 15 minutes", as an example. With no SLO yet, agree a provisional one with the product owner.

Which metrics should the criteria cover?

Latency for the requests users wait on, an error rate, the request rate achieved and, mostly as warnings, server resources.

Latency per request type. A global percentile mixes a fast health check with a slow search. In k6, tagged requests create sub-metrics with their own limits; Gatling's details() targets one request or group. A test that slowed its own load down reports flattering percentiles (coordinated omission).
Errors, counted broadly. A fast 200 with an error page is still a failure, so assert on content. A failed JMeter Response Assertion marks its sample unsuccessful; k6 checks reach the exit code only through a threshold.
A request-rate floor. In a closed workload model a slow system receives fewer requests, so a run can meet its latency limits at a fraction of the planned load (open vs closed workload model).
Server resources. CPU, memory and pool usage come from your monitoring. Treat them as warnings unless a capacity plan sets a ceiling.

How do you pick the numbers?

Take the ceiling from the SLO and the regression limit from earlier runs of the same test. The SLO says what users tolerate; earlier runs say what this build normally does.

The SRE book warns against setting an SLO from current performance. A regression limit exists to notice change, so current performance is exactly its input. Run the unchanged build several times with the same environment and load, note how far each metric moves, and set the limit above the worst run with some headroom, still under the SLO-derived ceiling where there is one.

If runs of one build vary by more than the regression you want to catch, the test is too noisy to gate on: lengthen the steady phase or isolate the environment first. Record new reference runs after any deliberate change.

Which criteria should fail the build, and which should only warn?

Fail the build on limits that come from an SLO or check correctness. Warn on tail percentiles, resource use and slow drifts, which need judgment; a gate that fails on noise soon gets switched off.

Artillery's ensure plugin has a warning tier: a condition with strict: false is optional, so a miss leaves the exit code alone (checked on 9 October 2026). In k6, a check without a threshold never fails the run. Elsewhere, keep warnings out of the assertions.

config:
  plugins:
    ensure:
      thresholds:
        - "http.response_time.p95": 400
      conditions:
        - expression: "http.response_time.p99 < 1200"
          strict: false

When should a load test stop early?

When carrying on can only do harm or prove nothing: errors far past the limit, a system already down or a generator that cannot keep up. Give the rule a grace period so start-up noise cannot trip it.

Make abort rules looser than pass limits: they protect a shared environment. k6's abortOnFail stops the test when a threshold fails, after an optional delayAbortEval. Locust's extension docs show a background greenlet that quits once the failure ratio passes a threshold. Gatling's crashLoadGeneratorIf stops the generator with a failure code when a condition in a user's session holds. A JMeter Thread Group can stop the test on a sampler error, which suits a correctness check (all checked on 9 October 2026).

How does each tool encode the criteria?

k6 and Artillery turn a missed hard limit into a non-zero exit code, Locust does so for any failed request or for limits you code yourself, and Gatling's assertions fail the build. JMeter leaves the verdict to a step after the run.

k6 thresholds sit in the script's options. This example fails on the p95 of requests tagged name: checkout or a 0.5% error rate, and aborts after a minute if errors pass 5%:

export const options = {
  thresholds: {
    "http_req_duration{name:checkout}": ["p(95)<400"],
    http_req_failed: [
      "rate<0.005",
      { threshold: "rate<0.05", abortOnFail: true,
        delayAbortEval: "1m" },
    ],
  },
};

Gatling assertions combine a scope (global, forAll, details), a statistic such as a percentile or requestsPerSec, and a condition. They are evaluated after the run, and one failure fails the simulation.

Locust exits with code 1 when any request failed, and --exit-code-on-error changes that code. For a ratio or a percentile, set environment.process_exit_code in a quitting listener, which Locust's source honours first.

JMeter assertions judge single samples, and its command-line options include no run-level limit. CI decides: a short script over the results file, as in our GitHub Actions guide, or the third-party jmeter-maven-plugin, whose results goal fails the build on any failed sample unless errorRateThresholdInPercent sets a rate (checked on 9 October 2026).

LoadFocus grades runs against pass/fail thresholds saved on the test: maximum p95 and p99 response times, a maximum error rate and a minimum request rate, each optional. Each run gets a PASS or FAIL badge with actual and target values, and an API hands the verdict to a pipeline. LoadFocus runs k6 arrival-rate scenarios with a fixed number of users, so there a rate floor checks what those users achieved.

FAQ

Should the average response time be a pass/fail criterion? Rarely. The SRE book prefers percentiles: an average hides the long tail and can stay flat while the slowest requests get much slower.

How many pass/fail criteria should a load test have? A handful of hard limits, plus warnings. The SRE book advises having as few SLOs as possible and says that if quoting one can never win a conversation about priorities, it is probably not worth having. Apply the same rule to each load test limit.

Can a load test pass with some failed requests? Yes, if the error rate stays under its limit. Check your tool's default: Locust fails the run on any failed request.

Little's law in load testing: size and check your tests

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Little's law in load testing says that the average number of users or requests inside a system equals the rate at which they pass through it times the average time each one spends there: L = λW. With a fixed number of virtual users, that becomes users = loops per second × (response time + think time), which shows before a run whether a plan can reach its target rate and, after it, whether the results add up. It works on averages over a steady stretch of the test, so it says nothing about percentiles.

The worked examples below are arithmetic with no test run behind them. Tool facts come from each project's documentation, checked on 9 October 2026.

Term	Little's definition	Fixed virtual users (closed model)	Arrival rate (open model)
L	Average number of items in the system	Virtual users, waiting or pausing	Iterations in progress
λ	Average arrival rate	Loops completed per second	The rate you set
W	Average time spent inside	Response time + think time per loop	Iteration duration

What does Little's law say?

The long-run average number of items in a queuing system equals their average arrival rate times the average time each one spends in it. Know any two of the three and you can work out the third.

John D. C. Little published the proof in 1961 in Operations Research as A Proof for the Queuing Formula: L = λW. His 50th-anniversary review of 2011 says it holds under remarkably general conditions: for one queue or a whole network, for a subclass of items or all of them, and whatever the order of service. It also maps the terms onto a server: requests in process, requests per second and average response time.

What do L, λ and W mean in a load test?

That depends on where you draw the boundary. Around the server, L is the requests in progress and W is response time; around a virtual user's whole loop, L is every virtual user and W is response time plus think time.

In a closed-model test each user is always somewhere in its loop, waiting for a response or pausing between steps. With the boundary around the loop, all of them count as inside, λ is the rate at which loops complete, and W is the average loop time, including the think time the script adds:

users = loops per second × (response time + think time)

Around the server, the same law gives the requests in flight: requests per second × average response time, the figure your connection pools and worker threads must hold. Keep all three numbers on one boundary and one stretch of time; mixing them gives plausible wrong answers.

How do you size a closed-model test with Little's law?

Turn the target request rate into loops per second and multiply by the expected time per loop: the result is the number of virtual users the plan needs. Use the response time you expect at the target load, which is usually higher than what a few users see.

A worked example, arithmetic only. The goal is 100 requests per second, each loop sends 4 requests expected to take 0.5 seconds each, and the script pauses 3 times for 2 seconds on average:

loops per second = 100 requests/s ÷ 4 requests  = 25
time per loop    = 4 × 0.5 s + 3 × 2 s          = 8 s
virtual users    = 25 loops/s × 8 s             = 200

Treat 200 as the minimum. If responses take 1 second at that load, a loop lasts 10 seconds and the same target needs 250 users.

Why do the same users send fewer requests when the system slows down?

The number of users is fixed, so when the time per loop grows, the loops per second must shrink. The load falls just as the system starts to struggle.

Keep the 200 users and let each request take 2 seconds. A loop now lasts 4 × 2 + 6 = 14 seconds, and 200 ÷ 14 gives about 14.3 loops, or about 57 requests per second. The test still shows 200 users while running at little over half the planned rate. The open vs closed workload model article explains why that hides saturation; Little's law predicts the size of the drop.

It also shows what a capacity limit looks like. Once the request rate cannot rise, each extra user adds only waiting: at a ceiling of 25 loops per second, 300 users mean 12 seconds per loop and 400 users mean 16. LoadFocus's response time vs virtual users chart plots average and maximum response time against active virtual users, one point per user level reached, which is where that steady climb shows up.

How do you check a finished test against Little's law?

Take the average request rate and response time from a stretch where the user count stayed flat, rebuild the number of users from them, and compare it with what you configured. A large gap means time is going somewhere your numbers do not show.

Another arithmetic example. A run holds 200 users and reports 80 requests per second at a 0.6-second average, with the same 4 requests and 6 seconds of pauses per loop. That is 20 loops per second at 4 × 0.6 + 6 = 8.4 seconds each, enough for 168 users. The other 32 mean each loop really took 10 seconds, so about 1.6 seconds per loop is unexplained. Places to look:

Connection setup. In k6, http_req_duration leaves out DNS lookup and connecting (metrics reference, checked on 9 October 2026).
The load generator. A busy generator delays the script between requests.
Think time. Random pauses can average more than the plan says.
The window. If it overlaps a ramp, fewer users were active on average.

k6 also measures the loop directly: vus counts active virtual users and iteration_duration times one full iteration, so active users should land close to iterations per second × average iteration duration.

What about arrival-rate tests?

The law holds with the boundary around each iteration: iterations in progress equal the arrival rate times the iteration duration. That is how many virtual users the tool must keep ready.

The k6 guide to allocating VUs for arrival-rate executors estimates them as median iteration duration × rate, plus a constant for variance (checked on 9 October 2026). That is Little's law with the median standing in for the mean; a few slow iterations pull the mean above the median, so the margin matters.

LoadFocus runs k6 scripts in its cloud, and there arrival-rate scenarios run with a fixed number of users; the script validator warns about it when you add such a script. Size those tests with the closed-model arithmetic above and expect the rate to fall if responses slow down.

Where does Little's law stop working?

It ties together averages over a stretch where the system is in balance. Percentiles, short windows and ramps fall outside it.

Averages only. A run can satisfy the law while one request in twenty waits ten times the mean, so judge results on percentiles too.
A steady stretch. The 1961 proof assumes a stationary process, and the 2011 review shows the formula is also exact over a finite window that starts and ends empty. A test plateau is neither, so pick a long one, where users finishing loops keep pace with users starting them.
Ramps. While users are being added, the configured peak is not L: use the window's time-averaged user count, or leave the ramp out.
Errors. Fast failures count as completed requests with short times, so a failing system can still balance the equation. Check the error rate first.

FAQ

Does Little's law need Poisson arrivals or a particular queue? No. The proofs in Little's review make no assumption about the arrival pattern or service times, and the result holds whatever the order of service. It needs only averages that exist over the period you measure.

Can Little's law predict p95 response times? No. It links three averages, so it can check a mean response time, but it says nothing about the shape of the distribution or the slowest few percent.

Locust load testing: a review of the Python tool

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Locust is a free, MIT-licensed Python load testing tool in which every test is a Python program, run by lightweight users from a web UI or the command line. Locust load testing suits teams that write Python and mostly test HTTP services. Teams that need many protocols out of the box or a GUI test builder are better served by JMeter, and teams that write JavaScript by k6.

We have not run Locust for this review. The facts come from Locust's own documentation, repository and PyPI page, checked on 9 October 2026, and the judgements are ours.

Topic	Locust
Licence	MIT
Runtime	Python 3.11 or later
Current release	2.46.7 (4 October 2026)
Tests written as	Python classes in a locustfile
Ways to run	Web UI on port 8089, or headless
Built-in protocols	HTTP and HTTPS
Scaling out	One master and any number of workers
Results	Live charts, CSV, HTML, JSON, OpenTelemetry
CI	Exit code 1 when any request failed

Who makes Locust, and is it still maintained?

Lars Holmberg maintains Locust, and Microsoft sponsors the work. Releases are frequent: 17 so far in 2026.

The Locust README names both. PyPI lists 2.46.7, published on 4 October 2026, as the latest release, with Python 3.11 or later required (checked on 9 October 2026).

How does a locustfile work?

A locustfile is an ordinary Python module with at least one user class. Each simulated user runs its tasks one after another and pauses for its wait_time in between.

Per the locustfile guide, HttpUser gives each user a client that wraps python-requests, keeps cookies and records every request. Methods marked @task are picked at random, weighted by the number in the decorator, and on_start runs once per user. This user logs in, browses four times as often as it checks out, and starts a task at most once every 10 seconds:

from locust import HttpUser, task, constant_pacing


class Shopper(HttpUser):
    wait_time = constant_pacing(10)

    def on_start(self):
        self.client.post("/login", json={"user": "demo"})

    @task(4)
    def browse(self):
        self.client.get("/products?page=2", name="/products")

    @task
    def checkout(self):
        with self.client.post(
            "/checkout", catch_response=True
        ) as resp:
            if "order_id" not in resp.text:
                resp.failure("no order id")

catch_response lets you fail a request on its content, and name groups changing URLs into one row. The other wait functions are constant, between, constant_throughput and, since release 2.46.6, poisson; our guide to think time in load testing explains how pauses shape the load. The docs add that wait time can only limit throughput and cannot start users to reach a target rate, so a Locust test is a fixed pool of users: a closed workload model.

Should you use the web UI or run headless?

Use the web UI while you explore and headless mode for repeatable runs.

Running locust starts a web interface on port 8089, where you set the host, users and ramp-up, watch live charts and change the load mid-test. Headless mode takes the same settings as flags:

locust -f locustfile.py --headless -u 200 -r 10 -t 10m \
  --host https://staging.example.com \
  --csv results --html report.html

-u sets the users, -r how many start per second and -t the run time, ramp-up included. The same settings also work in locust.conf or pyproject.toml.

How does distributed mode work?

A master process runs the web UI and coordinates, while worker processes run the users and report back. You need one worker per CPU core, because a Python process cannot fully use more than one.

The distributed load docs put one process at over a thousand requests per second for a simple test plan, so bigger tests scale out:

# on the master machine
locust -f locustfile.py --master
# on each worker machine, one worker per core
locust -f - --worker --master-host 10.0.0.5 --processes -1

-f - fetches the locustfile from the master. --processes relies on fork(), so it does not work on Windows, and every process needs a restart after the locustfile changes. The docs also cover a Kubernetes operator and Azure Load Testing (checked on 9 October 2026).

LoadFocus, which publishes this site, does not run Locust tests. It runs JMeter test plans and k6 scripts from 25 cloud regions, so for Locust the routes above are the ways to scale.

Can Locust test protocols other than HTTP?

Yes, if you bring a Python client for the protocol. HTTP and HTTPS are the only protocols built in.

The page on other protocols shows how to wrap a client so each call is recorded, with examples for XML-RPC and gRPC and experimental users for MQTT, Socket.IO and OpenAI. The separate locust-plugins project adds Kafka, FTP, Selenium and Playwright users (checked on 9 October 2026).

For HTTP there is also FastHttpUser, which uses geventhttpclient and much less CPU: the docs' best case for one process is about 16,000 requests per second, against 4,000 with HttpUser. FastHttpUser does not make single requests faster, and it skips TLS certificate checks by default.

What reports does Locust give you, and how does it fit into CI?

You get live charts in the web UI and result files on request. In CI, Locust exits with code 1 if any request failed.

Per the configuration reference, --csv writes statistics, history and failures files, --html saves a report and --json prints the final statistics. --otel exports to an OpenTelemetry backend. The headless docs show a listener that fails the run on failure ratio, average response time or 95th percentile, and our guide to load testing in GitHub Actions puts it in a workflow.

What does Locust do well, and where are its limits?

It is strong on test logic and on many concurrent, mostly idle users. It is weaker on request rate per process and on protocols you wire up yourself.

Plain Python means loops, branches, generated data and your own API client all work, and tests sit in version control. The What is Locust page says the low cost of each user suits highly concurrent workloads, while granting that other tools may send more requests per second on the same hardware.

Limits the docs state themselves:

One Python process uses one CPU core.
Libraries doing I/O in C block the process unless gevent can patch them.
HttpUser does not parse HTML or load page resources.
Wait time cannot add users to hold a request rate.

Which other Python load testing tools are there?

Locust is the actively developed Python engine. Molotov, the main alternative, has had no release on PyPI since 2022, and Taurus runs Locust through one of its executors.

Molotov is a Python 3.8+ tool built on asyncio and aiohttp; its latest release on PyPI is 2.6, from 21 October 2022, and the last commit on its main branch dates from November 2023 (checked on 21 October 2026). Taurus runs a locustfile, or GET and POST requests written in YAML, through its locust executor. You install Locust yourself, Taurus starts only the master in distributed mode, and its throughput setting has no effect with this executor (checked on 21 October 2026).

When should you choose k6, JMeter or Gatling instead?

Choose another tool when its language, protocols or load model fit your team better.

Our overview of open source load testing tools sets Locust next to five others.

Choose k6 instead when your team writes JavaScript, wants pass/fail thresholds in the script, or needs an open workload model, which k6's arrival-rate executors provide (checked on 9 October 2026). LoadFocus runs k6 scripts from 25 cloud regions, though it runs arrival-rate scenarios with a fixed number of users, so for a held request rate run them with k6 itself.

Choose JMeter instead when you need JDBC, JMS, LDAP or FTP out of the box, as its README lists, or when your testers build tests in a GUI. Locust vs JMeter compares the two in detail.

Choose Gatling instead when your team works on the JVM and wants open and closed injection profiles in one API (checked on 9 October 2026). Gatling vs JMeter covers it further.

FAQ

Is Locust free for commercial use? Yes. Locust is released under the MIT licence, which permits commercial use, modification and redistribution.

Which Python version does Locust need? Python 3.11 or later, according to its PyPI metadata. Install it with pip install locust, or run it in a temporary environment with uvx locust.

Can LoadFocus run Locust tests? No. LoadFocus runs JMeter test plans and k6 scripts. To scale a Locust test, use its master and worker mode, the Kubernetes operator or Azure Load Testing.

Locust vs JMeter: Python scripts or JMeter test plans?

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Locust and Apache JMeter are both free, open source load testing tools, and the better fit depends mostly on how your team wants to build tests. Pick Locust if your team writes Python and wants test logic as ordinary code, run by lightweight event-based users. Pick JMeter if you need protocols beyond HTTP such as JDBC, JMS or LDAP, prefer to build tests in a GUI, or already own a library of .jmx test plans.

The facts in this Locust vs JMeter comparison come from each project's own documentation and repository, checked on 9 October 2026.

	Locust	Apache JMeter
Licence	MIT (repository)	Apache 2.0 (repository)
Runtime	Python 3.11 or later	Java 8 or later (release 5.6.3)
Tests written as	Python code in a locustfile	Test plans built in a GUI, saved as XML `.jmx` files
GUI	Web UI to start, adjust and watch a running test	Desktop IDE to record, build and debug test plans
Built-in protocols	HTTP and HTTPS; others through a Python client	HTTP(S), SOAP/REST, FTP, JDBC, LDAP, JMS, mail, TCP, shell commands, Java objects
Virtual users	One greenlet per user on gevent (event-based)	One Java thread per user
Distributed mode	One master and any number of workers	One client driving remote JMeter servers
CI and headless runs	`--headless`, exit code 1 when requests fail	CLI mode (`jmeter -n`)
Reporting	Live web UI charts, CSV files, HTML report	HTML dashboard, live results through a Backend Listener

What is the difference between Locust and JMeter?

In Locust a test is a Python program. In JMeter a test is a plan you assemble in a desktop GUI and save as a file, and most other differences follow from that.

Locust's README describes tests written with plain Python loops, conditions and libraries. It installs with pip install locust on Python 3.11 or later, according to its package metadata.

A JMeter test is a tree of elements: a thread group for the users, samplers for the requests, and assertions and listeners for the results. JMeter 5.6.3, the current release on the download page, requires Java 8 or later. The README on JMeter's development branch already calls for Java 17 (checked on 10 October 2026). When a step needs real logic, a JSR223 sampler runs a Groovy script inside the plan.

How do you write a test in each tool?

In Locust you write a Python class for each kind of user, with methods marked as tasks. In JMeter you add a thread group and samplers in the GUI, save the plan and run it from the command line.

A Locust user that waits one to three seconds between tasks and lists users three times as often as it opens one:

from locust import HttpUser, task, between


class ApiUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def list_users(self):
        self.client.get("/api/users")

    @task
    def view_user(self):
        self.client.get("/api/users/42", name="/api/users/[id]")

Run it headless with 50 users, starting five per second, for five minutes:

locust -f locustfile.py --headless -u 50 -r 5 \
  --run-time 5m --host https://staging.example.com

In JMeter the same test is a Thread Group with 50 threads and a ramp-up period, holding two HTTP Request samplers. The getting started guide runs a saved plan in CLI mode, and -e -o adds the HTML report at the end:

jmeter -n -t api-users.jmx -l results.jtl -e -o report/

The .jmx file is XML, as the examples shipped with JMeter show, and harder to review in a pull request than Python. JMeter can also record a plan from a browser; Locust's closest match is har2locust, a beta HAR-to-locustfile converter (checked on 9 October 2026).

Which protocols can each tool test?

JMeter covers many more protocols out of the box. Locust ships with HTTP and HTTPS and expects you to bring a Python client for anything else.

JMeter's README lists HTTP and HTTPS, SOAP and REST, FTP, databases through JDBC, LDAP, JMS messaging, SMTP, POP3 and IMAP mail, TCP, native commands and Java objects. JDBC and JMS need driver jars from your database or JMS provider.

Locust's page on other protocols shows how to wrap a protocol library, with examples for gRPC, XML-RPC and MQTT, and points to locust-plugins for Kafka, Selenium and Playwright (checked on 9 October 2026). The library must cooperate with gevent, because I/O in compiled C code blocks the whole process; the docs name the PostgreSQL driver psycopg2 as one that needs an extra package. For database load tests, JMeter's JDBC sampler is the shorter path.

How do virtual users work in Locust and JMeter?

JMeter runs each virtual user as a Java thread. Locust runs each user as a greenlet on gevent's event loop, which keeps the cost of each user low.

Each thread in a JMeter thread group runs the whole plan independently. JMeter starts with a 1 GB heap by default, which its guide says may not be enough for larger plans and thread counts.

Locust's README says a single process can handle many thousands of concurrent users. Its distributed load docs add that a Python process cannot fully use more than one CPU core, so you run one worker per core, which --processes sets up on a single machine.

How do you run a test across several machines?

Both have a distributed mode built in. Locust's master directs workers that run the users, and JMeter's client starts the full test plan on every remote server.

In Locust, one process started with --master runs the web UI and tells the --worker processes when to start and stop users. The Locust docs also cover a Kubernetes operator and Azure Load Testing, a Microsoft service that runs Locust tests in the cloud (checked on 9 October 2026).

JMeter's remote testing mode lets one client control any number of servers over RMI. Each server runs the whole plan, so 1,000 threads on six servers means 6,000 threads. All nodes should run the same JMeter and Java versions, RMI needs a keystore unless SSL is disabled, and data files must be copied to each server.

There is a third route for JMeter test plans. LoadFocus, which publishes this site, runs existing JMeter test plans unchanged from 25 cloud regions, with no remote servers to configure. LoadFocus does not run Locust tests, so for Locust the options above remain the ways to scale.

How do they fit into CI, and what reports do you get?

Both run headless from the command line. Locust also exits with code 1 when requests fail, and JMeter produces the richer HTML report.

Locust's headless docs show a short event listener that sets the exit code from the failure ratio, the average response time or the 95th percentile, plus a GitHub Actions example. Its --csv and --html options save results and a report after the run.

JMeter's documentation is firm that load tests run in CLI mode, with the GUI kept for building and debugging, and its README points to third-party libraries for Maven, Gradle and Jenkins. The JMeter dashboard adds an APDEX table, a statistics table with three configurable percentiles, error summaries and response time charts.

Which one should you choose?

Choose Locust when your team writes Python and wants load tests reviewed like the rest of its code. Choose JMeter instead when you need protocols beyond HTTP, when your testers prefer a GUI, or when you already have .jmx plans.

Locust suits scenarios with loops, branching and generated data, tests that reuse an internal Python client for your API, and high concurrency from users who spend most of their time waiting.

JMeter is the better choice for JDBC, JMS, LDAP, FTP or mail testing, where Locust would need a client you maintain yourself, and for testers who record a flow in a browser and refine it visually. A company that already runs JMeter also keeps its plans and know-how.

FAQ

Is Locust faster than JMeter? We have not measured this ourselves, so we will not name a winner. The fair test is the same scenario with both tools on your own hardware.

Can Locust run JMeter .jmx files? Not according to its documentation, which describes tests only as Python locustfiles. Moving a JMeter test plan to Locust means rewriting it in Python.

Can LoadFocus run Locust tests? No. LoadFocus runs JMeter and k6 scripts. To scale a Locust test beyond one machine, use Locust's master and worker mode, its Kubernetes operator or Azure Load Testing.

Load testing in GitHub Actions: a practical guide

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Load testing in GitHub Actions takes one workflow file: install the tool, run a short test against a staging URL and let the tool's exit code fail the job when a threshold is missed. Upload the results as an artifact so every run leaves a record. Keep the test small, because a standard GitHub-hosted runner is one virtual machine with a few CPU cores.

Tool and GitHub facts below come from each vendor's own documentation, checked on 9 October 2026, and the workflows follow that documented usage.

Tool	Installed by	What fails the job	Results to keep
k6	Grafana's `setup-k6-action`	A failed threshold	JSON output
Artillery	`artilleryio/action-cli`	A failed `ensure` check	`report.json`
Locust	`pip install locust`	Any failed request, or your own rule	CSV and HTML files
JMeter	The release archive	Nothing built in: add a check step	`.jtl` file and HTML report

What does a load test workflow need?

A trigger, a runner, a step that installs the tool, a step that runs the test with thresholds and a step that uploads the results. Everything else is optional.

Keep the test script in the repository, for example under tests/load/. Read the staging URL from a repository variable through the vars context, and set timeout-minutes on the job: the default is 360 minutes (checked on 9 October 2026).

Which events should start a load test?

Run a short test on pull requests and a longer one on a schedule or by hand.

Use pull_request to catch a regression before the merge, workflow_dispatch for manual runs and schedule for cron runs (UTC unless you set a timezone). GitHub's event reference adds limits (checked on 9 October 2026): schedules run only on the default branch, at most every five minutes, and can be delayed at the start of the hour. Pull requests from forks get no secrets except GITHUB_TOKEN.

How do you run k6 in GitHub Actions?

Use Grafana's two official actions, one to install k6 and one to run the script. The step fails whenever k6 exits with a non-zero code, which a failed threshold causes.

Both setup-k6-action and run-k6-action are on v1 (checked on 9 October 2026):

name: load-test
on:
  pull_request:
  workflow_dispatch:

jobs:
  k6:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      BASE_URL: ${{ vars.STAGING_URL }}
    steps:
      - uses: actions/checkout@v7
      - uses: grafana/setup-k6-action@v1
      - uses: grafana/run-k6-action@v1
        with:
          path: tests/load/smoke.js
          flags: --out json=results.json
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: k6-results
          path: results.json
          retention-days: 14

The thresholds live in the script, which reads BASE_URL from the environment:

import http from "k6/http";

export const options = {
  vus: 10,
  duration: "1m",
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<500"],
  },
};

export default function () {
  http.get(`${__ENV.BASE_URL}/health`);
}

k6's documentation notes that checks do not affect the exit status, so the pass/fail decision has to come from thresholds.

How do you run Artillery, Locust or JMeter instead?

Swap the install and run steps, keep the rest of the workflow, and use each tool's own switch for failing on a missed target.

Artillery has an official GitHub Action whose README says @v2 tracks the latest 2.x release (checked on 9 October 2026). The step is uses: artilleryio/action-cli@v2 with command: run --target ${{ vars.STAGING_URL }} tests/load/api.yml --output report.json, and the ensure plugin exits non-zero when a target is missed. The script's config section:

config:
  target: "https://staging.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
  plugins:
    ensure:
      thresholds:
        - "http.response_time.p95": 500

Locust exits with code 1 when any request failed, and for latency rules you set environment.process_exit_code in a quitting event listener, as that page shows (checked on 9 October 2026). After actions/setup-python@v7 with python-version: "3.13":

      - run: pip install locust
      - run: >
          locust -f tests/load/locustfile.py --headless
          -u 20 -r 5 -t 2m -H "$BASE_URL"
          --csv results --html report.html

JMeter runs without its GUI through -n, and the command-line options in its user manual include no pass/fail threshold (checked on 9 October 2026). A short step can read the results file, CSV with a success column by default, and fail the job:

      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: "21"
      - name: Install JMeter 5.6.3
        env:
          JM_BASE: https://archive.apache.org/dist/jmeter/binaries
        run: curl -sSL "$JM_BASE/apache-jmeter-5.6.3.tgz" | tar -xzf -
      - name: Run the test plan
        run: >
          apache-jmeter-5.6.3/bin/jmeter -n -t tests/load/plan.jmx
          -l results.jtl -e -o report
      - name: Fail above 1% errors
        run: |
          python3 - <<'EOF'
          import csv, sys
          rows = list(csv.DictReader(open("results.jtl")))
          bad = sum(r["success"] != "true" for r in rows)
          print(f"{bad} of {len(rows)} samples failed")
          sys.exit(1 if not rows or bad > 0.01 * len(rows) else 0)
          EOF

How do you make the job fail when the test misses a target?

Write the targets into the test as thresholds and let the exit code do the rest. GitHub marks a step as failed when its command exits with a non-zero code, and the job fails with it.

Start with a strict error rate and a generous latency percentile, then tighten the latency limit once you know how much your numbers move between runs. The step-by-step load testing guide covers choosing those targets.

Where should the results of each run go?

Upload them as a workflow artifact, with if: always() so a failed test still keeps its report. Artifacts are kept for 90 days by default, and retention-days shortens that.

The upload-artifact README documents the 90-day default (checked on 9 October 2026). Set if-no-files-found: error so a missing report fails loudly.

How much load can a GitHub-hosted runner generate?

Enough for a smoke test or a small regression check. A standard Linux runner has 4 CPUs and 16 GB of RAM in public repositories, 2 CPUs and 8 GB in private ones, and a job stops after 6 hours.

Those figures come from GitHub's runner reference and Actions limits (checked on 9 October 2026), which give no figure for network bandwidth.

A job runs on one machine, so it is one load generator: if its CPU is saturated, response times include time spent waiting on the runner. Every request also leaves from that machine, so rate limiters that count per client address see one very busy client. GitHub advises against allowlisting the Azure IP ranges its Ubuntu runners use; larger runners on Team and Enterprise Cloud plans add CPU, memory and static IP addresses.

When should the load come from somewhere else?

When you need more load than one machine produces, traffic from several regions, or runs longer than the job limit.

With k6, run-k6-action runs the test on Grafana Cloud k6 when you set cloud-run-locally: false and pass K6_CLOUD_TOKEN, K6_CLOUD_PROJECT_ID and, with k6 v2, K6_CLOUD_STACK_ID. A cloud run ignores the runner's environment by default, so add -e BASE_URL=${{ vars.STAGING_URL }} to flags (checked on 10 October 2026). Artillery's action can call run-fargate or run-lambda to generate load from your own AWS account (both checked on 9 October 2026). The k6 vs Artillery comparison covers both.

For JMeter, LoadFocus documents a GitHub Actions integration built on its npm client. You save the test in your LoadFocus account, store the API key and team ID as secrets, and call run-test. The flags below follow the client's own help text:

      - run: npm install -g @loadfocus/loadfocus-api-client
      - env:
          LF_KEY: ${{ secrets.LOADFOCUS_API_KEY }}
          LF_TEAM: ${{ secrets.LOADFOCUS_TEAM_ID }}
        run: |
          test -n "$LF_KEY"
          test -n "$LF_TEAM"
          loadfocus-api config set \
            --apikey "$LF_KEY" --teamid "$LF_TEAM"
          loadfocus-api jmeter run-test --name checkout-api \
            --thresholds "avgresponse<=200,errors==0,p95<=250"

The client waits up to 1,800 seconds by default (--wait-timeout changes it), so set the job's timeout-minutes above 30, such as 40. It exits with code 1 when a threshold fails and with 0 when it has no API key or team ID, so the test -n lines fail the step first when a secret is empty. The load comes from LoadFocus, which runs JMeter scripts from 25 cloud regions.

FAQ

How long should a load test in a pull request run? A few minutes: long enough to get past the ramp-up and collect a stable percentile, short enough that nobody waits on it.

Is it allowed to load test from GitHub-hosted runners? GitHub's terms for Actions forbid disrupting any service and limit hosted runners to work on the repository's own project (checked on 9 October 2026). Load testing that project's own staging environment, without burdening GitHub's servers, is the kind of testing those terms describe.

Can I use a self-hosted runner instead? Yes. It runs on your own hardware and network, and GitHub's limits page allows a self-hosted job up to 5 days. It is still one load generator.

Open vs closed workload model: which one should you use?

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

The open vs closed workload model question comes down to how users reach your system. Use an open model, which starts new iterations at a set arrival rate, for public websites and APIs where users keep coming however slow you are, and a closed model, a fixed number of virtual users that each wait for their last iteration to finish, only where the system itself caps concurrency. The choice matters most under saturation, where a closed test slows its own load down and makes the system look healthier than it is.

The tool facts below come from each project's own documentation, checked on 9 October 2026.

	Open model	Closed model
You set	New iterations or users per second	Concurrent virtual users
When responses slow down	Work keeps arriving and queues grow	Users wait, so less load is sent
Real systems like this	Most public websites and APIs	Call centres, ticket queues, worker pools
k6	`constant-arrival-rate`, `ramping-arrival-rate`	`constant-vus`, `ramping-vus`
Gatling	`injectOpen`	`injectClosed`
Locust	Approximated with `constant_throughput` or `poisson`	Users with a `wait_time`
JMeter	Open Model Thread Group, Arrivals plugin	Thread Group

What is the difference between an open and a closed workload model?

In an open model the load generator decides when work arrives, and in a closed model the system's response time does.

The k6 documentation puts it plainly: in the closed model, a virtual user's next iteration starts only when the last one finishes, while in the open model new iterations arrive independently of iteration completion (checked on 9 October 2026). A closed test with 50 virtual users never has more than 50 iterations in flight. An open test at 50 iterations per second keeps starting 50 every second, and if each one takes longer, more of them overlap.

Why does a closed model hide latency when the system saturates?

Because the test slows down together with the system. When responses take longer, each closed-model user sends fewer requests, so the load drops at the moment you most wanted to see it rise.

The k6 docs describe the effect: slower responses mean longer iterations and a lower arrival rate of new ones. Gatling's page on workload models goes further and calls a closed model used against a system that is really open a broken test of an imaginary behaviour (checked on 9 October 2026).

A closed test therefore shows response times that level off, because requests in flight can never exceed the number of users. An open test shows the queue growing, with response times that climb for as long as arrivals outpace the system. That second picture is the one your users would see, and the p95 and p99 response times you read at the end are only as honest as the load model behind them.

The k6 docs note that some testing literature calls this problem coordinated omission. A 2006 USENIX paper that Gatling's docs point to, Open Versus Closed: A Cautionary Tale, found that the two models give significantly different results even with the same load and service demands.

Which model matches your real traffic?

Pick an open model for anything the public can reach, and a closed one only where the system itself limits how many users are active at once.

Gatling's docs name call centres where every operator is busy, and ticketing sites that queue users at full capacity, as closed systems, and say most websites behave as open ones. A batch job with a fixed number of workers or a queue consumer with set concurrency is closed too. Public APIs, web pages and webhooks are open: nothing holds back the next request because the last one was slow.

Many systems sit in between: a shopper arrives on their own schedule, then waits for each page within the visit. The paper above calls this a partly open model. In k6 or Gatling, set the arrival rate of sessions and make each iteration a whole user journey with its own pauses.

How do you set each model in k6?

You choose it with the executor. constant-vus and ramping-vus give you a closed model, while constant-arrival-rate and ramping-arrival-rate give you an open one.

A closed scenario sets executor: "constant-vus" with a number of vus and a duration. An open scenario sets a rate instead, and you pre-allocate enough virtual users to sustain it:

export const options = {
  scenarios: {
    open_model: {
      executor: "constant-arrival-rate",
      rate: 50,
      timeUnit: "1s",
      duration: "5m",
      preAllocatedVUs: 100,
    },
  },
  thresholds: {
    dropped_iterations: ["count<10"],
  },
};

k6 starts an iteration only while it has a free virtual user, and its docs advise against leaning on maxVUs, because adding users mid-test can overload the load generator. When no user is free, k6 records a dropped iteration, which late in a test often means the system is slowing down; the threshold above fails the run at ten. Leave sleep() out of the end of an arrival-rate iteration, since the executor already paces the starts (checked on 9 October 2026).

If you run k6 scripts on LoadFocus, its k6 cloud runner takes the load from the test's own settings, and its script validator warns that arrival-rate scenarios run there with a fixed number of users, so the request rate is not held. For a true open-model test, run the arrival-rate executors with k6 itself, or on a service that runs them as written.

How do Gatling, Locust and JMeter express each model?

Gatling separates the two models in its API, JMeter offers both through different thread groups, and Locust is built around a fixed number of users.

Gatling. A scenario uses either injectOpen or injectClosed. Open steps such as constantUsersPerSec and rampUsersPerSec set arrival rates, and closed steps such as constantConcurrentUsers and rampConcurrentUsers set concurrency. One injection profile cannot mix the two (checked on 9 October 2026).

Locust. You set a number of users, and each user's wait_time controls the pause between tasks. constant_throughput and constant_pacing adapt that pause so each user runs at most a set number of tasks per second, but the docs say wait time can only limit throughput and cannot start new users to reach a target. poisson() draws random pauses, which the docs call closer to real arrivals. Their example for 500 task iterations per second is constant_throughput(0.1) with 5,000 users, a closed pool with plenty of headroom (checked on 9 October 2026).

JMeter. The classic Thread Group is a closed pool of threads. The Constant Throughput and Precise Throughput timers pace that pool toward a rate, and the reference warns that throughput falls if the server cannot keep up or there are not enough threads. For arrivals, JMeter 5.5 added the Open Model Thread Group, still marked experimental, with schedules such as rate(10/sec) random_arrivals(1 min). The community Arrivals Thread Group plugin starts new threads when all existing ones are busy, with a concurrency limit as a safety valve (checked on 9 October 2026).

What should you watch in an open-model test?

Watch the load generator as closely as the system under test. In an open model, a slow system means more concurrent users on your side, which cost memory and CPU.

Set a ceiling, such as maxVUs in k6 or the plugin's concurrency limit in JMeter, and treat reaching it as a result: the system could not absorb the rate you asked for.

FAQ

Is a closed workload model ever the right choice? Yes. Use it when the real system caps concurrency, such as a fixed pool of workers, a queue consumer or a call centre, or when you want to know how many simultaneous sessions a component can hold. For public web traffic an open model is the better fit.

Does adding think time turn a closed model into an open one? No. Think time lowers how often each user sends requests, but every user still waits for the previous response, so a slow system still throttles the test. Only an arrival rate set independently of response times gives you an open model.

How many virtual users does an arrival-rate test need? Enough to cover every iteration in flight at the same time. The k6 documentation suggests the median iteration duration multiplied by the target rate, plus a margin for variance, found through short trial runs before the full test.

Think time in load testing: what it is and how to set it

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

Think time in load testing is the pause a virtual user takes between two actions, standing in for the seconds a real person spends reading a page or filling in a form. It decides how hard each virtual user pushes: the longer the pause, the fewer requests per second the same number of users sends. Measure it from real-user data, randomise it around that value, and write it into the script with your tool's own pause command.

Think time per loop	One loop takes	Requests per second from 100 users
0 s	0.5 s	200
1 s	1.5 s	about 67
3 s	3.5 s	about 29
5 s	5.5 s	about 18
10 s	10.5 s	about 10

This is arithmetic with no test behind it: one request per loop, a 0.5-second response, and users divided by (response time + think time).

What is think time in load testing?

Think time is the deliberate wait a test script inserts between the steps of a user journey. It models the human part of a session: reading, comparing, typing.

In a shop script it sits between "open the product page" and "add to cart", outside any request. LoadFocus's guide to delays in load testing separates it from a ramp-up delay, which adds users gradually, and a startup delay between users starting, and notes that many tools use the bare word "delay" for the ramp-up kind. When a tool offers a "delay" field, check which one it means.

Why does think time change the load from the same number of users?

With a fixed number of virtual users, each one waits for a response, pauses, and only then sends the next request. Its loop takes response time plus think time, so every added second of think time lowers the requests each user sends per second.

One hundred users with no pause send 200 requests per second against a 0.5-second response; with a 5-second pause they send about 18, so two runs "at 100 users" can be very different tests. The k6 documentation describes the same effect for its closed model: the next iteration starts only when the previous one finishes, so slower responses also lower the rate (checked on 9 October 2026).

Cutting think time to get more load from fewer users has a cost: twenty users with no pauses can match the request rate of several hundred who pause, while holding far fewer sessions and connections open than production would.

What is the difference between think time and pacing?

Think time is a pause after a step, so a loop gets longer whenever responses slow down. Pacing fixes the time from the start of one loop to the start of the next, and shortens the wait when the work took longer.

With 30 seconds of pacing, a user starts a loop every 30 seconds whether its requests took 2 seconds or 20, which suits a test built from business volumes such as "500 orders an hour". The docs for Locust's constant_pacing spell out the edge case: if the task runs longer than the interval, the wait drops to zero (checked on 9 October 2026). Many scripts use both, with think time between steps and pacing on the whole loop.

Should think time be fixed or random?

Random, around an average you measured. A fixed pause keeps users that started together in step, so they hit the same page at the same moment, loop after loop.

Those synchronised waves can create spikes production never sees, and a random pause spreads users out. Keep the average where your data puts it: a uniform pause between 2 and 8 seconds averages 5, the same mean load as a fixed 5-second pause. Uniform is the simple default; a normal (Gaussian) shape clusters pauses around a typical value, and an exponential one gives many short waits and a few long ones.

How do you choose think time values?

Take them from what real users do on each step, using web analytics, access logs or real-user monitoring. Use a separate value per step, because reading a product page takes longer than pressing a button.

Time on page in analytics, or the gap between consecutive requests of one session in access logs, is close to the think time for that step. Use the median as the average pause and the spread to set the random range, and drop idle gaps such as a tab left open over lunch.

An API called by other programs has no human in the loop, so model its request rate directly. Without data, write your assumed values into the test plan so a later run can replace them; the step-by-step guide to running a load test shows where they fit.

How do you add think time in k6, JMeter, Gatling, Locust and Artillery?

Each tool has its own pause command, and most also have a pacing or rate option.

Tool	Fixed pause	Random pause	Pacing or rate
k6	`sleep(3)`	`sleep(2 + Math.random() * 6)`	Arrival-rate executors
JMeter	Constant Timer	Uniform or Gaussian Random Timer	Constant Throughput Timer
Gatling	`pause(3)`	`pause(2, 8)`	`pace(30)`
Locust	`constant(3)`	`between(2, 8)`	`constant_pacing`
Artillery	`think: 3`	`think` + `jitter` (in `config.http.defaults`)	Arrival rate per phase

The table above and the notes below come from each project's documentation, checked on 9 October 2026; all five tools appear in our review of open source load testing tools.

k6. sleep(t) pauses the virtual user for t seconds; the docs build random pauses with Math.random() or randomIntBetween() from k6-utils:

import http from "k6/http";
import { sleep } from "k6";

export default function () {
  http.get("https://example.com/products");
  sleep(2 + Math.random() * 6); // 2 to 8 seconds
  http.get("https://example.com/cart");
}

JMeter. Among the timers, Constant Timer takes a delay in milliseconds, the Uniform and Gaussian Random Timers add a random part to a fixed offset, and Constant Throughput Timer varies its pauses to hold a target of samples per minute. A timer runs before each sampler in its scope, so one timer at thread group level pauses before every request.

Gatling. pause(2, 8) waits a uniform random time, and pace(30) in a loop runs it at most once every 30 seconds. At simulation level you can switch all pauses to a normal or exponential distribution, or turn them off with disablePauses().

Locust. wait_time runs after each task, so a task with two requests under constant_throughput(2) sends four requests per second per user.

Artillery. A think step pauses for a number of seconds, and a think.jitter setting under config.http.defaults adds random variance to every think step. For two of these tools side by side, see k6 vs Artillery.

Does think time matter in an arrival-rate test?

Less, as far as the request rate goes. When the tool starts new iterations at a set rate, pauses decide how long each user stays and so how many are active at once.

k6's arrival-rate executors pace iterations through their rate setting, and the docs advise against a sleep at the end of the iteration; Artillery's load phases likewise create a set number of new virtual users per second (both checked on 9 October 2026). Think time between steps still shapes each session.

How do LoadFocus's script tools set think time?

They write it into the k6 or JMeter script as a starting value for you to adjust.

The import from cURL, HAR, Postman or OpenAPI turns a captured request into a k6 script that ends each iteration with sleep(1); treat that second as a placeholder for your measured values. LoadFocus's AI test generator is instructed to give every step a think time, writes it as sleep() for k6 or a Constant Timer for JMeter, and warns when a multi-step scenario has none.

FAQ

Can you run a load test with zero think time? Yes, to find the maximum throughput of an endpoint or for a stress test, if you report the result as requests per second. It says little about how many real users the system can serve.

Does think time count in response times? Not in request timings. In k6, http_req_duration covers sending, waiting and receiving, while iteration_duration covers the whole loop (metrics reference, checked on 9 October 2026). A JMeter Transaction Controller leaves timers out of its time unless you tick the option that includes them.

WebSocket load testing tools: six options compared

Fri, 09 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

The WebSocket load testing tools with native support are k6, Artillery, Gatling and Tsung, while JMeter needs the WebSocket Samplers plugin and Locust needs client code you write or its experimental Socket.IO user. Choose by the language your team writes, then by how closely you must check the messages the server pushes back.

We have not run these tools against a WebSocket server for this article. The facts come from each project's own documentation and repository, checked on 9 October 2026, and the "choose instead" notes are our judgement.

Tool	WebSocket support	Tests written in	Checking what the server sends
k6	Built in: `k6/websockets`, legacy `k6/ws`	JavaScript	Event handlers in your script
Artillery	Built-in `ws` and `socketio` engines	YAML, plus JavaScript functions	Socket.IO responses and acknowledgements
Gatling	Part of the HTTP SDK	Java, Kotlin, Scala, JavaScript	Checks that wait for an expected message
Apache JMeter	WebSocket Samplers plugin	Test plans built in the GUI	Request-response and read samplers
Locust	Your own client, or the experimental `SocketIOUser`	Python	Whatever your code reports
Tsung	Built-in `ts_websocket` session type	XML	`ack` waits for a server reply

What makes WebSocket load different from HTTP load?

A WebSocket user opens one connection and keeps it for the whole session, and the server can send messages whenever it likes. A test therefore has two dials, open connections and messages per second, and each loads the server in its own way.

k6's WebSocket guide describes full-duplex traffic over one TCP connection and explains that a virtual user stops looping over the main function and runs an event loop while the socket is open. Gatling's WebSocket reference adds that traffic flows both ways, so the HTTP request and response model no longer fits.

Plan three shapes of test. Many idle connections show how many sockets your servers can hold. A smaller crowd of chatty clients tests message handling. A broadcast, where one message fans out to every subscriber, tests the push path.

What should you measure in a WebSocket load test?

Measure connect time, message latency and dropped connections, alongside how many connections stay open. These show what users of a real-time app feel.

k6's built-in metrics include ws_connecting, ws_ping, ws_session_duration, ws_sessions and message counts. None of them times an application message, so the delay between a client's message and the server's reply needs a custom Trend metric. Artillery's ws engine reports message counts and rates, and its Socket.IO engine adds socketio.response_time, from emitting a message to receiving the response.

For dropped connections, count close and error events your script did not cause, and compare sessions started with sessions still open at the end of each stage.

When is k6 the right WebSocket tool?

When your team writes JavaScript and wants each virtual user to hold several sockets at once. k6 has WebSocket support built in, with no extension to install.

The k6/websockets module follows the browser WebSocket API and uses a global event loop, which lets one virtual user keep multiple connections open. The older k6/ws module is still documented, but k6 recommends k6/websockets for new tests, and k6/experimental/websockets is deprecated.

import { WebSocket } from 'k6/websockets';

export default function () {
  const ws = new WebSocket('wss://example.com/ws');
  ws.onopen = () => ws.send('hello');
  ws.onmessage = (e) => {
    console.log(e.data);
    ws.close();
  };
}

Thresholds work on custom metrics too, so a missed latency goal gives CI a non-zero exit code. Choose Artillery instead when your server speaks Socket.IO and you want emits and acknowledgements checked without writing client code. Our k6 vs Artillery comparison covers the wider differences.

When does Artillery fit?

When you want short YAML scenarios, or when your server uses Socket.IO. Both engines ship with Artillery itself.

A scenario on the WebSocket engine is built from five actions: connect, send, think, loop and function. The Socket.IO engine goes further: an emit can declare the response it expects, validate an acknowledgement with a JSONPath match, and share a scenario with HTTP requests, which suits apps that log in over HTTP before opening a socket.

Choose k6 or Gatling instead when a test must check what a plain WebSocket server pushes back, because the ws engine's documentation describes no step that waits for or matches a reply.

Who should use Gatling for WebSockets?

Teams on the JVM, or on Gatling's JavaScript SDK, who want message checks written into the scenario. WebSocket support is part of Gatling's HTTP SDK, so HTTP and WebSocket steps share one simulation.

The reference covers connect, sendText, sendBytes and close, plus checks that block until the expected message arrives or a timeout passes. Protocol settings can reconnect a socket the server closed, reply automatically to chosen text messages such as a ping, and buffer messages no check matched. Gatling's repository lists WebSocket in the Apache 2.0 Community Edition, with no limits.

Choose k6 instead when you prefer to react to pushed messages in event handlers over blocking checks.

Can JMeter load test WebSockets?

Yes, with a plugin. JMeter itself has no WebSocket sampler, and the usual add-on is WebSocket Samplers by Peter Doornbosch.

JMeter's component reference does not mention WebSocket at all. The plugin's repository describes six samplers: open connection, request-response, ping-pong, single read, single write and close. They run on the thread group's own thread, support wss and binary frames, and can filter out frames a test does not care about. You install it through the Plugins Manager or as a jar in lib/ext; version 1.3.2 was tagged in January 2026 (checked on 9 October 2026).

Choose another tool when you would rather not maintain a third-party plugin on every load generator.

What about Locust and Tsung?

Locust reaches raw WebSockets only through client code you add, and Tsung supports them in XML configuration but releases rarely. Both can work, with more effort.

Locust's documentation says its built-in support covers only HTTP and HTTPS; other protocols are added by wrapping a client library and firing a request event after each call. The library must be one gevent can patch, or it blocks the whole process. For Socket.IO there is an experimental SocketIOUser. Choose Locust when your team writes Python and is happy to own that client code.

Tsung is an Erlang project that calls itself a multi-protocol distributed load testing tool. Its session documentation defines WebSocket connect, message and close requests, with an ack attribute to wait for a response. Its latest tagged release, 1.8.0, dates from March 2023.

Where does a hosted runner like LoadFocus fit?

Holding many open sockets from your own machines takes work, and a hosted runner takes over the load generation. It fits best when your tests are already k6 scripts.

LoadFocus, the publisher of this site, runs k6 scripts from 25 cloud regions, and its k6 page lists WebSocket connections among the supported k6 features. It also offers a WebSocket ramp-up and ramp-down template. We have not checked whether it runs JMeter plugins such as WebSocket Samplers, so we make no claim there.

How do you choose between them?

Start with the language your team writes, then ask whether you test raw WebSockets or Socket.IO and how strictly you must check pushed messages. That usually leaves one or two candidates.

Write the same short connect, send and receive test in both and compare connect time and message latency at low load. Our guide on how to run a load test covers the steps, and the open source load testing tools roundup compares these projects beyond WebSockets.

FAQ

Which WebSocket load testing tool should I start with? If your team writes JavaScript, start with k6, which has WebSocket support built in. For Socket.IO apps, Artillery's engine needs the least code, and JVM teams will feel at home in Gatling.

Can JMeter test WebSockets without a plugin? Its component reference lists no WebSocket sampler, so the practical route is the WebSocket Samplers plugin, installed through the Plugins Manager or as a jar in lib/ext.

Should I use k6/ws or k6/websockets? Use k6/websockets for new tests, as k6's documentation recommends. The legacy k6/ws module is still documented for existing scripts.

How to run a load test: a step-by-step guide

Thu, 08 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, a load testing service mentioned in this article.

To run a load test, decide what a good result looks like, script one realistic user journey, check that script with a single user, then increase the load in steps while you watch response times and errors. Compare what you measured with the goal you set at the start, and change one thing at a time before the next run.

Step	What you do	Done when
1. Set the goal	Pick the load to test for and the numbers that count as a pass	You can write the pass criteria in one sentence
2. Script a journey	Turn one real user path into a script	The script runs without errors
3. Smoke test	Run it with one virtual user	Every check passes
4. Ramp up	Increase the load in steps and hold at the target	You reach the target or a stop condition
5. Read the results	Compare percentiles and errors with the goal	You know whether it passed and what to change

What should you decide before running a load test?

Decide what load you are testing for and what result counts as a pass. Without both, a load test produces numbers but no answer.

Start from real traffic if you have it. Your analytics or access logs tell you the busiest hour, its requests per second and which endpoints take most of them. Set the target above that peak, because the point is to know you have headroom. Then write the pass criteria as numbers, for example: the 95th percentile response time stays under 500 ms and fewer than 1% of requests fail at 200 virtual users.

Two practical decisions belong here too. Pick an environment that looks like production in size and configuration, because a staging server with a quarter of the capacity tells you little about production. And make sure you have permission to test it: load testing a system you do not own, or one shared with other teams, without telling them can look exactly like an attack.

How do you script a realistic user journey?

Script the path that matters most to the business, one step at a time, with pauses where a real person would read or type. A script that hits one URL as fast as it can measures something, but rarely the thing you care about.

For a shop: open the home page, search, open a product, add it to the cart. For an API: log in, list items, fetch one item. Add a short pause between steps (think time), use varied test data so caches behave the way they do in real life, and add a check on each response so a page of errors cannot pass as a fast response.

Here is the script we used for the example run in this guide, written for k6. It ramps up to 10 virtual users, holds, and ramps down, with two pass criteria built in:

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "10s", target: 10 },
    { duration: "20s", target: 10 },
    { duration: "5s", target: 0 },
  ],
  thresholds: {
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  const res = http.get("http://127.0.0.1:8765/");
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

A real journey has several requests and real URLs. This one stays tiny so the steps are easy to follow.

Why run a single-user smoke test first?

A smoke test with one virtual user proves the script works before you put real load on anything. It is the cheapest way to catch a wrong URL, a missing login token or a check that always fails.

With k6 you can override the load from the command line, so the same script runs once with a single user:

k6 run --vus 1 --iterations 1 script.js

When we ran it, the single check passed, so the script was ready for load. If a check fails at this stage, fix the script before going further: errors at one user turn into noise at a hundred.

How should you ramp up the load?

Increase the load in steps and hold each step long enough to settle, instead of jumping straight to the target. A step-wise ramp shows you the level at which things start to degrade.

A common shape is to ramp up to a first step, hold for a few minutes, ramp to the next step, hold again, and so on up to the target, then ramp down. Watching each hold separately tells you whether response times climb smoothly with load or stay flat and then jump. That jump, the point where every extra user makes every request slower, is the most useful thing a load test can find. Our example holds 10 users for 20 seconds: enough to show the stages, far too short for a real test.

What should you watch while the test runs?

Watch response time percentiles, the error rate and the throughput on the load generator side, and CPU, memory and database load on the system under test. The first three tell you how users would feel, and the last three tell you why.

Agree on stop conditions before you start, for example: stop if the error rate passes 5% or response times pass several seconds. Pushing a system that has already fallen over teaches nothing, and on a shared environment it gets in other people's way.

Watch the load generator too. If it is at full CPU, the slow responses may come from your own laptop rather than the server.

How do you read the results of a load test?

Read the percentiles and the error rate against the pass criteria you wrote down at the start. The average response time is the least useful number in the report.

Here is what our example run reported (35 seconds, up to 10 virtual users, a small static page served on a laptop, run on 8 October 2026):

Metric	Value
Requests	282 (8 per second)
Failed requests	0%
Average response time	0.69 ms
90th percentile	0.66 ms
95th percentile	0.71 ms
99th percentile	2.68 ms
Slowest request	15.98 ms
Thresholds	Both passed

The numbers are tiny because the target was a static file on the same machine, so they are no benchmark. The shape is the lesson: the average came out higher than the 90th percentile, because a handful of slow requests pulled it up. Percentiles describe what most requests experienced, and the 99th percentile and the slowest request show the tail that a few users hit. Judge the run on the percentile in your pass criteria, and use the tail to decide whether something needs a closer look.

If a run misses its goal, change one thing (a query, a cache setting, an instance size) and run the same test again, so you know what made the difference.

When should you run the test from the cloud?

Move the load generator to the cloud when one machine cannot produce the load you need, or when it matters where your users are. A laptop on office Wi-Fi measures your office connection as much as your server.

The same k6 script can run on infrastructure you manage, on Grafana's hosted service, or on LoadFocus, which runs k6 scripts from 25 cloud regions. Its reports break results down by location and plot response time against the number of virtual users, which is where the jump described above becomes visible (how to read LoadFocus results).

FAQ

How long should a load test run? Long enough to hold the target load for several minutes after the ramp-up, so short spikes do not decide the result. Holding for at least 10 minutes is a sensible start, and much longer when you are looking for memory leaks or slow degradation.

How many virtual users do I need? Work it out from real traffic. Take the busiest hour's requests per second, estimate how many requests one user journey makes and how long it takes, and size the virtual users so the test produces that rate with some headroom on top.

Is it safe to load test production? It can be, with care: test at a quiet time, tell the people who run the system, set stop conditions and start well below the expected peak. Many teams prefer a production-sized staging environment, so a failed test cannot affect customers.

k6 vs Artillery: which load testing tool fits your team?

Thu, 08 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

k6 and Artillery are both open source, code-first load testing tools, and either one will load test an HTTP API well. Pick k6 if you want a single binary, test logic in JavaScript and pass/fail thresholds built in. Pick Artillery if your team already lives in Node.js, prefers short YAML scenarios, or wants Playwright browser tests and distributed runs on AWS or Azure from the command line.

The facts below come from each project's own documentation and repository, checked on 8 October 2026.

	k6	Artillery
Licence	AGPL-3.0 (repository)	MPL-2.0, some Azure modules under BSL (repository)
Runtime	One Go binary with an embedded JavaScript engine	Node.js, latest LTS release
Test scripts	JavaScript	YAML, TypeScript or JavaScript
Core protocols	HTTP, WebSockets, gRPC	HTTP, WebSocket, Socket.IO (gRPC and Kinesis as add-on engines)
Browser testing	Built-in browser module	Playwright
Pass/fail checks	Thresholds	`ensure` plugin
Scaling out	k6-operator on Kubernetes, or Grafana Cloud k6	`run-lambda` or `run-fargate` on AWS, `run-aci` on Azure

What is the difference between k6 and Artillery?

They differ mainly in runtime and in how you describe load. k6 is a compiled Go tool that runs JavaScript test code, while Artillery is a Node.js tool that reads YAML or JavaScript scenarios.

That difference shows up on day one. Installing k6 means one package or a standalone binary, with no language runtime to manage. Artillery needs Node.js and installs through npm, which feels natural in a JavaScript project and is one more runtime to keep current anywhere else.

On protocols the overlap covers most web work: both speak HTTP and WebSockets out of the box. k6 also ships with gRPC, and Artillery with Socket.IO. Artillery gets gRPC and Kinesis from add-on engines that you install separately (checked on 9 October 2026), and k6 reaches further through its extension ecosystem.

How do you write a test in each tool?

In k6 you write a JavaScript module with options at the top and a default function that each virtual user runs. In Artillery you usually write a YAML file with a config block for the load and a scenarios block for what each virtual user does.

A k6 test that ramps to 20 virtual users and fails if the 95th percentile response time goes above 500 ms:

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "30s", target: 20 },
    { duration: "1m", target: 20 },
    { duration: "15s", target: 0 },
  ],
  thresholds: { http_req_duration: ["p(95)<500"] },
};

export default function () {
  const res = http.get("https://staging.example.com/api/users");
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

The same idea in Artillery's YAML format, with ten new virtual users arriving every second for a minute:

config:
  target: "https://staging.example.com"
  phases:
    - duration: 60
      arrivalRate: 10
  plugins:
    ensure:
      thresholds:
        - "http.response_time.p95": 500
scenarios:
  - flow:
      - get:
          url: "/api/users"

The two examples model load in different ways. The k6 script sets a number of concurrent virtual users, and the Artillery file sets how many new virtual users arrive each second. k6 can also drive arrival rates through its scenario executors, and Artillery can cap concurrency with maxVusers, so choose the model that matches how real traffic reaches your system rather than the one each tool shows first.

How do k6 and Artillery decide whether a test passed?

Both turn a performance goal into an exit code, which is what a CI pipeline needs. A missed goal ends the run with a non-zero exit code in either tool.

In k6 the goals are thresholds, set in the script's options as in the example above. In Artillery they live in the ensure plugin, which supports simple thresholds and conditions that combine several metrics.

The practical difference is where the goals sit. k6 thresholds live next to the test logic in the same file. Artillery's checks live in the config block, where someone who never touches the scenarios can still adjust them.

How do you run a bigger test than one machine can generate?

Both tools can spread a test across many machines, through quite different routes. k6 leans on Kubernetes or Grafana's hosted service, and Artillery leans on your own AWS or Azure account.

For k6, the k6-operator splits a test across pods in a Kubernetes cluster, and Grafana Cloud k6 runs the same scripts as a hosted service. For Artillery, the run-lambda and run-fargate commands launch workers in your AWS account after a one-time IAM setup through a CloudFormation stack. On Azure, run-aci does the same with Azure Container Instances, though using it beyond evaluation needs a paid Artillery subscription (checked on 9 October 2026). If you already run Kubernetes, the operator route will feel familiar. If you would rather not operate anything, Artillery's serverless route has fewer moving parts, as long as the test can live in AWS or Azure.

There is a third route for k6 scripts. LoadFocus runs k6 scripts on its own cloud infrastructure from 25 regions, so the script you run on your laptop becomes a distributed test without a cluster or an AWS account. LoadFocus does not run Artillery scripts, so for Artillery its own AWS and Azure commands remain the way to scale.

Where do the results go?

Both tools print a summary at the end of a run and can send metrics to other systems for dashboards and alerting.

k6 offers summary statistics or granular metrics exported to the service of your choice, according to its repository. Artillery lists more than 20 integrations for monitoring, observability and CI/CD in its repository. Either tool can feed the dashboards you already have. Check that the specific output you need is on the list before you commit, because both lists change between releases.

Which one should you choose?

Choose k6 when you want one binary in CI, JavaScript test logic with real control flow, and thresholds in the same file as the test. It also suits teams that already use Grafana, since the open source tool and the hosted service share one scripting API.

Choose Artillery instead when your team writes Node.js every day, when a short YAML file is easier for the rest of the team to review than a script, or when you need Playwright browser tests and Socket.IO in the same tool. Its serverless scaling is also the shorter path if your infrastructure already lives in AWS and Kubernetes is not part of it.

If you are still undecided, write the same smoke test in both. Twenty lines are enough to see which one your team finds easier to read in a code review.

FAQ

Is k6 free to use? Yes. k6 is open source under the AGPL-3.0 licence and runs on your own machines at no cost. Grafana Cloud k6 is a separate hosted service with its own pricing.

Is Artillery free to use? Mostly. Most of Artillery is under the MPL-2.0 licence, while some Azure-specific modules use a source-available licence that requires a commercial licence for production use on Azure, according to the project repository (checked on 8 October 2026).

Can LoadFocus run Artillery tests? No. LoadFocus runs k6 and JMeter scripts. To scale an Artillery test beyond one machine, use Artillery's own commands for AWS Lambda, Fargate or Azure Container Instances in your own cloud account.

Which tool uses fewer resources on the load generator? We have not measured this ourselves, so we will not name a winner. The k6 project says even lower-end machines can simulate a lot of traffic, and the fair way to compare is to run the same scenario with both tools on your own hardware.

Open source load testing tools: six options reviewed

Thu, 08 Oct 2026 00:00:00 GMT

Disclosure: loadtest.app is published by LoadFocus, which sells a load testing service covered in this article.

Six widely used open source load testing tools today are Apache JMeter, k6, Gatling, Locust, Artillery and Vegeta. Choose by the language your team writes tests in and by how you need to scale: JMeter for the widest protocol coverage, k6 or Artillery for JavaScript, Gatling for the JVM, Locust for Python and Vegeta for quick fixed-rate HTTP checks.

We have not benchmarked these tools against each other for this article. The facts come from each project's own repository and documentation, checked on 8 October 2026, and the "good for" notes are our judgement.

Tool	Licence	Tests written in	Runs on	Spreading load across machines
Apache JMeter	Apache 2.0	Test plans built in a desktop GUI	Java	Built-in remote (client-server) mode
k6	AGPL-3.0	JavaScript	One Go binary	k6-operator on Kubernetes
Gatling	Apache 2.0 (some components under an enterprise licence)	Java, Kotlin, Scala, JavaScript, TypeScript	JVM	Commercial Enterprise edition
Locust	MIT	Python	Python	Built in
Artillery	MPL-2.0 (some Azure modules under BSL)	YAML, JavaScript, TypeScript	Node.js	AWS Lambda, Fargate or Azure ACI from the CLI
Vegeta	MIT	Command line or Go code	One Go binary	Run several instances

Is Apache JMeter still worth using?

Yes, especially when you need protocols beyond HTTP. JMeter covers the widest range of the six and has the largest body of existing test plans and know-how behind it.

JMeter is a Java application under the Apache 2.0 licence. Its protocol list covers HTTP and HTTPS, SOAP and REST web services, FTP, databases through JDBC, LDAP, JMS messaging, mail and raw TCP. You build test plans in a desktop GUI, run them headless from the command line for real tests, and use the client-server remote mode to spread a test across machines.

Good for mixed-protocol systems, teams with existing JMeter plans and testers who prefer to build tests visually. Choose something else when your team wants tests as plain code that is reviewed like the application, because GUI-built XML plans are hard to read in a pull request.

When is k6 the right choice?

When your team writes JavaScript and wants tests in code with pass/fail thresholds built in. It ships as a single Go binary, which keeps CI setup simple.

k6 is licensed under AGPL-3.0 and speaks HTTP, WebSockets and gRPC, with a browser module for front-end checks. Thresholds turn a goal such as a 95th percentile limit into a non-zero exit code for CI. To scale out, the k6-operator runs distributed tests on Kubernetes, and Grafana sells a hosted service.

Good for developers who keep API tests next to their code, and for CI pipelines. Choose something else when your team does not want JavaScript, or when the AGPL licence conflicts with how you embed or ship the tool. We compare it head to head in k6 vs Artillery.

Who should use Gatling?

Teams on the JVM who want tests as code and a detailed HTML report out of the box. It also offers a JavaScript and TypeScript SDK on the same engine.

Gatling runs virtual users on a non-blocking engine built on Netty, so one machine can drive a lot of load. Simulations are written in Java, Kotlin, Scala, JavaScript or TypeScript and run through Maven, Gradle, sbt or npm plugins. The community edition is Apache 2.0 and covers HTTP/1.1, HTTP/2, WebSocket, Server-Sent Events and JMS, while its gRPC, MQTT and GraphQL components use Gatling's enterprise component licence.

Good for Java and Scala teams, and for anyone who wants rich reports without extra tooling. Choose something else when you need to spread load across machines from the open source edition alone: generating load from managed or private infrastructure is what the commercial Gatling Enterprise is for.

Is Locust a good choice for Python teams?

Yes. Locust tests are plain Python, so loops, conditions and any Python library work the way they do in ordinary code.

Locust is MIT-licensed and event-based on gevent, which lets a single process handle many users. Tests run from the command line or from a web UI that shows throughput, response times and errors live, and Locust distributes a test across multiple machines on its own. HTTP is built in, and other protocols are added by writing Python.

Good for Python teams, scenarios with complex logic and anyone who likes watching a test in a browser. Choose something else when you want declarative, config-style tests, or when a ready-made plugin for an unusual protocol matters more than writing your own.

When does Artillery make sense?

When your team lives in Node.js, likes short YAML scenarios, or wants Playwright browser tests in the same tool as its API tests.

Artillery is mostly MPL-2.0, with some Azure modules under a source-available licence. It ships with HTTP, WebSocket and Socket.IO, adds gRPC and Kinesis through add-on engines, and its run-lambda and run-fargate commands spread a test across workers in your own AWS account. A run-aci command does the same on Azure Container Instances, with a paid Artillery subscription for anything beyond evaluation (checked on 9 October 2026).

Good for Node.js teams, mixed API and browser testing, and companies already on AWS. Choose something else when you would rather not run Node.js on your load generators, or when large tests have to run outside AWS and Azure.

What is Vegeta good for?

Quick, constant-rate HTTP checks from the command line. It does one job, sending requests at a fixed rate, and keeps that job simple.

Vegeta is an MIT-licensed Go tool that works as a command line program or as a Go library, with a CLI designed to compose with other Unix tools. Because it holds the request rate constant rather than modelling users, it answers questions like "what happens at 500 requests per second" better than it models user journeys.

Good for smoke-checking an endpoint, fixed-rate capacity checks and shell scripts. Choose something else when you need multi-step journeys, think time or detailed reports.

Where does a hosted runner like LoadFocus fit?

None of these tools needs a hosted service, but generating large or geographically spread load from your own machines takes work. A hosted runner takes over the load generation and leaves your scripts as they are.

LoadFocus, the publisher of this site, is one such service. It runs existing JMeter test plans and k6 scripts from 25 cloud regions. It does not run Gatling, Locust, Artillery or Vegeta tests, so for those tools the scaling options above apply. Grafana Cloud k6 and Gatling Enterprise are the vendors' own hosted routes for k6 and Gatling.

How do you choose between them?

Start from the language your team already writes and the protocols you must test, then check how each tool scales. That usually narrows six options to one or two.

Then write the same short test in your top two choices. Reading both side by side settles most debates faster than any feature table, and our guide on how to run a load test walks through that first test step by step.

FAQ

Which open source load testing tool is the easiest to start with? For a single HTTP endpoint, Vegeta or k6 give you a result within minutes because each is one binary. For multi-step journeys, pick the tool whose test language your team already writes.

Can open source load testing tools run distributed tests? Yes, through different routes: JMeter has a built-in remote mode, Locust distributes across machines natively, k6 uses the k6-operator on Kubernetes and Artillery uses AWS Lambda, Fargate or Azure Container Instances. For Gatling, load generation from managed or private infrastructure is part of the commercial Enterprise edition.

Is JMeter better than k6? Neither is better in general. JMeter covers more protocols and suits GUI-based test design, while k6 suits developers who want tests as JavaScript code in CI.

loadtest.app

Amdahl's law: formula, examples and scaling limits

What does the law say in simple terms?

Where does the law come from?

What does it mean for load and capacity testing?

How do you find the serial part in a load test?

What is Gustafson's law?

What does the Universal Scalability Law add?

How do you fit the USL to a load test series?

Where do these models stop being useful?

FAQ

Related reading

Queueing theory: what it is and why latency spikes

What is queueing theory?

Where did queueing theory come from?

What does Kendall's notation such as M/M/1 mean?

How does response time grow with utilization?

Why does latency knee before the CPU reaches 100%?

Why does variability matter as much as utilization?

How do more servers (M/M/c) help?

How does Little's law fit in?

Where do these models break down in load testing?

FAQ

Related reading

SLO vs SLA: the difference, SLIs and error budgets

What do SLI, SLO and SLA stand for?

What is an error budget?

How much downtime does 99.9% allow?

What does a real SLA look like?

How do you choose SLIs for latency and availability?

Where do SLOs go wrong?

What does load testing have to do with SLOs?

FAQ

Related reading

TPC-H: what the benchmark measures and how to run it

What does TPC-H measure?

How is the TPC-H score calculated?

Who publishes TPC-H results, and which ones count?

How do you run TPC-H yourself?

How do you read TPC-H results honestly?

When should you use something other than TPC-H?

FAQ

Related reading

redis-benchmark: how to use it and read its numbers

How do you run redis-benchmark?

How do you read redis-benchmark output?

Why does pipelining change the numbers so much?

What happens when you add more clients?

How do you benchmark your own command or a Lua script?

Why does your application see different numbers?

Are these laptop numbers Redis's real capacity?

When is redis-benchmark the wrong tool?

FAQ

Related reading

How to use sysbench for CPU, disk and MySQL benchmarks

Is sysbench still maintained?

How does a sysbench run work?

How do you benchmark CPU and memory with sysbench?

How do you benchmark MySQL or PostgreSQL with sysbench?

Which sysbench options matter most?

Does sysbench use a closed or an open workload model?

How do you read sysbench output?

When is sysbench the wrong tool?

FAQ

Related reading

YCSB: workloads A-F, how to run it and its limits

What is YCSB, and is it still maintained?

How do you run a YCSB benchmark?

What do -threads and -target control?

How do you read YCSB output?

Does YCSB account for coordinated omission?

Which databases can YCSB benchmark?

When is YCSB the wrong tool?

FAQ

Related reading

pgbench: how to benchmark and load test PostgreSQL

What does pgbench measure?

How do you set up a pgbench database?

Which built-in scripts does pgbench include?

How do you write a custom pgbench script?