Performance testing

Functional tests ask whether the API gives the right answer. Performance tests ask how long the answer takes, and what happens to that time when many clients ask at once. An endpoint that returns the correct product in 9 seconds is broken for anyone waiting on a phone.

This lesson covers measuring a single endpoint properly, reading the numbers, and running a small load test. The API in these lessons is shared by everyone who uses the tutorial, so the load examples here are deliberately tiny.

Latency and throughput

Latency is how long one request takes, from the moment the client sends it to the moment the response has arrived. Throughput is how many requests the system handles per second. They are different numbers and they fail differently.

With a few clients, latency is usually flat. As you add clients, the server spends more time waiting for CPU, database connections or threads, and requests start to queue. Throughput keeps rising for a while, then levels off at whatever the system can manage. Past that point, extra clients don't get more work done. They only make every request slower, and eventually some time out. Finding where that point is, and whether it is comfortably above your real traffic, is most of what load testing is for.

Why averages lie

The endpoint /api/chaos/random-latencysign in to see is built to behave like a real service with a slow tail. According to its documentation, about 90% of calls take 10 to 100 ms on the server, 9% take 500 to 1500 ms, and 1% take 3 to 5 seconds. That shape is common in practice: most requests hit a cache or a fast query, and a few hit a cold path, a lock or a garbage collection pause.

This script sends 30 requests one after another, records how long each took from the client's side, and prints the mean and some percentiles. It needs Node 18 or later for the built-in fetch. Save it as latency.mjs.

const URL = "https://api.testingapis.com/api/chaos/random-latency";
const N = 30;

const timings = [];
for (let i = 0; i < N; i++) {
  const start = performance.now();
  const res = await fetch(URL);
  await res.text();
  timings.push(performance.now() - start);
}

timings.sort((a, b) => a - b);
const pct = (p) => timings[Math.ceil((p / 100) * timings.length) - 1];
const mean = timings.reduce((s, t) => s + t, 0) / timings.length;

console.log(`requests: ${timings.length}`);
console.log(`mean: ${mean.toFixed(0)} ms`);
console.log(`p50:  ${pct(50).toFixed(0)} ms`);
console.log(`p95:  ${pct(95).toFixed(0)} ms`);
console.log(`p99:  ${pct(99).toFixed(0)} ms`);
console.log(`max:  ${timings.at(-1).toFixed(0)} ms`);

One run from a laptop on a home connection:

$ node latency.mjs
requests: 30
mean: 328 ms
p50:  320 ms
p95:  409 ms
p99:  536 ms
max:  536 ms

A percentile answers the question "how slow were the slowest N% of requests". The p50, also called the median, is the value half the requests beat. The p95 means 95% of requests were at least that fast and 5% were slower. The p99 describes the worst 1%.

In this run the mean and the median are close, and nothing was slow. That happens: with a 10% chance of a slow call on each request, 30 requests will sometimes dodge all of them. An earlier run of the same script on the same day got a mean of 624 ms, a median of 353 ms, a p95 of 1800 ms and a p99 of 5153 ms. The median barely moved. The mean nearly doubled, because a couple of multi-second requests were enough to drag it up. If you only report the mean, you can't tell the difference between "everything got a bit slower" and "most requests are fine but one user in a hundred waits five seconds". Those need different fixes.

Two more things to notice. First, with 30 samples, the p99 is the single slowest request, because 99% of 30 rounds up to all 30. You need a few hundred samples before a p99 means much, and thousands before it is stable. Second, the server says most calls take under 100 ms, but the median here was 320 ms. The difference is the network: distance to the server, the TLS handshake, and the time for the bytes to travel back. Where you measure from changes the numbers, so always write down where that was.

Where the time goes

A single number for the whole request hides which part was slow. curl can break it down with -w, which prints variables after the transfer finishes. On Windows, run this in Git Bash or WSL, or replace /dev/null with NUL and use curl.exe in PowerShell.

curl -s -o /dev/null -w "dns:     %{time_namelookup}\nconnect: %{time_connect}\ntls:     %{time_appconnect}\nttfb:    %{time_starttransfer}\ntotal:   %{time_total}\n" https://api.testingapis.com/api/health

dns:     0.074056
connect: 0.102971
tls:     0.178730
ttfb:    0.501674
total:   0.501782

The values are in seconds and each one counts from the start of the request, so you subtract to get the length of each step:

The waiting step includes the server's processing time plus one trip across the network and back. If that step grows while the others stay the same, the server got slower. If DNS or connect time jumps, the problem is between you and the server, and no amount of backend tuning will fix it.

Cold and warm requests

You can pass curl several URLs in one command, and it reuses the connection for the later ones. Here are two requests to /api/health in one command, using the same -w format for each:

dns:     0.006454
connect: 0.054908
tls:     0.130421
ttfb:    0.404310
total:   0.404372
dns:     0.000000
connect: 0.000000
tls:     0.000000
ttfb:    0.264396
total:   0.264467

The second request shows zero for DNS, connect and TLS, because none of them happened. It went over the connection the first request had already opened, and finished in 264 ms instead of 404 ms. Browsers, mobile apps and most HTTP libraries reuse connections the same way.

This is the simplest form of a cold versus warm difference, and there are others. A server that started a few seconds ago may have empty caches and code that hasn't been optimised by the runtime yet. Some hosting platforms stop idle servers and start them again on the next request, which can add several seconds to that one call. When you measure, run a few warm-up requests first and leave them out of the numbers, or report cold and warm timings separately. Mixing them makes both meaningless.

Kinds of load test

Everything so far used one client sending one request at a time. A load test runs many clients at once. People use the following names, though teams are not always consistent about them.

• A load test runs the traffic you expect on a normal busy day, for long enough to see whether latency and error rates stay within your targets.
• A stress test keeps increasing the load past that level until something breaks. The point is to learn what breaks first and whether the system recovers when the load drops.
• A spike test jumps from low to very high load in seconds, the way traffic does when a push notification goes out or a sale starts.
• A soak test runs normal load for hours. It finds problems that build up slowly: memory leaks, connection pools that never give connections back, log files filling a disk.

Load testing has a precondition that is easy to forget: you need permission, and you need an environment you are allowed to hurt. Run it against your own staging environment, sized like production if possible. Never point a load test at someone else's server. That includes this one. The API in these lessons is shared, so if you try the script below, keep it to a few virtual users for 30 seconds, as written.

A small load test with k6

k6 is a free, open-source load testing tool. You write the test in JavaScript and run it from the command line. Other tools such as JMeter, Gatling and Locust do the same job; k6 is used here because the scripts are short. Install it from the k6 website, save this as load.js, and run k6 run load.js.

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  vus: 3,
  duration: "30s",
  thresholds: {
    http_req_duration: ["p(95)<800"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  const res = http.get("https://api.testingapis.com/api/Products?limit=10");
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

The options block starts 3 virtual users. Each one runs the default function in a loop for 30 seconds: send a request, check the status, wait one second. Because of the sleep(1), that works out to roughly 3 requests per second, around 90 in total. Without a sleep, each virtual user sends its next request the moment the last one finishes, which is far more load than the number 3 suggests.

The thresholds are the assertions. This script fails if the p95 of request duration goes over 800 ms, or if more than 1% of requests fail. The 800 ms figure is an example. Pick yours from what your users need, and from what a measurement like the one above says is realistic from where the test runs.

When the run ends, k6 prints a summary. The line to read first is http_req_duration, which shows the average, minimum, median, maximum, p90 and p95 of request times. http_req_failed is the share of requests that failed, and http_reqs is the total count with the rate per second, which is your throughput. The checks line shows how many of your check() calls passed. Each threshold is marked as passed or failed, and if any threshold fails, k6 exits with a non-zero exit code. That last part is what lets a CI pipeline treat a slow build like a failing one.

A failed check does not fail the run by itself. Checks are counted and reported, and only thresholds decide the result. If a check matters, add a threshold on it, for example checks: ["rate>0.99"].

Performance budgets

A performance budget is a limit written down in advance, such as "p95 under 500 ms for the product list at 50 requests per second". Without one, a load test produces numbers that everybody looks at and nobody acts on. With one, it produces a pass or a fail, the same as any other test.

You can also put a loose time limit in ordinary functional tests. This is a test in the style of the automation lesson:

test("product 1 responds within 2 seconds", async () => {
  const start = performance.now();
  const res = await fetch(`${BASE_URL}/api/Products/1`);
  await res.json();
  const ms = performance.now() - start;
  assert.equal(res.status, 200);
  assert.ok(ms < 2000, `took ${Math.round(ms)} ms, budget is 2000 ms`);
});

Keep a budget like this generous. A single request in a functional test is one sample, taken on a shared CI machine over a network you don't control, and it will occasionally be slow for reasons that have nothing to do with your code. A limit of 2 seconds catches an endpoint that has gone from 200 ms to 8 seconds, which is the kind of regression you want to hear about. Tight limits belong in load tests, where you have hundreds of samples and percentiles to judge them by. If a single-request budget fails now and then for no reason, that is a flaky test, and the next lesson covers what to do about those.

Record results over time, too. A p95 that creeps from 180 ms to 260 ms over ten releases never fails a 500 ms budget, but it tells you something is getting worse, and it is much cheaper to find the cause after one release than after ten.