Timeouts, retries and rate limits

Everything so far assumed a healthy server that answers every request in a few hundred milliseconds. Production is less polite. Servers get slow under load, return 503 for the thirty seconds a deploy takes, drop connections, and refuse requests when a client sends too many. None of that shows up in a test suite that only runs against a healthy server.

This lesson covers two sides of the same problem. One is testing the features an API provides for bad days: idempotency keys, rate limits and Retry-After headers. The other is testing clients, including your own test code, when the server misbehaves. This API has a Chaos group built for the second part.

Timeouts

fetchhas no timeout by default. If the server accepts the connection and never answers, the request waits until something lower down gives up, which can take minutes. A test runner eventually kills the test, and you get a vague "test timed out" with no hint about which request hung.

Set a timeout on every request. /api/chaos/timeoutsign in to see hangs for as many seconds as you ask, so it is easy to see what happens:

const BASE = "https://api.testingapis.com";

try {
  const res = await fetch(`${BASE}/api/chaos/timeout?seconds=10`, {
    signal: AbortSignal.timeout(2000),
  });
  console.log("got", res.status);
} catch (err) {
  console.log(err.name, err.message);
}

TimeoutError The operation was aborted due to timeout

curl does the same with -m, the maximum time in seconds:

$ curl -sS -m 2 "https://api.testingapis.com/api/chaos/timeout?seconds=10"
curl: (28) Operation timed out after 2003 milliseconds with 0 bytes received

Exit code 28 means a timeout. In a shell script you can check for it and tell it apart from a connection refused (7) or a DNS failure (6).

Picking the number is the hard part. Too short and you fail requests that would have succeeded. Too long and one slow dependency ties up every worker in your service. A reasonable starting point is a few times the p99 latency you measure in normal conditions (the performance lesson covers percentiles). For tests, pick something short enough that a hung request fails the test quickly with a clear message.

To test how your own client or UI handles a slow response, add ?delay=N to any endpoint on this API. It waits N seconds, up to 10, before answering normally. /api/Products/1?delay=2 took about 2.5 seconds end to end when I ran it: the two-second delay plus the usual network time.

The request you gave up on may still succeed

A client timeout ends the client's wait. It doesn't necessarily stop the server. This API stops waiting as soon as the client disconnects, but plenty of servers finish the work anyway. If the request was a GET, that doesn't matter. If it was a POST that charged a card, the client now has no idea whether the charge happened. That is the problem idempotency keys solve, further down.

Retries

Many failures are temporary, and the right response is to try again. The question is which failures, how many times, and how long to wait.

The last row matters most. As the HTTP methods lesson explained, GET, PUT and DELETE are idempotent: sending them twice has the same effect as sending them once. POST and PATCH are not. Retrying a POST after a timeout can create two orders.

Testing retry logic deterministically

Retry code is hard to test against a real server because real failures are random. /api/chaos/retry/{key}sign in to see fails a fixed number of times and then succeeds. The key is any name you choose, and the server counts attempts per key:

$ curl -i "https://api.testingapis.com/api/chaos/retry/my-key?succeedAfter=3"
HTTP/1.1 503 Service Unavailable
retry-after: 1
x-attempt: 1

{"status":503,"error":"Service Unavailable","message":"Attempt 1 failed. The first 3 attempt(s) fail; attempt 4 succeeds.","attempt":1,"succeedAfter":3,"remainingFailures":2}

(attempts 2 and 3 fail the same way)

HTTP/1.1 200 OK
x-attempt: 4

{"message":"Succeeded on attempt 4.","attempt":4,"succeedAfter":3}

$ curl -X DELETE https://api.testingapis.com/api/chaos/retry/my-key

Every response has an X-Attempt header, and failures include Retry-After: 1. Browsers can't read X-Attempt because the API doesn't expose it to cross-origin scripts, but the body has the same number in attempt. The DELETE resets the counter so the next run starts from attempt 1.

Here is a small retry wrapper, run against a fresh key:

const BASE = "https://api.testingapis.com";

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

async function fetchWithRetry(url, { attempts = 5, baseDelayMs = 200, ...init } = {}) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch(url, init);
    if (!RETRYABLE.has(res.status) || attempt === attempts) return res;

    // Honour Retry-After when the server sends it, otherwise back off
    // exponentially: 200, 400, 800 ms... plus up to 100 ms of jitter.
    const retryAfter = Number(res.headers.get("retry-after"));
    const delay = retryAfter > 0
      ? retryAfter * 1000
      : baseDelayMs * 2 ** (attempt - 1) + Math.random() * 100;
    console.log(`attempt ${attempt}: ${res.status}, waiting ${Math.round(delay)} ms`);
    await new Promise((r) => setTimeout(r, delay));
  }
}

const key = `retry-demo-${Date.now()}`;
const res = await fetchWithRetry(`${BASE}/api/chaos/retry/${key}?succeedAfter=3`);
console.log(res.status, await res.json());
await fetch(`${BASE}/api/chaos/retry/${key}`, { method: "DELETE" });

attempt 1: 503, waiting 1000 ms
attempt 2: 503, waiting 1000 ms
attempt 3: 503, waiting 1000 ms
200 { message: 'Succeeded on attempt 4.', attempt: 4, succeedAfter: 3 }

Three details in that function are easy to get wrong. It gives up after a fixed number of attempts, so a server that is down for an hour doesn't get retried forever. It respects Retry-After when the server sends one, which here is why every wait is exactly one second. And when there is no Retry-After, it doubles the wait each time and adds a random amount on top. That random part is called jitter. Without it, a thousand clients that failed at the same moment all retry at the same moment, and the server that was recovering falls over again.

Random failures

/api/chaos/flakysign in to see fails with a 503 at random, half the time by default. Ten requests in a row gave me 200 200 503 200 200 200 503 503 503 503. Press Send a few times:

Use this one to see how a client behaves over many requests, not to test retry logic. A test that calls it once passes or fails by chance, which is the definition of a flaky test. Set ?failRate=1 or ?failRate=0 if you need it to be predictable.

Idempotency keys

Payment APIs have the timeout problem in its worst form: a POST that moves money, where the client can't tell whether it went through. The common fix, used by Stripe and many others, is an Idempotency-Key header. The client generates a unique value per logical operation, usually a UUID, and sends the same value on every retry of that operation. The server remembers the key and the response. If the same key arrives again, it returns the stored response instead of doing the work twice.

POST /api/Paymentssign in to see requires one. Here are four real requests. The first two send identical bodies with the same key, the third changes the amount, and the fourth leaves the header out:

curl -i -X POST https://api.testingapis.com/api/Payments \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: 1298984a-69f8-4027-acbf-da8050d8e02b" \
  -d '{"userId":101,"amount":49.99,"currency":"usd","method":"card","cardLast4":"4242","description":"Order #1001"}'

# First request
HTTP/1.1 201 Created

{"message":"Payment created successfully","idempotencyKey":"1298984a-69f8-4027-acbf-da8050d8e02b","data":{"id":13,"userId":101,"orderId":null,"amount":49.99,"currency":"USD","method":"card","status":"succeeded","description":"Order #1001","cardLast4":"4242","createdAt":"2026-09-30T18:34:06Z","refundedAt":null}}

# Same key, same body
HTTP/1.1 201 Created
idempotent-replayed: true

{"message":"Payment created successfully","idempotencyKey":"1298984a-69f8-4027-acbf-da8050d8e02b","data":{"id":13, ... same as above}}

# Same key, amount changed to 99.99
HTTP/1.1 422 Unprocessable Entity

{"status":422,"error":"Unprocessable Entity","message":"This Idempotency-Key was already used with a different request body."}

# No Idempotency-Key header
HTTP/1.1 400 Bad Request

{"status":400,"error":"Bad Request","message":"Missing Idempotency-Key header. Send a unique value (e.g. a UUID) per logical payment."}

The replay returns the same payment, id 13, with an Idempotent-Replayed: true header. No second payment was created. Reusing a key with a different body is an error, because it almost always means a client bug: two different payments sharing a key.

Test one more case in any API that uses keys: what happens when two requests with the same key arrive at the same time? The server can't replay a response it hasn't produced yet. This API holds a request open with ?processingSeconds=3, so you can send a duplicate while the first is still running:

# Two requests with the same key, the second sent one second after the first.
# ?processingSeconds=3 keeps the first one open for three seconds.

HTTP/1.1 409 Conflict
{"status":409,"error":"Conflict","message":"A request with this Idempotency-Key is still being processed. Retry shortly."}

HTTP/1.1 201 Created
{"message":"Payment created successfully","idempotencyKey":"573ab03f-8b15-49ca-bf8d-bddd8e602fe2","data":{"id":14, ...}}

The duplicate gets a 409 and the original completes. A server that got this wrong would create two payments here, and that bug only appears when a client retries quickly, which is exactly when a network is misbehaving.

The button below sends the payment with a fixed key. Everyone who reads this page shares that key, so unless you are the first since the server last restarted, you will get a replay:

{"userId":101,"amount":49.99,"currency":"usd","method":"card","cardLast4":"4242","description":"Order #1001"}

Payments can't be deleted on this API, so each new key you try adds one record. Payments reset when the server restarts.

Rate limits

Most public APIs limit how many requests a client can send in a period of time. When you go over, you get a 429 and should wait. /api/RateLimitsign in to see has a real limiter: five requests per 60 seconds per client. You identify yourself with an X-Client-Id header of your choosing.

$ curl -i -H "X-Client-Id: lesson-10868" https://api.testingapis.com/api/RateLimit
HTTP/1.1 200 OK
x-ratelimit-limit: 5
x-ratelimit-remaining: 4
x-ratelimit-reset: 1790793324

{"message":"Request allowed.","clientId":"id:lesson-10868","used":1,"remaining":4,"limit":5,"windowSeconds":60}

(requests 2 to 5 succeed, remaining counts down to 0)

HTTP/1.1 429 Too Many Requests
retry-after: 56
x-ratelimit-limit: 5
x-ratelimit-remaining: 0
x-ratelimit-reset: 1790793324

{"status":429,"error":"Too Many Requests","message":"Rate limit of 5 requests per 60s exceeded. Retry in 56s.","retryAfter":56}

The limiter sends three headers on every response: the limit, how many requests are left, and X-RateLimit-Reset, the Unix time when the window resets. The 429 adds Retry-After in seconds. Two more endpoints let you look at the window without using it up, and reset it:

$ curl -H "X-Client-Id: lesson-10868" https://api.testingapis.com/api/RateLimit/status
{"clientId":"id:lesson-10868","used":5,"remaining":0,"limit":5,"resetsInSeconds":56}

$ curl -X POST -H "X-Client-Id: lesson-10868" https://api.testingapis.com/api/RateLimit/reset
{"message":"Rate limit window reset.","clientId":"id:lesson-10868"}

Things to test on a rate-limited endpoint:

Rate limit tests should use a client id that no other test uses, and reset the window when they finish. Otherwise they interfere with each other, and the suite fails depending on which test ran first.

Two things to check on any rate limiter

Send a request without X-Client-Id and look at the client id the server picked:

$ curl https://api.testingapis.com/api/RateLimit/status
{"clientId":"ip:203.0.113.7","used":0,"remaining":5,"limit":5,"resetsInSeconds":60}

Without the header, the limiter goes by your IP address; you'll see your own address where this shows 203.0.113.7, an address reserved for documentation. When this lesson was first written, the limiter saw ::1here, the address of the proxy in front of the API, so every client without the header shared a single window of five requests per minute. That has since been fixed, but it's a common bug in real systems: a limiter keyed on the connection address works in development and turns into a global limit once a load balancer is put in front of it. The fix is to read the original address from a header the proxy sets, such as X-Forwarded-For, and to trust that header only when it comes from your own proxy. To test for it, check that two clients without the header get separate windows.

The second check: make sure the headers mean something. When this lesson was first written, every other endpoint of this API also sent X-RateLimit-* headers that never changed: 1000, 999 and a reset time in July 2024. A client that trusted them would have thought it had 999 requests left forever. They have since been removed, but the check stands for any API: make two requests and see that Remaining goes down.

Dropped connections

Some failures have no status code at all. The server closes the connection halfway, or never sends anything. Node's fetch rejects with a TypeErrorwhose message is "fetch failed". curl reports errors such as "Empty reply from server" (exit code 52) or a transfer that ended early (exit code 18).

This API has two endpoints that drop the connection on purpose. /api/chaos/abortsign in to see closes it without answering, and /api/chaos/partialsign in to see promises 10,000 bytes, sends about 70, then closes. Here is what you actually get from the public address:

$ curl -sS -o /dev/null -w "%{http_code} %{size_download}\n" https://api.testingapis.com/api/chaos/abort
503 118

$ curl -sS -o /dev/null -w "%{http_code} %{size_download}\n" https://api.testingapis.com/api/chaos/partial
curl: (18) end of response with 9936 bytes missing
200 64

The two endpoints come back differently. The dropped connection becomes a 503 with a 118-byte plain-text body, written by the proxy in front of the API, not by the API: the connection broke between the proxy and the application, and the proxy answered for it. The cut-off body gets through as it is: a 200 status, 64 of the promised 10,000 bytes, and curl exit code 18. If you ran the API locally, with no proxy in front, the first would be the raw network error described above too.

This goes beyond this API. The failure a client sees depends on everything between it and the server: load balancers, CDNs and gateways each translate errors in their own way. A client needs to handle both a network error and a 5xx whose body isn't JSON, even though the API only ever returns JSON. Code that calls res.json() on every response throws a parse error on that plain-text body, and the log then shows a JSON error instead of the 503 that caused it. Check the status and Content-Type before parsing.

Simulating errors in your own tests

When the thing under test is a client, such as a web page or a mobile app, you need the API to fail on demand. Every endpoint on this API accepts ?error= with one of 400, 401, 403, 404, 408, 429, 500, 502 or 503:

HTTP/1.1 503 Service Unavailable
Content-Type: application/json; charset=utf-8
x-simulated: true

{"status":503,"error":"Service Unavailable","message":"Simulated error. Use ?error={code} to test different status codes.","simulated":true}

Combine it with delay, as in ?delay=3&error=503, to check that a loading indicator appears and is then replaced by an error message rather than hanging. In your own projects the same idea usually takes the form of a mock server or a feature flag that makes one dependency fail. The point is the same: error handling that has never run is error handling that doesn't work.