Testing a client's resilience

Error handling has an awkward property: it runs rarely. On a normal day the server answers, the retry loop never loops and the circuit breaker never opens. So a bug in that code can sit unnoticed for months and show up during an outage, the worst possible moment. The only way to know it works is to make the failures happen on purpose, in tests. This is called fault injection, and this lesson puts the whole client from the tutorial under it.

Two kinds of resilience test

The tests in the example project come in two kinds, and a good suite uses both:

Fakes check that the code does what you meant. Real failures check that what you meant matches reality. Lesson 1 is the argument for the second kind: no fake would have predicted that a dropped connection comes back as a plain-text 503 from a proxy that retried it.

The failures you can inject

Every failure in this tutorial came from one of these:

The failure injection API reference lists every parameter. Prefer the deterministic ones in tests: retry/{key} fails exactly when you say, while flaky with a rate between 0 and 1 is random and belongs in demos and soak tests, not in a suite that must pass every time.

The whole client

src/client.mjs combines everything from lessons 2 to 7. It returns a request function that is used much like fetch:

// Everything from the tutorial in one client: a deadline per attempt, retries with backoff that respect
// Retry-After, idempotency keys for POST, and a circuit breaker around every attempt.

import { BASE_URL, apiHeaders } from "./api.mjs";
import { CircuitBreaker } from "./circuit-breaker.mjs";
import { withRetries } from "./retry.mjs";
import { fetchWithTimeout } from "./timeout.mjs";

export function createClient({
  baseUrl = BASE_URL,
  timeoutMs = 5000,
  maxAttempts = 3,
  breaker = new CircuitBreaker(),
  onRetry,
} = {}) {
  return async function request(path, { method = "GET", body, idempotencyKey, headers = {} } = {}) {
    const extra = {};
    if (body !== undefined) extra["Content-Type"] = "application/json";
    if (idempotencyKey) extra["Idempotency-Key"] = idempotencyKey;

    const attempt = () =>
      breaker.call(async () => {
        const response = await fetchWithTimeout(`${baseUrl}${path}`, {
          method,
          timeoutMs,
          headers: apiHeaders({ ...extra, ...headers }),
          body: body === undefined ? undefined : JSON.stringify(body),
        });
        // Read the body inside the attempt, so a body that is cut off counts as a failed attempt too.
        const text = await response.text();
        const isJson = response.headers.get("content-type")?.includes("json");
        return { status: response.status, headers: response.headers, body: isJson && text ? JSON.parse(text) : text };
      });

    return withRetries(attempt, { method, idempotencyKey, maxAttempts, onRetry });
  };
}

The order of the layers is deliberate. From the inside out:

• One attempt is fetchWithTimeout plus reading the body. Reading inside the attempt means a body that gets cut off is a failed attempt, which the retry loop can repeat, not an error that escapes later.
• The circuit breaker wraps each attempt, so it counts every failure, and once open it stops attempts before they reach the network.
• The retry loop is outermost. It uses the method and idempotency key to decide what is safe to repeat, follows Retry-After, and stops as soon as the breaker refuses, because CircuitOpenErrorisn't a temporary failure.

Testing the client

Each test in test/client.test.mjs injects one failure and checks one behavior. First, the everyday case: two failures, then success.

test("recovers from two temporary failures", async () => {
  const request = createClient({ maxAttempts: 3 });
  const response = await request(`/api/chaos/retry/${key}?succeedAfter=2`);
  assert.equal(response.status, 200);
  assert.equal(response.body.attempt, 3);
});

response.body.attempt is 3 because the client parsed the JSON and the server counted three attempts. Next, a server that hangs. The test checks that the client gives up, and how fast:

test("gives up on a server that hangs, within a known time", async () => {
  const request = createClient({ timeoutMs: 1000, maxAttempts: 2 });
  const started = Date.now();
  await assert.rejects(request("/api/chaos/timeout?seconds=10"), { name: "TimeoutError" });
  // Two attempts of 1 s each, plus at most 200 ms of backoff, plus some slack for the network.
  assert.ok(Date.now() - started < 4000);
});

Asserting on time

Two attempts of 1 second each and up to 200 ms of backoff add up to 2.2 seconds, and the test allows 4. Why not assert 2.2 exactly? Because real requests take a variable time, and so does the machine running the tests. A test that demands exact timings fails now and then for no real reason, and a test that fails at random soon gets ignored. Assert a bound instead, with room for the network: well above what a working client takes, well below what a broken one would. A client with no timeout would take 10 seconds here, so the test still catches the bug it exists for.

The body that gets cut off, which should count as a failed attempt and be retried once:

test("treats a body that is cut off as a failed attempt", async () => {
  const retries = [];
  const request = createClient({ maxAttempts: 2, onRetry: ({ outcome }) => retries.push(outcome.message) });
  await assert.rejects(request("/api/chaos/partial"), { message: "terminated" });
  assert.deepEqual(retries, ["terminated"]);
});

The breaker, which should stop the retries and then refuse new requests without touching the network:

test("stops calling a service that keeps failing", async () => {
  const breaker = new CircuitBreaker({ failureThreshold: 2, openMs: 30_000 });
  const request = createClient({ maxAttempts: 3, breaker });

  // Attempts 1 and 2 fail and open the circuit; attempt 3 is refused without being sent.
  await assert.rejects(request("/api/chaos/flaky?failRate=1"), CircuitOpenError);
  assert.equal(breaker.state, "open");

  const started = Date.now();
  await assert.rejects(request("/api/products/1"), CircuitOpenError);
  assert.ok(Date.now() - started < 50, "failed without touching the network");
});

The last assertion uses time to prove a negative: a request that finished in under 50 ms can't have gone to the server and back. And the test that a permanent failure is left alone:

test("does not retry a request the server will always refuse", async () => {
  const retries = [];
  const request = createClient({ onRetry: (r) => retries.push(r) });
  const response = await request("/api/products/9999");
  assert.equal(response.status, 404);
  assert.equal(retries.length, 0);
});

The full run:

$ npm test
✔ recovers from two temporary failures (2254.2108ms)
✔ gives up on a server that hangs, within a known time (2219.1968ms)
✔ treats a body that is cut off as a failed attempt (2512.919ms)
✔ stops calling a service that keeps failing (3243.0976ms)
✔ does not retry a request the server will always refuse (83.4976ms)
...
ℹ tests 33
ℹ pass 33
ℹ fail 0
ℹ duration_ms 30901

About 30 seconds, nearly all of it spent waiting on timeouts and Retry-After. That's normal for this kind of suite, and a reason to keep it separate from fast unit tests, as the next section explains.

Rules for resilience tests that stay reliable

• Make failures deterministic. Use counters like retry/{key}, fake clocks and fixed random numbers. A test that depends on luck will eventually fail on a day when nothing is wrong.
• Isolate shared state. Every test that uses a counter or a rate limit gets its own key or client ID, and resets it in an after hook. Two people running the suite at once should never affect each other.
• Assert bounds on time, never exact values, with room for a slow network.
• Test what must not happen. No retry of a 404, no retry of a POST without a key, no network call while the breaker is open. These are the bugs that cause duplicate charges and retry storms.
• Keep slow tests apart.Run unit tests and fakes on every change, and the real-failure suite in CI or before a release. The REST tutorial's lesson on running tests in CI covers splitting suites.
• Only inject failures where you're allowed to.Use test endpoints like these, a local mock, or your own staging environment. Hammering someone else's production service to see what happens is not testing.

What to test in any API client