Retries: when to send a request again

Lesson 1 showed that many failures are transient: they go away by themselves. The obvious response is to send the request again, which is called a retry. Retries are the most useful resilience technique there is, and also the easiest to get wrong. Retry the wrong failure and you waste time. Retry the wrong request and you charge a customer twice. Retry too eagerly and you can knock over the server you were waiting for. This lesson is about the decision: when a retry helps, and when it's safe.

Three questions before every retry

• Could the failure go away? If not, a retry gets the same answer and only adds delay.
• Is it safe to send the request again? If the first attempt may have changed something, a second one could change it again.
• Have we already tried enough? A failure that lasts longer than a few tries is probably not a blip.

The rest of the lesson takes these one at a time, and then puts them into code.

Could the failure go away?

This is the transient-or-permanent question from lesson 1. For statuses, the answer is fairly settled; these are the ones commonly retried:

Is it safe to send again?

HTTP gives an answer based on the method. RFC 9110, HTTP Semantics, defines two properties that matter here:

• A safe method only reads. Sending it changes nothing on the server. GET, HEAD, OPTIONS and TRACE are safe.
• An idempotent method can change things, but sending it twice leaves the server in the same state as sending it once. All safe methods are idempotent, and so are PUT and DELETE.

The word sounds technical, but the idea is everyday. Pressing a lift's call button five times brings one lift: the presses are idempotent. Paying at a till five times charges you five times. In HTTP terms:

Two details in that table trip people up. The second DELETE probably gets a 404, but idempotency is about the state of the server, not the response: either way, the todo is gone. And PATCH can go either way. "Set completed to true" is idempotent; "add 1 to the quantity" isn't.

Now combine this with lesson 1's question, whether the server acted on the request. A 429 means the server refused without acting, so even a POST can be repeated. After a 503, a timeout or a dropped connection, nobody knows. For GET, PUT and DELETE that doesn't matter, because doing it twice is harmless. For POST it matters a lot, so the default is not to retry it, unless the request carries an idempotency key, which lets the server recognize a repeat. Lesson 6 covers those.

How many times?

Every retry policy needs a maximum. If the server is truly down, retrying forever turns your client into a source of load that never stops. Three or four attempts in total is a common choice for requests someone is waiting on. Watch the wording: three attempts means the first try plus two retries. The code below counts attempts.

Retries also multiply. If each layer of a system retries on its own, one click can turn into many requests at the bottom:

browser ──► your web server ──► orders service ──► payment API
  3 tries       3 tries each        3 tries each
     1      ×       3          ×       3          =  up to 27 requests to the payment API

All of them land on the payment API at the moment it's struggling, which makes recovery harder. Lesson 1 showed that it can happen without your knowledge too: the proxy in front of this API retried a dropped request on its own. Retry in one layer, the one that knows whether repeating the request is safe, and let the others pass failures up.

The decision in code

The three questions become two functions and a loop in src/retry.mjs:

// Deciding whether to send a request again, and a loop that does it.

import { sleep } from "./api.mjs";
import { fullJitterDelay } from "./backoff.mjs";
import { classify } from "./classify.mjs";
import { retryAfterMs } from "./rate-limit.mjs";

// Statuses that often mean "try again later". 501, 505 and the 4xx family fail the same way every time.
const RETRYABLE_STATUSES = new Set([408, 429, 500, 502, 503, 504]);

// Methods RFC 9110 defines as idempotent: sending one twice has the same effect on the server as sending it once.
const IDEMPOTENT_METHODS = new Set(["GET", "HEAD", "OPTIONS", "TRACE", "PUT", "DELETE"]);

/** Is this failure the kind that might go away if we try again? */
export function isTemporary(outcome) {
  if (typeof outcome?.status === "number") return RETRYABLE_STATUSES.has(outcome.status);
  return ["timeout", "network", "truncated"].includes(classify(outcome));
}

/** Is it safe and useful to send this request again? */
export function shouldRetry(outcome, { method = "GET", idempotencyKey } = {}) {
  if (!isTemporary(outcome)) return false;

  // A 429 means the server refused the request without acting on it, so any method can be sent again.
  if (outcome?.status === 429) return true;

  // For everything else we can't be sure the server didn't act on the request. Repeat it only if repeating is
  // harmless: the method is idempotent, or an idempotency key lets the server spot the repeat.
  return IDEMPOTENT_METHODS.has(method.toUpperCase()) || Boolean(idempotencyKey);
}

/**
 * Calls send() until it succeeds, fails in a way that repeating won't fix, or reaches maxAttempts.
 * send() returns a response or throws. The last response is returned, or the last error thrown.
 */
export async function withRetries(
  send,
  { method = "GET", idempotencyKey, maxAttempts = 3, delay = fullJitterDelay, maxWaitMs = 30_000, onRetry = () => {} } = {},
) {
  for (let attempt = 1; ; attempt++) {
    let outcome;
    let threw = false;
    try {
      outcome = await send(attempt);
    } catch (error) {
      outcome = error;
      threw = true;
    }

    const retry = attempt < maxAttempts && shouldRetry(outcome, { method, idempotencyKey });
    // The server's Retry-After, when it sends one, beats our own guess.
    const wait = retry ? (retryAfterMs(outcome?.headers?.get("retry-after")) ?? delay(attempt)) : 0;

    // Waiting longer than maxWaitMs is worse than failing now: the caller has probably given up already.
    if (!retry || wait > maxWaitMs) {
      if (threw) throw outcome;
      return outcome;
    }

    onRetry({ attempt, outcome, wait });
    await sleep(wait);
  }
}

• isTemporary answers the first question. A response is judged by its status; an error by the names from lesson 1.
• shouldRetry adds the second. A 429 is always safe to repeat. Anything else is repeated only for idempotent methods, or when an idempotency key is present.
• withRetries runs send in a loop. It doesn't care how the request is made; it only looks at what sendreturned or threw. After each failure it either stops, returning the last response or throwing the last error, or waits and tries again. The wait comes from the server's Retry-After header when there is one, and from delay otherwise. Lesson 4 is about delay, and lesson 5 about Retry-After and maxWaitMs.

The whole decision table fits in one test that needs no network:

test("the retry decision", () => {
  const status = (code) => new Response(null, { status: code });
  const timeout = new DOMException("The operation was aborted due to timeout", "TimeoutError");
  const network = new TypeError("fetch failed");

  assert.equal(shouldRetry(status(503), { method: "GET" }), true); // temporary, safe to repeat
  assert.equal(shouldRetry(timeout, { method: "GET" }), true);
  assert.equal(shouldRetry(status(404), { method: "GET" }), false); // will fail the same way again
  assert.equal(shouldRetry(status(400), { method: "GET" }), false);
  assert.equal(shouldRetry(status(501), { method: "GET" }), false);

  assert.equal(shouldRetry(status(503), { method: "POST" }), false); // might have been processed already
  assert.equal(shouldRetry(network, { method: "POST" }), false);
  assert.equal(shouldRetry(status(429), { method: "POST" }), true); // refused, so not processed
  assert.equal(shouldRetry(status(503), { method: "POST", idempotencyKey: "k-1" }), true); // the key makes it safe
});

Reading it line by line is a good way to check you understand the lesson. The last three lines are the interesting ones: the same 503 is not retried for a POST, is retried once a key is added, and a 429 is retried either way.

Testing against a server that fails on demand

Random failures make poor tests: a test that passes or fails depending on luck tells you nothing. This API has an endpoint that fails a set number of times and then succeeds. It counts attempts per key, a name you choose:

$ curl -i https://api.testingapis.com/api/chaos/retry/my-test?succeedAfter=3
HTTP/1.1 503 Service Unavailable
retry-after: 1
x-attempt: 1

{"status":503,"error":"Service Unavailable","message":"Attempt 1 failed. The first 3 attempt(s) fail; attempt 4 succeeds.","attempt":1,"succeedAfter":3,"remainingFailures":2}

With succeedAfter=3, attempts 1 to 3 fail and attempt 4 succeeds. Every response says which attempt it was in X-Attempt, failures include Retry-After: 1, and DELETE /api/chaos/retry/my-testsign in to see resets the count. Here is withRetries working through it, with each attempt printed:

$ node demos/retry-timeline.mjs
    0 ms  attempt 1 sent
  203 ms  attempt 1 got 503, Retry-After: 1
 1217 ms  attempt 2 sent
 1249 ms  attempt 2 got 503, Retry-After: 1
 2260 ms  attempt 3 sent
 2299 ms  attempt 3 got 503, Retry-After: 1
 3300 ms  attempt 4 sent
 3328 ms  attempt 4 got 200, Retry-After: none
 3328 ms  done: 200 {"message":"Succeeded on attempt 4.","attempt":4,"succeedAfter":3}

Three failures, each followed by a pause of about one second because the server asked for it, then success on the fourth attempt, 3.3 seconds after the first. The test version checks the same thing with assertions:

test("two failures, then success on the third attempt", async () => {
  const url = counter("recovers");
  const retries = [];
  const response = await withRetries(() => fetch(`${url}?succeedAfter=2`, { headers: apiHeaders() }), {
    maxAttempts: 3,
    onRetry: ({ attempt, outcome, wait }) => retries.push(`attempt ${attempt}: ${outcome.status}, waiting ${wait} ms`),
  });

  assert.equal(response.status, 200);
  assert.equal(response.headers.get("x-attempt"), "3");
  assert.deepEqual(retries, ["attempt 1: 503, waiting 1000 ms", "attempt 2: 503, waiting 1000 ms"]);
});

counter() builds a key unique to this test and this run, from the process ID and the time, and remembers it so that an after hook can reset every counter when the file finishes. Without unique keys, two people running the tests at the same moment would share counters, and their attempts would mix.

And the test that a POST is not retried after a 503:

test("a POST that got 503 is not sent again", async () => {
  const url = counter("post-503");
  const response = await withRetries(() => fetch(`${url}?succeedAfter=1`, { method: "POST", headers: apiHeaders() }), {
    method: "POST",
  });
  assert.equal(response.status, 503);
  assert.equal(response.headers.get("x-attempt"), "1");
});

Retries hide problems

A client that retries well can make a struggling server look healthy: every request eventually succeeds, just slower. Log or count every retry, with the status or error that caused it. A rising retry rate is often the first sign that something is going wrong, well before anything fails outright. That's what the onRetry option is for.