Backoff and jitter: how long to wait between tries

Lesson 3 decided whether to retry. This lesson decides when. The pause between attempts is called backoff, because the client backs off and gives the server room before trying again. It sounds like a detail, but the choice of pause decides whether retries help a struggling server recover or keep it down.

Why not retry at once?

Think about what usually causes a transient failure: a server that is overloaded, restarting, or waiting on something slow. Retrying a millisecond later finds it in the same state. Worse, an overloaded server now gets your request twice. Multiply that by every client and the retries become a large part of the load that caused the problem. A pause gives the cause time to clear, and the right kind of pause spreads the retries out so they don't arrive together.

Fixed and exponential backoff

The simplest choice is a fixeddelay: wait one second before every retry. It's easy to understand, but it treats a one-second blip and a five-minute outage the same way, hitting the server at the same rate for as long as the outage lasts.

Exponential backoff doubles the delay after each failure: 200 ms, then 400, 800, 1600 and so on. The formula is the base delay times 2 to the power of (attempt minus 1). Short problems are retried quickly, and the longer a problem lasts, the less often the client asks. A cap stops the delay growing without limit, since by the tenth retry the formula would ask for nearly two minutes. In src/backoff.mjs:

// How long to wait before retry number `attempt` (1 for the first retry, 2 for the second, and so on).

/** Doubles each time: 200, 400, 800, 1600 ... up to capMs. */
export function exponentialDelay(attempt, { baseMs = 200, capMs = 10_000 } = {}) {
  return Math.min(capMs, baseMs * 2 ** (attempt - 1));
}

/**
 * "Full jitter": a random wait between 0 and the exponential delay. Clients that failed at the same moment
 * then retry at different moments instead of all together.
 */
export function fullJitterDelay(attempt, options = {}, random = Math.random) {
  return Math.round(random() * exponentialDelay(attempt, options));
}

And the sequence it produces, with a 200 ms base and a 10-second cap:

test("exponential backoff doubles the wait, up to a cap", () => {
  const waits = [1, 2, 3, 4, 5, 6, 7].map((attempt) => exponentialDelay(attempt, { baseMs: 200, capMs: 10_000 }));
  assert.deepEqual(waits, [200, 400, 800, 1600, 3200, 6400, 10_000]);
});

The thundering herd

Exponential backoff fixes the rate, but it has a flaw that only shows up with many clients. Clients rarely fail one at a time. When a server restarts, every client connected to it fails at the same moment. If they all use the same backoff, they all wait the same 200 ms and all retry at the same moment, then all wait 400 ms, and so on. The server faces a wave of requests at each step. This is called the thundering herd.

demos/thundering-herd.mjs simulates it. A thousand clients fail at time zero, and each retries three times. The script counts how many retries land in each quarter of a second, for three strategies. Nothing is sent over the network; this part is arithmetic. The heart of the script is the three ways of choosing a delay:

const strategies = {
  "fixed 1 s": () => 1000,
  exponential: (attempt) => exponentialDelay(attempt, options),
  "full jitter": (attempt) => fullJitterDelay(attempt, options, random),
};

$ node demos/thundering-herd.mjs
time          fixed 1 s  exponential  full jitter
0.00 s                0            0          588
0.25 s                0            0          673
0.50 s                0         1000          303
0.75 s                0            0          317
1.00 s             1000            0          284
1.25 s                0            0          189
1.50 s                0         1000          138
1.75 s                0            0          132
2.00 s             1000            0          122
2.25 s                0            0          103
2.50 s                0            0           79
2.75 s                0            0           52
3.00 s             1000            0           19
3.25 s                0            0            1
3.50 s                0         1000            0
3.75 s                0            0            0

Reading the columns:

• Fixed 1 s: all thousand clients retry at exactly 1, 2 and 3 seconds. The server sees nothing, then a thousand requests at once, three times.
• Exponential: the waves move to 0.5, 1.5 and 3.5 seconds and get further apart, but each one is still all thousand clients at once.
• Full jitter: the same 3,000 retries, spread out. The busiest quarter second gets 673 instead of 1,000, and from there the load tails off instead of arriving in waves.

Every column adds up to 3,000. Jitter doesn't reduce the number of retries; it changes when they arrive. A server that can handle 300 requests in a quarter of a second fails under the first two strategies and copes, after a rough start, with the third.

Full jitter

Jitter means adding randomness to the delay. With full jitter, each client waits a random time between zero and the exponential delay. On the third retry the exponential delay is 800 ms, so one client waits 112 ms, another 605, another 790. The average wait halves, and the clients no longer move in step.

That's the fullJitterDelay function above: random() * exponentialDelay(...), rounded. It's also the default delay in withRetriesfrom lesson 3. Other variants exist. "Equal jitter" keeps half the exponential delay fixed and randomizes the other half, which guarantees some minimum wait. "Decorrelated jitter" bases each delay on the previous one. An article on the AWS Architecture Blog, "Exponential Backoff And Jitter", compares them with simulations much like the one above, and is a good next read.

Testing code that uses randomness

A random function can't be tested by comparing its output with a fixed value. The solution is to make the source of randomness a parameter. fullJitterDelay takes a random function, which defaults to Math.random, and a test can pass its own:

test("full jitter picks a random wait between 0 and the exponential delay", () => {
  assert.equal(fullJitterDelay(3, {}, () => 0), 0);
  assert.equal(fullJitterDelay(3, {}, () => 0.5), 400);
  assert.equal(fullJitterDelay(3, {}, () => 0.999), 799);

  for (let i = 0; i < 1000; i++) {
    const wait = fullJitterDelay(4);
    assert.ok(wait >= 0 && wait <= 1600);
  }
});

Passing () => 0.5 makes the result predictable: half of 800 is 400. The loop at the end then checks the real random version a thousand times, to confirm it never goes outside the allowed range. The simulation uses the same trick with a seeded random generator, so it prints the same table every time. This technique, handing a function the things it depends on instead of letting it reach for them, comes back in lesson 7 with a fake clock.

How long can all this take?

Timeouts and backoff together decide the longest a caller can wait before getting an answer or an error. With a 2-second timeout, 3 attempts and full jitter on a 200 ms base, the worst case is:

If the caller can only wait 5 seconds, this policy is too generous, and the fix is fewer attempts or a shorter timeout. Work this sum out for every retry policy you write. Lesson 8 has a test that checks a client really gives up within its budget.