Circuit breakers: stop calling a broken service

Timeouts and retries handle failures that pass in a second or two. Some don't: a database fails over, a region goes down, a deploy goes wrong, and a service is unavailable for minutes. During that time, a client with retries does exactly the wrong thing. Every request waits through every timeout and every retry, and then fails anyway. A circuit breaker notices that a service is failing and stops calling it for a while, so requests fail at once instead of slowly, and the service gets a break while it recovers.

What failing slowly costs

Take the retry policy from lesson 4: a 2-second timeout and 3 attempts, which can take 6.6 seconds in the worst case. Now suppose the service it calls stops answering, and your application handles 100 requests a second that need it:

• Every user waits up to 6.6 seconds, and then gets an error anyway.
• The dead service receives up to 300 requests a second, all of them retries of requests that won't succeed.
• About 660 requests are waiting at any moment, each holding memory and a connection in your application, which may now run out and start failing requests that never needed the broken service.

None of that work achieves anything. Once it's clear the service is down, the useful response is to stop trying for a while, fail fast, and check back now and then. That's what a circuit breaker automates.

Three states

The name comes from the electrical device in a building's fuse box, which cuts the power when it detects a fault so the damage doesn't spread. In software, a breaker sits in front of calls to one service and is always in one of three states:

failures in a row reach the threshold
   ┌────────┐ ─────────────────────────────────► ┌────────┐
   │ CLOSED │                                    │  OPEN  │  calls fail at once,
   └────────┘ ◄───────────┐                      └────────┘  nothing is sent
   calls go through,      │                           │
   failures are counted   │ trial succeeds            │ the open period ends
                          │                           ▼
                          │   trial fails      ┌───────────┐
                          └─────────────────── │ HALF-OPEN │  one trial call
                                 (back to OPEN)└───────────┘  goes through

"Closed" and "open" can feel backwards at first. Think of the electrical circuit: when it's closed, current flows; when it's open, the connection is broken and nothing gets through.

Watching one work

demos/breaker-timeline.mjs simulates a service that is down for 4 seconds and then recovers, using /api/chaos/flakysign in to see with failRate=1 for the first 4 seconds and failRate=0 after. A request goes through a breaker every 400 ms. The breaker opens after 3 failures in a row and stays open for 2 seconds:

$ node demos/breaker-timeline.mjs
    0 ms  request  1  HTTP 503                     breaker: closed
 1255 ms  request  2  HTTP 503                     breaker: closed
 1738 ms  request  3  HTTP 503                     breaker: open
 2174 ms  request  4  CircuitOpenError, not sent   breaker: open
 2579 ms  request  5  CircuitOpenError, not sent   breaker: open
 2983 ms  request  6  CircuitOpenError, not sent   breaker: open
 3384 ms  request  7  CircuitOpenError, not sent   breaker: open
 3785 ms  request  8  HTTP 503                     breaker: open
 4224 ms  request  9  CircuitOpenError, not sent   breaker: open
 4629 ms  request 10  CircuitOpenError, not sent   breaker: open
 5035 ms  request 11  CircuitOpenError, not sent   breaker: open
 5438 ms  request 12  CircuitOpenError, not sent   breaker: open
 5841 ms  request 13  HTTP 200                     breaker: closed
 6272 ms  request 14  HTTP 200                     breaker: closed
 6711 ms  request 15  HTTP 200                     breaker: closed
 7154 ms  request 16  HTTP 200                     breaker: closed

• Requests 1 to 3: the service fails each one. After the third failure in a row, the breaker opens.
• Requests 4 to 7: the breaker refuses them immediately. Nothing is sent, so the service gets no traffic, and the caller learns of the problem in under a millisecond instead of waiting.
• Request 8: 2 seconds have passed since the breaker opened, so this one goes through as a trial. The service is still down, so the trial fails and the breaker opens again for another 2 seconds.
• Requests 9 to 12:refused again. The service had in fact recovered at the 4-second mark, after request 8, but the breaker can only find out by sending a trial, and the next one isn't due until 2 seconds after request 8.
• Request 13: the next trial. It succeeds, the breaker closes, and normal traffic resumes.

While the service was down, during the first 4 seconds, 8 requests were made and the service received 4 of them: three that showed it was failing, and one trial. In a real system the saving is much bigger, because the open period is usually tens of seconds and the traffic much heavier. Requests 9 to 12 show the price: for up to one open period after the service recovers, the breaker keeps refusing calls that would have worked. A shorter open period notices recovery sooner but sends more trials to a service that may still be struggling.

Writing one

// A circuit breaker: after enough failures in a row, stop calling the service for a while and fail at once
// instead. Then let a single trial request through to see whether it has recovered.
//
//   closed     normal: every call goes through; failures in a row are counted
//   open       every call fails at once with CircuitOpenError, without touching the network
//   half-open  the wait is over: one trial call goes through; success closes the circuit, failure opens it again

export class CircuitOpenError extends Error {
  constructor(retryInMs) {
    super(`Circuit open: not calling the service for another ${retryInMs} ms`);
    this.name = "CircuitOpenError";
    this.retryInMs = retryInMs;
  }
}

export class CircuitBreaker {
  state = "closed";
  #failures = 0;
  #openedAt = 0;
  #trialInFlight = false;

  constructor({ failureThreshold = 5, openMs = 30_000, isFailure = (result) => result.status >= 500, now = Date.now } = {}) {
    this.failureThreshold = failureThreshold;
    this.openMs = openMs;
    this.isFailure = isFailure;
    this.now = now;
  }

  async call(fn) {
    if (this.state === "open") {
      const waited = this.now() - this.#openedAt;
      if (waited < this.openMs) throw new CircuitOpenError(this.openMs - waited);
      this.state = "half-open";
    }

    const isTrial = this.state === "half-open";
    if (isTrial) {
      if (this.#trialInFlight) throw new CircuitOpenError(0); // one trial at a time
      this.#trialInFlight = true;
    }

    try {
      const result = await fn();
      if (this.isFailure(result)) this.#failure();
      else this.#success();
      return result;
    } catch (error) {
      this.#failure();
      throw error;
    } finally {
      if (isTrial) this.#trialInFlight = false;
    }
  }

  #success() {
    this.#failures = 0;
    this.state = "closed";
  }

  #failure() {
    this.#failures++;
    if (this.state === "half-open" || this.#failures >= this.failureThreshold) {
      this.state = "open";
      this.#openedAt = this.now();
    }
  }
}

Going through call(fn), which wraps any function that makes a request:

• If open: it checks how long ago the breaker opened. Inside the open period it throws CircuitOpenError without calling fn. After it, it switches to half-open and carries on.
• If half-open:this call is the trial. A flag makes sure only one trial runs at a time; any call that arrives while the trial is in flight is refused, so a recovering service isn't hit by everyone at once.
• The call itself: it runs fn. A thrown error, such as a timeout or a network error, is a failure. A returned response is a failure if isFailure says so: by default, any status of 500 or above.
• Counting: a success resets the count and closes the breaker. A failure adds one, and opens the breaker when the count reaches the threshold, or at once if it was the trial.

now is a parameter for the same reason random was in lesson 4: tests can pass a fake clock and move time forward without waiting.

What counts as a failure

A breaker should open when the serviceis unhealthy, not when a caller makes a mistake. So the default counts 5xx responses, timeouts and network errors, and ignores 4xx. A burst of 404s for products that don't exist says nothing about the service's health, and opening the breaker over them would cut everyone else off. A 429 isn't counted either: the service is working, it's just limiting you, and lesson 5's handling of Retry-After is the right response.

Choosing the numbers

• Threshold: too low and a couple of unlucky failures cut off a healthy service; too high and requests keep piling up after the service has clearly gone. Five failures in a row is a common start.
• Open period: long enough for a typical outage to clear, often 10 to 60 seconds. Too short and the trials themselves become steady load on a struggling service.
• Consecutive failures or a failure rate: this breaker counts failures in a row, which is easy to follow. Production libraries usually look at the share of failures over a recent window, with a minimum number of calls, which copes better with heavy traffic where a few successes slip between many failures.

What to do while the breaker is open

A breaker turns a slow failure into a fast one. What the user sees depends on what you do with the CircuitOpenError. Some options, from most to least helpful:

• Serve the last good response from a cache, marked as possibly out of date. Product details and prices often allow this.
• Use a sensible default: no recommendations rather than a broken recommendations panel.
• Turn off the feature that needs the service and keep the rest of the page working.
• Accept the request and queue it, to be sent when the service is back. Orders and emails often work this way.
• Show a clear message: "Payments are temporarily unavailable. Please try again in a minute."

Each of these is a fallback. Which one fits is a product decision more than a technical one, so decide it with whoever owns the feature.

Where breakers go

• One breaker per service you call.If the recommendations service is down, there's no reason to stop calling the payments service. A shared breaker would do exactly that.
• Around each attempt, inside the retry loop. Then every failed attempt is counted, and once the breaker opens, the next attempt throws CircuitOpenError. That error isn't temporary by lesson 3's rules, so the retry loop stops at once instead of waiting through its remaining attempts. Lesson 8's client is built this way.
• Per process.Each running copy of your application has its own breaker and opens it from its own failures. That's usually fine: every copy notices the outage within a few requests.

Testing a breaker

Testing state changes that depend on time would be slow if the test had to wait. With a fake clock, the test holds the time in a variable and moves it forward by hand:

test("closed, open, half-open, closed", async () => {
  let now = 0;
  const breaker = new CircuitBreaker({ failureThreshold: 3, openMs: 10_000, now: () => now });

  for (let i = 0; i < 3; i++) await breaker.call(failing);
  assert.equal(breaker.state, "open");

  // While open, calls fail at once and the function isn't called.
  let called = false;
  await assert.rejects(breaker.call(async () => (called = true)), CircuitOpenError);
  assert.equal(called, false);

  now += 10_000; // the wait is over: the next call is a trial
  await breaker.call(healthy);
  assert.equal(breaker.state, "closed");
});

now += 10_000moves time forward by the whole open period in no time at all. The test also checks that while the breaker is open, the wrapped function isn't called, which is the point of the whole pattern. The second test checks that a failed trial opens the breaker again:

test("a failed trial opens the circuit again", async () => {
  let now = 0;
  const breaker = new CircuitBreaker({ failureThreshold: 1, openMs: 5000, now: () => now });

  await breaker.call(failing);
  now += 5000;
  await breaker.call(failing); // the trial
  assert.equal(breaker.state, "open");
  await assert.rejects(breaker.call(healthy), CircuitOpenError);
});

And a live test counts how many requests reach a service that always fails:

test("against a service that always fails, only the first calls reach it", async () => {
  const breaker = new CircuitBreaker({ failureThreshold: 3, openMs: 30_000 });
  let sent = 0;
  let rejected = 0;

  for (let i = 0; i < 10; i++) {
    try {
      await breaker.call(() => {
        sent++;
        return fetch(`${BASE_URL}/api/chaos/flaky?failRate=1`, { headers: apiHeaders() });
      });
    } catch (error) {
      if (!(error instanceof CircuitOpenError)) throw error;
      rejected++;
    }
  }

  assert.equal(sent, 3);
  assert.equal(rejected, 7);
});