Disconnects and reconnecting

Every long-lived connection eventually drops. Phones switch from Wi-Fi to mobile data, laptops go to sleep, load balancers restart, servers get deployed. A WebSocket client that only works while the connection holds will fail for real users, often, and usually in a way nobody notices in a quick test on a good office network.

This happened while these lessons were being written. A ticker connection asked for 999 ticks stopped after tick 50 with close code 1006: no completion message, no reason. Runs before and after it finished normally. There was no way to tell from the client what went wrong, and that's the normal situation. Clients rarely learn why a connection died. They only learn that it did.

Noticing that the connection is gone

A clean close fires the closeevent at once. A dirty one might not. If a router between client and server loses power, or a mobile network silently drops the route, neither end is told. The client's readyState stays at OPEN, and it waits for messages that will never come. TCP only notices when one side tries to send something and gets no acknowledgement, which can take minutes. This is called a half-open connection, and it's why a client needs a way to check that the line is still alive.

Ping and pong

The protocol has control frames for this: one side sends a ping, the other must answer with a pong. Many servers ping every so often. Browsers answer pings automatically, but JavaScript can't send a ping or see one arrive. So web clients check liveness in the application instead: either the app defines its own ping message and expects a reply, or it uses messages it expects anyway. On the ticker, a tick should arrive every second, so five seconds of silence means something is wrong:

// The ticker sends a tick every second, so silence means trouble.
let lastMessageAt = Date.now();
ws.addEventListener("message", () => { lastMessageAt = Date.now(); });

const watchdog = setInterval(() => {
  if (Date.now() - lastMessageAt > 5000) {
    console.warn("no message for 5 s, assuming the connection is dead");
    clearInterval(watchdog);
    ws.close();          // then reconnect, as below
  }
}, 1000);

On the echo socket, any message you send works as a ping, because it comes straight back. Whatever the mechanism, test it by making the connection go silent without closing. Pause a proxy, block the traffic with a firewall rule, or suspend the server process. Then check how long the client takes to notice, and whether that's acceptable.

Reconnecting without making things worse

The obvious fix is to reconnect as soon as the connection closes. That works for a single client. For a hundred thousand clients whose server just restarted, it means a hundred thousand connection attempts in the same second, which can knock the server over again as it comes back up. Good clients wait longer after each failure (exponential backoff) and add randomness to the wait (jitter), so they don't all retry at once:

function connectWithRetry(url, { onMessage, maxDelayMs = 30000 } = {}) {
  let attempt = 0;
  let stopped = false;
  let ws;
  let retryTimer;

  function open() {
    if (stopped) return;
    ws = new WebSocket(url);
    ws.addEventListener("open", () => {
      attempt = 0; // a good connection resets the backoff
    });
    ws.addEventListener("message", (e) => onMessage?.(e.data));
    ws.addEventListener("close", (e) => {
      if (stopped || e.code === 1000) return; // finished on purpose: don't come back
      const base = Math.min(maxDelayMs, 1000 * 2 ** attempt);
      const delay = base / 2 + Math.random() * (base / 2); // jitter: between half and all of it
      attempt++;
      console.log(`closed ${e.code}, reconnecting in ${Math.round(delay)} ms (attempt ${attempt})`);
      retryTimer = setTimeout(open, delay);
    });
  }

  open();
  return {
    stop() {
      stopped = true;
      clearTimeout(retryTimer); // or a pending retry opens a new socket after we've stopped
      ws.close(1000);
    },
  };
}

Pointed at a path that doesn't exist, so every attempt fails, it backs off like this:

closed 1006, reconnecting in 655 ms (attempt 1)
closed 1006, reconnecting in 1106 ms (attempt 2)
closed 1006, reconnecting in 3975 ms (attempt 3)
closed 1006, reconnecting in 4762 ms (attempt 4)
closed 1006, reconnecting in 11629 ms (attempt 5)

Pointed at /ws/ticker?count=2sign in to see, it receives two ticks and the completion, sees close code 1000, and doesn't reconnect, which is right: the stream ended on purpose. There are a few decisions in those thirty lines that each deserve a test:

• Which codes trigger a reconnect. Not 1000, and usually not authentication failures, where reconnecting with the same expired token fails forever.
• When the backoff resets. This code resets as soon as a connection opens. If a server accepts and then immediately drops connections, the client retries at full speed forever. Resetting only after a connection has stayed up for a while avoids that.
• A maximum delay, and whether the client ever gives up and tells the user.
• Stopping during a wait. If the user signs out while a retry is pending, the timer must be cancelled. Leave out the clearTimeoutabove and a new connection opens after the client was told to stop, which is exactly the bug it's there to prevent.

Messages lost in the gap

Reconnecting restores the connection. It doesn't restore what was sent while the client was away. Anything the server pushed during the gap is gone, unless the application does something about it. The usual approaches are to have the client say where it got to when it reconnects ("send me everything after message 1,234"), or to fetch the current state over HTTP and then continue with live updates.

The ticker does neither. Every connection starts again at tick 1, as the previous lesson showed. A client that reconnects after tick 20 receives tick 1 again. With sequence numbers it can at least tell what happened:

let lastTick = 0;

function onTick(data) {
  if (data.tick === lastTick + 1) {
    lastTick = data.tick;
    render(data);
  } else if (data.tick <= lastTick) {
    // Seen it already, or the stream restarted after a reconnect
    console.warn(`tick ${data.tick} after ${lastTick}: duplicate or restart`);
  } else {
    console.warn(`missed ticks ${lastTick + 1} to ${data.tick - 1}`);
    lastTick = data.tick;
    render(data);
  }
}

Gaps, duplicates and restarts are the three cases to test, and they all need a disconnect in the middle of a stream. For each, decide with the team what the user should see. Showing an old price as if it were new is a worse bug than showing nothing.

How to cause a disconnect

• Turn off Wi-Fi, or switch the phone to aeroplane mode for ten seconds, then back.
• Put the laptop to sleep with the app open, and wake it a few minutes later.
• Restart or redeploy the server while clients are connected.
• Put a proxy such as Toxiproxy between client and server in a test environment. It can cut the connection, add delay, or stop passing data without closing, all on command from a test.
• Close the connection from the server with each close code the client handles differently.

After each one, check:

• The app shows that it's disconnected, and that it's trying again.
• It reconnects on its own, within a reasonable time, once the network is back.
• It resubscribes to everything it was subscribed to before. A client that reconnects but forgets to resubscribe looks connected and receives nothing.
• Each message is handled once. Adding message listeners again on every reconnect, without removing the old ones, makes every message appear twice after the first reconnect, three times after the second, and so on. It's one of the most common WebSocket client bugs.
• Missed data is recovered, or the user is told it might be stale.