Automating WebSocket tests
Turning events into something a test can wait for, with timeouts that fail clearly.
Testing a WebSocket by hand gets you a long way, but it doesn't scale and it doesn't run in CI. The difficulty in automating it is that the WebSocket API is built on events, and test code wants to say "send this, then wait for that". Most flaky WebSocket tests come from bridging the two badly, usually with a sleep(2000) and the hope that the message has arrived by then. This lesson builds the bridge properly. The REST equivalent is automating tests with code.
A helper that waits for messages
The examples use Node.js 22 or later, which has both the test runner and WebSocketbuilt in. Here is the helper. It's the longest part of the file, and you write it once:
// websocket.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";
import { request } from "node:https";
const BASE = process.env.WS_BASE ?? "wss://api.testingapis.com";
const KEY = process.env.TESTINGAPIS_KEY;
// Browsers can't set headers on a WebSocket, so this API also takes the key in the query string.
const url = (path) => (KEY ? `${BASE}${path}${path.includes("?") ? "&" : "?"}ta_key=${KEY}` : `${BASE}${path}`);
/**
* Opens a socket and turns its events into things a test can await:
* next() resolves with the next message, closedWithin() with the close event.
* Both fail after a timeout instead of hanging the test run.
*/
function connect(path, { timeoutMs = 5000 } = {}) {
const ws = new WebSocket(url(path));
ws.binaryType = "arraybuffer";
const queue = [];
const waiting = [];
ws.addEventListener("message", (e) => {
const next = waiting.shift();
if (next) next.resolve(e.data);
else queue.push(e.data);
});
const closed = new Promise((resolve) => ws.addEventListener("close", resolve));
const opened = new Promise((resolve, reject) => {
ws.addEventListener("open", resolve);
ws.addEventListener("close", (e) => reject(new Error(`closed before opening: ${e.code}`)));
});
function next() {
if (queue.length) return Promise.resolve(queue.shift());
return new Promise((resolve, reject) => {
const waiter = { resolve: (d) => { clearTimeout(timer); resolve(d); } };
const timer = setTimeout(() => {
waiting.splice(waiting.indexOf(waiter), 1);
reject(new Error(`no message within ${timeoutMs} ms`));
}, timeoutMs);
waiting.push(waiter);
});
}
function closedWithin(ms = timeoutMs) {
return Promise.race([
closed,
new Promise((_, reject) => setTimeout(() => reject(new Error(`not closed within ${ms} ms`)), ms)),
]);
}
return { ws, opened, next, closedWithin };
}Messages go into a queue as they arrive. next() takes the oldest one, or waits for the next to arrive, and fails with a clear error if nothing comes within the timeout. Because nothing is dropped, a test never misses a message that arrived before it started waiting, which is the race condition that breaks the simpler ws.onmessage = ... approach. closedWithin() does the same for the close event. Every wait has a timeout, so a broken server fails the test in seconds instead of hanging the pipeline until CI kills it.
The tests
Messages
test("echo greets with a welcome message", async () => {
const s = connect("/ws/echo");
await s.opened;
const welcome = JSON.parse(await s.next());
assert.equal(welcome.type, "welcome");
assert.match(welcome.message, /echo/i);
s.ws.close();
});
test("echo returns text, JSON and binary unchanged", async () => {
const s = connect("/ws/echo");
await s.opened;
await s.next(); // welcome
s.ws.send("héllo 👋");
assert.equal(await s.next(), "héllo 👋");
const json = JSON.stringify({ id: 1, tags: ["a", "b"] });
s.ws.send(json);
assert.equal(await s.next(), json);
s.ws.send(new Uint8Array([0, 1, 254, 255]));
const binary = await s.next();
assert.ok(binary instanceof ArrayBuffer, "binary should come back as binary");
assert.deepEqual([...new Uint8Array(binary)], [0, 1, 254, 255]);
s.ws.close();
});Each test reads the welcome message first, even when it doesn't check it. Otherwise the welcome would be the first thing next() returns, and the test would compare the echo of héllo against the greeting. Knowing exactly which messages a socket sends unprompted is part of writing tests for it.
Closing and streams
test("sending close ends the connection with 1000 Bye!", async () => {
const s = connect("/ws/echo");
await s.opened;
await s.next();
s.ws.send("close");
const e = await s.closedWithin(3000);
assert.equal(e.code, 1000);
assert.equal(e.reason, "Bye!");
assert.equal(e.wasClean, true);
});
test("a message one byte over 64KB is refused with 1009", async () => {
const s = connect("/ws/echo");
await s.opened;
await s.next();
s.ws.send("a".repeat(65537));
const e = await s.closedWithin(3000);
assert.equal(e.code, 1009);
});
test("ticker sends count ticks in order, then completes and closes", async () => {
const s = connect("/ws/ticker?count=3");
await s.opened;
const ticks = [];
for (let i = 0; i < 3; i++) ticks.push(JSON.parse(await s.next()));
assert.deepEqual(ticks.map((t) => t.type), ["tick", "tick", "tick"]);
assert.deepEqual(ticks.map((t) => t.data.tick), [1, 2, 3]);
assert.deepEqual(JSON.parse(await s.next()), { type: "complete", ticks: 3 });
const e = await s.closedWithin(3000);
assert.equal(e.code, 1000);
assert.equal(e.reason, "Ticker finished");
});These check the end of the connection as carefully as the messages: the code, the reason, and wasClean. The close timeout is 3 seconds because closing on this API takes about one, as the closing lesson measured. The ticker test checks the sequence numbers, the completion message, and the close, in that order, without any sleeps.
The handshake
// The browser API only reports 1006 for a failed handshake, so check the HTTP answer directly.
test("ticker rejects a count that isn't a number at the handshake", async () => {
const target = new URL(url("/ws/ticker?count=abc").replace(/^ws/, "http"));
const res = await new Promise((resolve, reject) => {
const req = request(target, {
headers: {
Connection: "Upgrade",
Upgrade: "websocket",
"Sec-WebSocket-Version": "13",
"Sec-WebSocket-Key": "dGhlIHNhbXBsZSBub25jZQ==",
},
});
req.on("response", resolve);
req.on("upgrade", () => reject(new Error("the handshake should have failed")));
req.on("error", reject);
req.end();
});
assert.equal(res.statusCode, 400);
let body = "";
for await (const chunk of res) body += chunk;
assert.deepEqual(JSON.parse(body).errors.count, ["The value 'abc' is not valid."]);
});A failed handshake looks like any other failure through the WebSocket API: close code 1006 and nothing else. To test that the server rejects bad input with the right status, this test sends the handshake as a plain HTTPS request with node:https and reads the 400 and its JSON body.
Running them
$ node --test
✔ echo greets with a welcome message (430.9779ms)
✔ echo returns text, JSON and binary unchanged (458.9761ms)
✔ sending close ends the connection with 1000 Bye! (1146.7238ms)
✔ a message one byte over 64KB is refused with 1009 (1339.6868ms)
✔ ticker sends count ticks in order, then completes and closes (3490.0908ms)
✔ ticker rejects a count that isn't a number at the handshake (148.2677ms)
ℹ tests 6
ℹ pass 6
ℹ fail 0Nine seconds, most of it the ticker's three ticks and the one-second closes. If a test fails, the message should tell you what happened without opening the code. Change the expected reason to Goodbye! and you get:
✖ sending close ends the connection with 1000 Bye! (1397.1208ms)
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
'Bye!' !== 'Goodbye!'And set the ticker test's timeout to 500 milliseconds, shorter than the gap between ticks:
✖ ticker sends count ticks in order, then completes and closes (734.1371ms)
Error: no message within 500 msThe first tick arrives straight away, so the test passes that point. The second is a second later, so next()gives up. Timeouts have to fit the protocol's rhythm. Set them from what the API promises, then add a margin for slow CI machines, rather than from what happened to work on your laptop.
The same tests in Python
In Python, the websockets library already lets you await the next message, so no queue is needed. With pytest and the pytest-asyncio plugin:
# test_websocket.py
import asyncio
import json
import os
import pytest
from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed, InvalidStatus
BASE = os.environ.get("WS_BASE", "wss://api.testingapis.com")
KEY = os.environ.get("TESTINGAPIS_KEY")
def url(path):
if not KEY:
return BASE + path
return f"{BASE}{path}{'&' if '?' in path else '?'}ta_key={KEY}"
async def next_message(ws, timeout=5):
return await asyncio.wait_for(ws.recv(), timeout)
@pytest.mark.asyncio
async def test_echo_returns_text_and_binary_unchanged():
async with connect(url("/ws/echo")) as ws:
welcome = json.loads(await next_message(ws))
assert welcome["type"] == "welcome"
await ws.send("héllo 👋")
assert await next_message(ws) == "héllo 👋"
await ws.send(bytes([0, 1, 254, 255]))
assert await next_message(ws) == bytes([0, 1, 254, 255])
@pytest.mark.asyncio
async def test_sending_close_ends_with_1000_bye():
async with connect(url("/ws/echo")) as ws:
await next_message(ws)
await ws.send("close")
with pytest.raises(ConnectionClosed) as closed:
await next_message(ws)
assert closed.value.rcvd.code == 1000
assert closed.value.rcvd.reason == "Bye!"
@pytest.mark.asyncio
async def test_ticker_sends_ticks_in_order_then_completes():
async with connect(url("/ws/ticker?count=3")) as ws:
ticks = [json.loads(await next_message(ws)) for _ in range(3)]
assert [t["data"]["tick"] for t in ticks] == [1, 2, 3]
assert json.loads(await next_message(ws)) == {"type": "complete", "ticks": 3}
@pytest.mark.asyncio
async def test_ticker_rejects_a_bad_count_at_the_handshake():
# Unlike the browser API, this library reports the HTTP status of a failed handshake.
with pytest.raises(InvalidStatus) as failed:
async with connect(url("/ws/ticker?count=abc")):
pass
assert failed.value.response.status_code == 400$ pip install websockets pytest pytest-asyncio
$ pytest -q test_websocket.py
.... [100%]
4 passed in 6.34sOne difference matters. When a handshake fails, the Python library raises InvalidStatuswith the server's response, so the status code is available directly. Only browsers hide it. Non-browser clients, including the ones in mobile apps and backend services, often see the real status, and their error handling can be tested more precisely.
What to automate
- Every message the server sends without being asked, such as greetings, and their exact shape.
- Each message type the client sends, with a valid example, an invalid one, and an unknown type.
- Every way the connection can end, with its code and reason.
- The size limit, exactly at it and one byte over.
- For streams, sequence, completeness and the final message, as in the ticker test.
- Handshake failures, tested at the HTTP level where the status is visible.
Exercise
Add a test that checks the empty message is echoed as an empty string, and another that sends 500 numbered messages as fast as possible and checks they all come back in order. Then write a test for the client's close: call close(4000) and assert on the code the close event reports. Should that test assert 4000 or 1000? Write a comment in the test explaining why.