Drift: when the spec and the API disagree

A spec and the API it describes are usually maintained separately, even when one is generated from the other, and over time they drift apart. A field gets added to the code but not the spec; the spec promises a status the code never sends; an error comes back in a format nobody documented. Each mismatch is drift. Finding it is one of the most useful things contract testing does, because every drift is either a bug in the API or a lie in the documentation, and consumers can't tell which. This lesson shows real drift found in this API while the tutorial was being written.

Drift that was fixed along the way

Some drift was so basic it had to be fixed before the tutorial could be written. Besides the missing response schemas from lesson 1, two more problems came to light:

nullable strings in the whole spec, before and after:   116 → 11
operations with a success-response schema:               1 → 20
POST /api/Products documented responses, before:         200, 201, 400, 500
POST /api/Products documented responses, after:          201, 400, 500

• Nullable everywhere. Every string in the spec was marked nullable, but the API refuses null for those fields: PUT /api/Todos/1 with "title": null gets a 400. The spec promised less than the API did, so a consumer reading it had to write null checks for values that can never be null. Now only fields that really can be null are marked.
• A response that never happens.The create endpoints documented a 200 alongside their 201, but they only ever answer 201. The framework had added a default 200 to every action that didn't declare its responses. A client generated from the spec would have handled a case that can't occur.

The rest of this lesson is about drift that is still there, recorded by tests.

An error format nobody documented

test("drift: a body that isn't valid JSON gets an error shape the spec doesn't document", async () => {
  const response = await send("POST", "/api/Products", { body: '{"title":' });
  assert.equal(response.status, 400);
  assert.deepEqual(checkResponse(spec, "POST", "/api/Products", response), [
    "POST /api/Products 400: content type application/problem+json isn't documented (documented: application/json)",
  ]);
});

The spec says a 400 from POST /api/Products is the Error object, as application/json. But a body that isn't valid JSON never reaches the code that writes that object: the framework rejects it first and answers in its own format, application/problem+json(RFC 9457's "problem details"). So this endpoint has two different 400 bodies, depending on what went wrong, and the spec describes only one. A client that reads error.message from every 400 finds nothing in the second kind.

A rule the spec doesn't mention

test("drift: the API requires a title on create, but the request schema doesn't say so", async () => {
  const body = { price: 10 };
  assert.deepEqual(checkRequest(spec, "POST", "/api/Products", body), []); // the spec sees nothing wrong

  const response = await send("POST", "/api/Products", { body });
  assert.equal(response.status, 400);
  assert.equal(response.body.message, "Missing required field: title");
});

According to the request schema, { "price": 10 }is a fine product to create: nothing is required. The API refuses it, because the code insists on a title. A developer building a form from the spec wouldn't know the title is mandatory until users started hitting 400s. This is the request-side version of drift, and checkRequest can only catch it once the spec says "required": ["title"].

Sometimes the client broke the contract

Not every surprise is the provider's fault. A contract binds both sides, and a client that sends something the spec doesn't allow can't expect the documented response:

test("a request can break the contract too: the client sent a non-integer ID", async () => {
  assert.deepEqual(checkRequest(spec, "GET", "/api/Products/abc"), [
    'GET /api/Products/{id} request: path parameter id: expected integer, got string "abc"',
  ]);

  // The server's answer is a bare 404 with no body, not the documented Error, because the request never reached
  // the action: the route only matches integers. Check your own requests before blaming the server.
  const response = await send("GET", "/api/Products/abc");
  assert.equal(response.status, 404);
  assert.equal(response.body, undefined);
});

The spec says id is an integer, and abcisn't one. The API answers with a bare 404 and no body, not the documented Error, because the request never matched the route at all. Looked at from the response side alone, that would be drift. Looked at from the request side, the client broke the contract first, and the provider is free to answer however it likes. That's why checking requests is part of contract testing: run checkRequestin a client's tests, and its mistakes show up before anyone files a bug against the server.

Which side to fix

Every drift needs a decision: change the API to match the spec, or the spec to match the API. The question to ask is who already depends on which behavior.

How to look for drift

• Check every documented operation and status you can trigger, as lesson 4 did.
• Trigger error paths on purpose: missing fields, wrong types, unreadable bodies, unknown IDs.
• Send requests that break the request schema, and see whether the API refuses them the way the spec says.
• Look for promises that are too weak to test: response schemas with nothing required, or everything nullable.
• Look for promises that never happen: documented statuses or fields no request produces.