Schemas and contract testing

Most tests written in the automation lesson look something like this:

test("GET /api/Products/1 returns the headphones", async () => {
  const res = await fetch(`${BASE}/api/Products/1`);
  assert.equal(res.status, 200);
  const body = await res.json();
  assert.equal(body.title, "Wireless Noise-Cancelling Headphones");
});

It checks one field. It keeps passing if a developer renames inStock to in_stock, drops rating, or starts sending the price as the string "249.99" instead of the number 249.99. Every one of those changes breaks some client. The string price is a good example of how quietly it can happen. In JavaScript, "249.99" * 2 is still 499.98, so the cart total looks fine, but "249.99" + 5 is "249.995", so the total with shipping is wrong. A Swift or Kotlin app that decodes the response into a typed model fails to decode it at all and shows an empty screen.

You could add an assertion for every field on every endpoint, but nobody keeps that up for long. The usual answer is to describe the shape of a response once, as a schema, and check every response against it.

JSON Schema

JSON Schema is a JSON document that describes other JSON documents: which fields exist, their types, which ones are required, and limits on their values. Here is a schema for a product, written by looking at what GET /api/Products/1 actually returns and at the categories from GET /api/Products/categories:

// product-schema.mjs
export const productSchema = {
  type: "object",
  required: ["id", "title", "price", "description", "category", "image", "rating", "inStock", "createdAt"],
  additionalProperties: false,
  properties: {
    id: { type: "integer", minimum: 1 },
    title: { type: "string", minLength: 1 },
    price: { type: "number", exclusiveMinimum: 0 },
    description: { type: "string" },
    category: { enum: ["electronics", "clothing", "books", "home", "sports"] },
    image: { type: "string", format: "uri" },
    rating: {
      type: "object",
      required: ["rate", "count"],
      additionalProperties: false,
      properties: {
        rate: { type: "number", minimum: 0, maximum: 5 },
        count: { type: "integer", minimum: 0 },
      },
    },
    inStock: { type: "boolean" },
    createdAt: { type: "string", format: "date-time" },
  },
};

A few of these choices are deliberate. required lists every field, because a client that reads product.rating.rate crashes if rating is missing. additionalProperties: false makes unknown fields an error, which is stricter than most people want; more on that below. The category enum means a new category fails the test, and that is intentional too: the product filter in the app has a fixed list of categories, and someone should decide what happens to a sixth one.

Validating in a test

Ajv is the most widely used JSON Schema validator for JavaScript. ajv-formats adds checks for formats such as date-time and uri, which Ajv ignores otherwise.

npm install --save-dev ajv ajv-formats

// products.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";
import Ajv from "ajv";
import addFormats from "ajv-formats";
import { productSchema } from "./product-schema.mjs";

const BASE = "https://api.testingapis.com";
const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validateProduct = ajv.compile(productSchema);

test("GET /api/Products/1 matches the product schema", async () => {
  const res = await fetch(`${BASE}/api/Products/1`);
  assert.equal(res.status, 200);
  const body = await res.json();
  assert.ok(validateProduct(body), ajv.errorsText(validateProduct.errors));
});

test("every product in the list matches the schema", async () => {
  const res = await fetch(`${BASE}/api/Products`);
  const products = await res.json();
  assert.ok(products.length > 0);
  for (const p of products) {
    assert.ok(validateProduct(p), `product ${p.id}: ${ajv.errorsText(validateProduct.errors)}`);
  }
});

$ node --test products.test.mjs
✔ GET /api/Products/1 matches the product schema (506.1345ms)
✔ every product in the list matches the schema (314.0142ms)
ℹ tests 2
ℹ pass 2
ℹ fail 0

The second test is the more useful one. A single product can look fine while product 17 has a null description because someone created it through a form that allowed one. Checking the whole list catches that. Note that the list includes records other people created on this API, so it may fail for reasons that have nothing to do with you. That is also realistic.

To see what a failure looks like, break a real product on purpose: turn the price into a string, add a field and remove another.

// Same Ajv setup as above, with validate = ajv.compile(productSchema)
const product = await (await fetch("https://api.testingapis.com/api/Products/1")).json();
const broken = { ...product, price: "249.99", discount: 10 };
delete broken.inStock;

validate(broken);
console.log(ajv.errorsText(validate.errors, { separator: "\n" }));

data must have required property 'inStock'
data must NOT have additional properties
data/price must be number

Three problems, reported at once because of allErrors: true. Without that option Ajv stops at the first error, and you fix them one test run at a time.

How strict to be

Adding a field to a response is normally safe. Clients that don't know about the field ignore it. So additionalProperties: false makes your test fail on a change that breaks nobody. Teams split on this. My preference: be strict in the test suite that belongs to the API itself, so every new field is a conscious decision and someone updates the schema and the docs. Be loose in tests that a consuming team writes against someone else's API, because there you only care about the fields you use, and you don't want your build to break every time the other team ships something.

What a schema won't catch

A schema checks shape, not truth. Take the real product and give it another product's title and a price that is a hundred times too low:

const product = await (await fetch("https://api.testingapis.com/api/Products/1")).json();
console.log(validate({ ...product, price: 2.49, title: "Smartwatch Series 6" }));
// true

Valid. The schema has no way to know that product 1 costs 249.99 or that it is headphones. The same goes for rules that link fields together, such as "refundedAt is set only when status is refunded". JSON Schema can express some of these with if and then, but the result is hard to read, and a plain assertion in the test is usually clearer. Schema validation replaces the long list of type checks. It doesn't replace the handful of value checks that tell you the API did the right thing.

OpenAPI as the contract

Writing schemas by hand works for a few endpoints. For a whole API, the schemas usually come from an OpenAPI document: a single file that lists every path, its parameters, request bodies and responses, with schemas for each. This API publishes one at /openapi/v1.jsonsign in to see. It describes about 290 paths. Here is the entry for getting a product, trimmed to the get operation:

"/api/Products/{id}": {
  "get": {
    "tags": ["Products"],
    "summary": "Get a product by ID.",
    "operationId": "Products_GetById",
    "parameters": [
      {
        "name": "id",
        "in": "path",
        "description": "Product ID.",
        "required": true,
        "schema": { "type": "integer", "format": "int32" }
      }
    ],
    "responses": {
      "200": {
        "description": "The product.",
        "content": { "application/json": { "schema": { "$ref": "#/components/schemas/ProductResponse" } } }
      },
      "404": {
        "description": "The product does not exist.",
        "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" } } }
      },
      "500": { "description": "Unexpected server error." }
    }
  }
}

The 200 response points at a schema, ProductResponse, and the 404 at a standard Errorschema. That wasn't always so. When this lesson was first written, the 200had a description and nothing else: 1 of the 428 success responses in the document declared a body schema, so the spec said which endpoints existed but couldn't be used to check what they returned. The products, posts and todos endpoints have since been fixed. Here is the response schema they point at:

"ProductResponse": {
  "required": ["id", "title", "price", "description", "category", "image", "rating", "inStock", "createdAt"],
  "type": "object",
  "properties": {
    "id": { "type": "integer", "format": "int32" },
    "title": { "type": "string" },
    "price": { "type": "number", "format": "double" },
    "description": { "type": "string" },
    "category": { "type": "string" },
    "image": { "type": "string" },
    "rating": { "$ref": "#/components/schemas/RatingResponse" },
    "inStock": { "type": "boolean" },
    "createdAt": { "type": "string" }
  },
  "additionalProperties": false,
  "description": "Product as the API returns it: every property is always present."
}

Every field is in required, none is nullable, and additionalProperties: false forbids extras, so tests can rely on it. You can load the whole spec into Ajv and point at the component by its path:

import Ajv from "ajv";

const BASE = "https://api.testingapis.com";
const spec = await (await fetch(`${BASE}/openapi/v1.json`)).json();

// The spec uses OpenAPI formats such as int32 and double, which Ajv
// doesn't know. Register them as "anything goes" so it doesn't refuse.
const ajv = new Ajv({ allErrors: true, strict: false });
for (const f of ["int32", "int64", "double", "float"]) ajv.addFormat(f, true);
ajv.addSchema(spec, "spec");

const validate = ajv.getSchema("spec#/components/schemas/ProductResponse");
const product = await (await fetch(`${BASE}/api/Products/1`)).json();

console.log("real product:", validate(product));
console.log("empty object:", validate({}));
console.log("extra field:", validate({ ...product, discount: 10 }));
console.log("title: null:", validate({ ...product, title: null }));

real product: true
empty object: false
extra field: false
title: null: false

The real product passes; an empty object, an extra field and a null title all fail. Before the fix, the only product schema had no required list and every string nullable, and the empty object and the null title both passed.

That earlier state is typical of a spec generated from code. The framework can see that a controller method returns "some result", but not what is inside it, and it marks reference types as nullable unless told otherwise. When a team says the OpenAPI file is the contract, test the file itself: every operation should have a response schema, and schemas should list their required fields. Tools such as Schemathesis go further and generate requests from the spec, then check the responses against it, but they can only be as strict as the document they read. The contract testing tutorial covers all of this in depth, including how these gaps were found.

Consumer-driven contract testing

Schemas describe what the provider says it returns. Contract testing flips that around and records what each consumer actually relies on. Pact is the tool most people mean when they say contract testing.

It works in two halves. The team that builds the consumer (say, the mobile app) writes tests against a mock server provided by Pact. Each test states a request and the parts of the response the app needs: "GET /api/Products/1 returns 200 with a numeric price and a string title". Running those tests produces a contract file. The provider team then runs that file against the real API in its own pipeline. Pact replays each request and checks the response still satisfies what the consumer asked for. If a provider change would break the app, the provider's build fails before anything is deployed.

This pays off when several teams own services that call each other and deploy on their own schedules, which is where the question "can I ship this without breaking someone" is hard to answer. It is a lot of machinery otherwise: a broker to store contracts, provider states to set up test data, and agreement between teams on who fixes what. If one team owns the API and its only client, schema tests in the API's own suite plus a few end-to-end tests cover the same risk for much less work.

Breaking and non-breaking changes

Whichever tool you use, the point is to notice breaking changes before clients do. Roughly:

APIs that need to make breaking changes publish a new version and keep the old one running for a while. This API has an example of that.

Versioning on this API

The profile endpoint exists in two versions, selected by the path:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
deprecation: @1751328000
link: </api/v2/profile>; rel="successor-version"
sunset: Thu, 31 Dec 2026 23:59:59 GMT
x-api-version: 1

{
  "id": 101,
  "name": "Ava Thompson",
  "email": "ava.thompson@example.com",
  "phone": "+1-555-0101",
  "address": "742 Evergreen Terrace, Springfield, IL 62704, USA",
  "memberSince": "2023-04-12"
}

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
x-api-version: 2

{
  "id": 101,
  "firstName": "Ava",
  "lastName": "Thompson",
  "displayName": "Ava T.",
  "contact": {
    "email": "ava.thompson@example.com",
    "phone": { "countryCode": "+1", "number": "555-0101" }
  },
  "address": {
    "street": "742 Evergreen Terrace",
    "city": "Springfield",
    "state": "IL",
    "postalCode": "62704",
    "country": "US"
  },
  "preferences": { "language": "en-US", "timezone": "America/Chicago", "newsletter": true },
  "createdAt": "2023-04-12T09:30:00Z",
  "apiVersion": "2"
}

Nearly every field changed. name was split into firstName and lastName. phone and address went from strings to objects. memberSince, a date, became createdAt, a full timestamp. Any one of these would break a v1 client, which is why v2 is a separate version rather than an update to v1.

The v1 response also carries headers that tell clients it is on the way out. Deprecation: @1751328000 is a Unix timestamp (1 July 2025), the moment v1 became deprecated. Sunset is the date it is scheduled to stop working. Link points to the replacement. These are standard headers, and a client or a monitoring job can log a warning when it sees them. The v2 response has none of them.

/api/versioned/profilesign in to see serves both versions from one path and picks one from an X-API-Version header, an api-version query parameter or the Accept media type, falling back to the latest. GET /api/versionssign in to see documents the order of precedence as header, then query, then Accept. The response tells you which source won:

$ curl -i -H "X-API-Version: 1" "https://api.testingapis.com/api/versioned/profile?api-version=2"
HTTP/1.1 200 OK
deprecation: @1751328000
sunset: Thu, 31 Dec 2026 23:59:59 GMT
x-api-version: 1
x-api-version-source: header

$ curl -i -H "X-API-Version: 3" https://api.testingapis.com/api/versioned/profile
HTTP/1.1 400 Bad Request

{"status":400,"error":"Bad Request","message":"API version '3' is not supported.","source":"header","supportedVersions":["1","2"]}

For a versioned API, write these tests:

• each selection method returns the version it asked for
• conflicting selections resolve in the documented order
• no selection returns the documented default
• an unsupported version fails clearly with a 400, not a silent fallback
• deprecated versions send Deprecation and Sunset, and current versions don't
• each version still matches its own schema, so fixing a bug in v2 doesn't change v1

The last point is where versioned APIs usually go wrong. Two versions often share code, and a change made for v2 leaks into v1. A schema per version, checked on every build, catches that.