Contract tests in CI

Contract checks are only useful if they run before a change reaches the consumers, every time, without anyone remembering to run them. That means CI. This lesson puts the tutorial's pieces into a pipeline: which checks run, where, against what, and what happens when one fails.

Two checks

They answer different questions. The diff asks whether the promise got weaker. Verification asks whether the API keeps its promises. A change can pass one and fail the other: an API that starts sending a string price without updating its spec passes the diff and fails verification; a spec edit that removes a field fails the diff before any code changes.

Scripts that fail the build

CI decides pass or fail from exit codes, so both scripts exit with 1 when they find a problem. The breaking-change check takes two spec files, or URLs:

// Compares two versions of an OpenAPI document and lists breaking changes. Each argument is a file or a URL.
// Exits with 1 if there are any, so CI can stop a release that would break clients.
// Usage: node scripts/check-breaking.mjs old.json new.json

import { readFileSync } from "node:fs";
import { apiHeaders } from "../src/api.mjs";
import { breakingChanges } from "../src/diff.mjs";

async function load(source) {
  if (/^https?:\/\//.test(source)) return (await fetch(source, { headers: apiHeaders() })).json();
  return JSON.parse(readFileSync(source, "utf8"));
}

const [oldSource, newSource] = process.argv.slice(2);
if (!oldSource || !newSource) {
  console.error("Usage: node scripts/check-breaking.mjs <old spec> <new spec>");
  process.exit(2);
}

const problems = breakingChanges(await load(oldSource), await load(newSource));

if (problems.length === 0) console.log("No breaking changes.");
else {
  console.log(`${problems.length} breaking change(s):`);
  for (const problem of problems) console.log(`  ${problem}`);
}

process.exitCode = problems.length ? 1 : 0;

$ node scripts/check-breaking.mjs main.json same.json
No breaking changes.
$ echo $?
0

$ node scripts/check-breaking.mjs main.json branch.json
5 breaking change(s):
  GET /api/Products 200 response[]: property "inStock" removed
  POST /api/Products 201 response.data: property "inStock" removed
  GET /api/Products/{id} 200 response: property "inStock" removed
  PUT /api/Products/{id} 200 response.data: property "inStock" removed
  GET /api/Products/category/{name} 200 response[]: property "inStock" removed
$ echo $?
1

The second run compares the production spec with a branch that removed inStock, and fails with every operation it affects. scripts/verify-consumer.mjs works the same way for consumer contracts. The tests check the exit codes themselves, since a check that always exits 0 would let everything through:

test("check-breaking exits 1 and lists the problems for a breaking change", async () => {
  const { code, stdout } = await script("scripts/check-breaking.mjs", oldFile, newFile);
  assert.equal(code, 1);
  assert.match(stdout, /property "inStock" removed/);
});

A pipeline for the provider

Here is how the checks fit together in a GitHub Actions workflow in the provider's repository, run on every pull request. It builds and starts the API from the branch, fetches both specs, and runs both checks:

# .github/workflows/contract.yml, in the provider's repository
name: Contract checks

on: pull_request

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      # Start the API built from this branch. Replace these two steps with however your provider runs.
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 9.0.x
      - name: Start the API from this branch
        run: |
          dotnet run --project api --urls http://localhost:5299 &
          for i in $(seq 1 60); do curl -sf http://localhost:5299/health > /dev/null && break; sleep 1; done

      - name: Fetch the spec in production and the spec on this branch
        run: |
          curl -sf https://api.example.com/openapi/v1.json -o main.json
          curl -sf http://localhost:5299/openapi/v1.json -o branch.json

      - name: No breaking changes
        working-directory: contract-testing
        run: npm run breaking -- ../main.json ../branch.json

      - name: Responses match the spec, and every consumer contract passes
        working-directory: contract-testing
        env:
          BASE_URL: http://localhost:5299
        run: |
          npm test
          npm run verify

• The baseline is the spec in production, not the one on the main branch, because production is what consumers use today. Something merged but not yet released can still be changed freely.
• The branch's spec comes from the running app, so a generated spec reflects the code in the pull request.
• Verification runs against the branch's API on localhost, so the pull request is tested before it merges. Running it against a shared staging environment as well catches configuration differences.
• The diff runs first. It needs no running API for its own work and fails fastest, so a breaking change is reported within seconds.

When the diff fails on purpose

Sometimes a breaking change is intended, at the contract stage of an expand-and-contract change (lesson 6). The pipeline shouldn't make that impossible, only deliberate. Common ways to allow it:

• A label on the pull request, such as breaking-change-approved, that a later step checks for.
• A file in the repository listing accepted breaking changes, reviewed like code.
• A new API version, so the old version's spec stays unchanged and the diff runs per version.

Whichever you choose, record who approved the change and how consumers were told.

The consumer's side

Consumers run checks too. When a consumer changes its contract, by starting to use a new field for example, its own pipeline verifies the new contract against the provider, usually a staging environment, before the consumer ships code that depends on it. And when the provider publishes a new spec, consumers can run the contract-against-spec check from lesson 7 to see early whether it affects them.

Contract testing checklist

• The spec describes every response consumers use, with required fields and accurate nullability.
• Responses are checked against the spec, error responses included.
• Requests in client tests are checked against the spec too.
• Every pull request is diffed against the production spec, and breaking changes need explicit approval.
• Known consumers have contracts, verified on every provider change.
• Drift tests are named for the side that's out of step, and replaced when the drift is fixed.

For running test suites in CI generally, the REST tutorial's lesson on running tests in CI covers splitting fast and slow checks.