Automating GraphQL tests

Everything so far has been done by hand. This lesson turns the checks from the earlier lessons into tests that run in a second. The REST version is automating tests with code; if you've done that one, most of this will look familiar, and the parts that don't are the parts specific to GraphQL.

You don't need a GraphQL library to test a GraphQL API. A request is a POST with a JSON body, and every language can send one. Client libraries like Apollo add caching and state management for apps, which is the opposite of what a test wants: you want to see every byte the server sends, not a cached copy of it.

A helper for every request

The examples use JavaScript with the test runner built into Node.js 18 and later, so there's nothing to install. Start with a small function that sends a query and returns the status and body:

// graphql.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";

const URL = process.env.GRAPHQL_URL ?? "https://api.testingapis.com/graphql";
const KEY = process.env.TESTINGAPIS_KEY;

async function gql(query, variables) {
  const res = await fetch(URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Accept: "application/graphql-response+json",
      ...(KEY ? { "X-TestingApis-Key": KEY } : {}),
    },
    body: JSON.stringify({ query, variables }),
  });
  return { status: res.status, body: await res.json() };
}

// Fails with the server's own error messages instead of a TypeError three lines later.
function assertNoErrors(body) {
  assert.equal(body.errors, undefined, JSON.stringify(body.errors, null, 2));
}

Three choices in there are deliberate.

• The URL and key come from environment variables, so the same tests can run against a local server, a staging copy and production.
• The Accept header is pinned. The errors lesson showed that it changes the status code; without pinning it, a test that checks the status depends on the defaults of whatever HTTP library you happen to use. Pin it to what your real client sends.
• assertNoErrors puts the errorsarray into the failure message. That one line saves more debugging time than anything else in the file, as you'll see below.

Testing queries

test("product 1 has the fields the product page needs", async () => {
  const { body } = await gql(
    `query Product($id: Int!) {
      product(id: $id) { id title price inStock rating { rate count } }
    }`,
    { id: 1 },
  );

  assertNoErrors(body);
  const p = body.data.product;
  assert.ok(p, "product 1 should exist");
  assert.equal(p.id, 1);
  assert.equal(typeof p.title, "string");
  assert.ok(p.price > 0, `price should be positive, got ${p.price}`);
  assert.ok(p.rating.rate >= 0 && p.rating.rate <= 5);
});

test("a missing book is null, with no error", async () => {
  const { body } = await gql(`{ book(id: 999) { title } }`);

  assertNoErrors(body);
  assert.equal(body.data.book, null);
});

The first test checks the fields a real screen depends on, and checks their types and ranges, not just their presence. The second pins down what "not found" looks like on this API: a null with no error. If someone changes it to return an error instead, this test fails and forces a conversation, which is the point. Clients were written against the old behaviour.

Notice that the queries use variables rather than pasting the ID into the string. Tests are code, and the reasons from the variables lesson apply to them too.

Testing errors

test("an unknown field is rejected before anything runs", async () => {
  const { status, body } = await gql(`{ book(id: 1) { publisher } }`);

  assert.equal(status, 400);
  assert.equal(body.data, undefined);
  assert.deepEqual(body.errors[0].path, ["book"]);
  assert.equal(body.errors[0].extensions.field, "publisher");
});

test("addTodo rejects a priority outside high, medium and low", async () => {
  const { body } = await gql(`mutation {
    addTodo(title: "Bad priority", userId: 1, priority: "urgent") { id }
  }`);

  assert.equal(body.data, null);
  assert.deepEqual(body.errors.map((e) => e.path), [["addTodo"]]);
});

The first test checks a request error: status 400 (fine here, because Accept is pinned), no data key at all, and an error that points at the right field. It checks path and extensions.fieldrather than the message, because messages change between server versions and the structured fields usually don't.

The second checks a field error, and it's strict on purpose: data must be exactly null, and there must be exactly one error, on addTodo. A looser check like "errors is not empty" would also pass if the mutation failed for a completely different reason, such as the server being unable to reach its database.

A create, read and delete flow

test("a created user can be read back, then deleted", async () => {
  const created = await gql(
    `mutation Create($name: String!, $job: String!, $city: String!) {
      createUser(name: $name, job: $job, city: $city) { id }
    }`,
    { name: "Test User", job: "Automation", city: "Lisbon" },
  );
  assertNoErrors(created.body);
  const id = created.body.data.createUser.id;

  try {
    const read = await gql(`query ($id: Int!) { user(id: $id) { name job city } }`, { id });
    assertNoErrors(read.body);
    assert.deepEqual(read.body.data.user, { name: "Test User", job: "Automation", city: "Lisbon" });
  } finally {
    const deleted = await gql(`mutation ($id: Int!) { deleteUser(id: $id) }`, { id });
    assert.equal(deleted.body.data.deleteUser, `User ${id} deleted successfully`);
  }

  const gone = await gql(`query ($id: Int!) { user(id: $id) { id } }`, { id });
  assert.equal(gone.body.data.user, null);
});

The delete is in a finally block, so the user is removed even when a check fails halfway. On a shared API that matters; on your own staging environment it stops test data piling up until someone notices the users page takes ten seconds to load. The last query confirms the delete actually happened, for the same reason you read back after a create.

The test also checks the exact text of deleteUser's answer. That's fragile, as the mutations lessonpointed out, but it's the only success signal this mutation gives. When an API makes you write a fragile test, the fragile test is evidence for changing the API.

Running the tests

$ node --test
✔ product 1 has the fields the product page needs (296.4892ms)
✔ a missing book is null, with no error (92.8912ms)
✔ an unknown field is rejected before anything runs (46.3161ms)
✔ addTodo rejects a priority outside high, medium and low (45.2961ms)
✔ a created user can be read back, then deleted (135.6521ms)
ℹ tests 5
ℹ pass 5
ℹ fail 0

Now break one. Change inStock to inStok in the first test and run again:

✖ product 1 has the fields the product page needs (179.1768ms)
  AssertionError [ERR_ASSERTION]: [
    {
      "message": "The field `inStok` does not exist on the type `Product`.",
      "locations": [
        {
          "line": 2,
          "column": 41
        }
      ],
      "path": [
        "product"
      ],
      ...

The failure says exactly what's wrong, and where, in the server's own words. Without assertNoErrors, the same mistake would surface as TypeError: Cannot read properties of undefined (reading 'product'), because datais missing, and you'd spend five minutes working out why.

The same tests in Python

With pytest and requests, the helper and two of the tests look like this:

# test_graphql.py
import json
import os

import requests

URL = os.environ.get("GRAPHQL_URL", "https://api.testingapis.com/graphql")
KEY = os.environ.get("TESTINGAPIS_KEY")


def gql(query, variables=None):
    headers = {"Accept": "application/graphql-response+json"}
    if KEY:
        headers["X-TestingApis-Key"] = KEY
    res = requests.post(URL, json={"query": query, "variables": variables}, headers=headers, timeout=10)
    return res.status_code, res.json()


def assert_no_errors(body):
    assert "errors" not in body, json.dumps(body["errors"], indent=2)


def test_missing_book_is_null_without_error():
    _, body = gql("{ book(id: 999) { title } }")

    assert_no_errors(body)
    assert body["data"]["book"] is None


def test_unknown_field_is_rejected_before_anything_runs():
    status, body = gql("{ book(id: 1) { publisher } }")

    assert status == 400
    assert "data" not in body
    assert body["errors"][0]["path"] == ["book"]

$ pip install pytest requests
$ pytest -q test_graphql.py
....                                                         [100%]
4 passed in 3.12s

The four tests in the full file are the same as the JavaScript ones minus the addTodo case. Note "data" not in body rather than body["data"] is None: a missing key and a null value mean different things in GraphQL, and the test should say which one it expects.

What's worth automating

A GraphQL API can be asked an endless number of different queries, so you can't test them all. Spend the tests here:

• The queries the real clients send. Ask the developers for them; apps often keep them in .graphql files. Those are the ones whose breakage someone will notice.
• Every mutation, with good input, bad input and a missing record.
• Nullable fields, with a record where the field is null. That's where client crashes come from.
• A schema diff against the last release, as described in the schema lesson. It covers every field without a test per field.