JSON Schema and parameterized tests

Validate whole responses against a schema, and run one test over many inputs.

Field-by-field checks only cover the fields you thought to check, and copying a test for every input makes a suite long and hard to change. This lesson tackles both: a JSON Schema checks the whole shape of a response in one line, and a JUnit parameterized test runs one test method over a table of inputs. Together they give broad coverage with very little code, in SchemaAndDataTest.java.

Checking a response against a schema

A JSON Schema describes which fields a document must have and what type and range each one can take. The example project keeps one for products in src/test/resources/schemas:

schemas/product.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Product",
  "type": "object",
  "required": ["id", "title", "price", "category", "rating", "inStock"],
  "properties": {
    "id": { "type": "integer", "minimum": 1 },
    "title": { "type": "string", "minLength": 1 },
    "price": { "type": "number", "minimum": 0, "exclusiveMinimum": true },
    "description": { "type": "string" },
    "category": { "enum": ["electronics", "clothing", "books", "home", "sports"] },
    "image": { "type": "string", "format": "uri" },
    "rating": {
      "type": "object",
      "required": ["rate", "count"],
      "properties": {
        "rate": { "type": "number", "minimum": 0, "maximum": 5 },
        "count": { "type": "integer", "minimum": 0 }
      }
    },
    "inStock": { "type": "boolean" },
    "createdAt": { "type": "string" }
  }
}

The test applies it with matchesJsonSchemaInClasspath("schemas/product.json"), from REST Assured's json-schema-validator module. If a future release makes price a string, drops inStock, adds a sixth category or returns a rating of 7, this one line fails, though no other test mentions those fields. The REST tutorial's lesson on schemas and contract testing covers writing schemas in more depth.

The draft 4 trap

JSON Schema has several versions, called drafts, and REST Assured's validator is built on a library that reads draft 4. The first version of this schema used the newer way of writing an exclusive minimum, and every schema test failed before checking anything:

Terminal

io.restassured.module.jsv.JsonSchemaValidationException:
com.github.fge.jsonschema.core.exceptions.InvalidSchemaException: fatal: invalid JSON Schema, cannot continue
Syntax errors:
[ {
  "level" : "error",
  "message" : "value has incorrect type (found integer, expected one of [boolean])",
  "schema" : { "pointer" : "/properties/price" },
  "keyword" : "exclusiveMinimum",
  "found" : "integer",
  "expected" : [ "boolean" ]
} ]

In draft 4, exclusiveMinimum is a true or false flag that modifies minimum; from draft 6 onwards it's a number of its own:

JSON

// Draft 6 and later: a number
"price": { "type": "number", "exclusiveMinimum": 0 }

// Draft 4, which REST Assured's validator reads: a minimum plus a boolean
"price": { "type": "number", "minimum": 0, "exclusiveMinimum": true }

So declare "$schema": "http://json-schema.org/draft-04/schema#", as the file above does, and write draft 4 rules. Schemas generated by tools, or copied from an OpenAPI 3.1 document, often use newer drafts and need translating, or a different validator library.

One test, many inputs

JUnit's @ParameterizedTest runs a method once per row of input. @CsvSource gives the rows inline, and @ValueSource a simple list:

SchemaAndDataTest.java

package com.testingapis.examples;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;

import static com.testingapis.examples.ApiSpecs.api;
import static io.restassured.RestAssured.given;
import static io.restassured.module.jsv.JsonSchemaValidator.matchesJsonSchemaInClasspath;
import static org.hamcrest.Matchers.equalTo;

/** Lesson 7: one schema for every product, and one test method for many inputs. */
class SchemaAndDataTest {

    @ParameterizedTest(name = "product {0} is in {1}")
    @CsvSource({
        "1,  electronics",
        "6,  clothing",
        "10, books",
        "14, home",
        "18, sports"
    })
    void eachProductMatchesTheSchemaAndItsCategory(int id, String category) {
        given()
            .spec(api())
            .pathParam("id", id)
        .when()
            .get("/api/Products/{id}")
        .then()
            .statusCode(200)
            .body(matchesJsonSchemaInClasspath("schemas/product.json"))
            .body("category", equalTo(category));
    }

    @ParameterizedTest(name = "product {0} does not exist")
    @ValueSource(ints = {0, -1, 9999})
    void idsWithoutAProductReturn404(int id) {
        given()
            .spec(api())
            .pathParam("id", id)
        .when()
            .get("/api/Products/{id}")
        .then()
            .statusCode(404);
    }
}

Source	Gives	Use it for
`@ValueSource`	One value per run	A list of IDs, strings or numbers
`@CsvSource`	Several values per run, inline	Small tables of input and expected result
`@CsvFileSource`	Rows from a CSV file in resources	Bigger tables, or tables testers edit without touching Java
`@MethodSource`	Rows built by a method	Inputs that need code to create, like generated objects

The first method checks one product from each category against both the schema and its expected category. The second checks three IDs with no product: 0, a negative number and a large one. Those are boundary values; the REST tutorial's lesson on writing test cases explains how to choose them. Eight tests from two short methods, and adding a case is one line.

Test names in reports

The name on each @ParameterizedTest gives every run a readable name, such as product 1 is in electronics, and IDEs show those. Maven's Surefire plugin, by default, ignored them in its XML report and wrote this:

Surefire XML report, default settings

eachProductMatchesTheSchemaAndItsCategory(int, String)[1]
eachProductMatchesTheSchemaAndItsCategory(int, String)[2]
...
idsWithoutAProductReturn404(int)[3]

A failure in CI then says "[3] failed", and someone has to count rows to find which input broke. The project's pom.xml sets Surefire's usePhrasedTestCaseMethodName option, and the same run reports:

Surefire XML report, with usePhrasedTestCaseMethodName

eachProductMatchesTheSchemaAndItsCategory(int, String) product "1" is in "electronics"
eachProductMatchesTheSchemaAndItsCategory(int, String) product "6" is in "clothing"
...
idsWithoutAProductReturn404(int) product 9999 does not exist

What to cover this way

One schema per kind of resource, applied to every response of that kind.
Parameterized tests for each equivalence class and boundary, not five rows from the same class.
Rows that should fail, with their expected status, next to rows that should pass.
Inputs that don't depend on data others can change: seed IDs are stable, counts aren't.

Exercise

Write a draft 4 schema for a todo, with priority limited to high, medium and low, and a parameterized test that checks todos 1 to 5 against it. Then add @CsvFileSource with a file of five product IDs and expected prices, and compare prices as Float values. Which is easier to change when a price changes: the CSV or the Java?