Data-driven testing with the Collection Runner

Many tests are the same request with different inputs. Check product 1, then product 6, then a product that doesn't exist. Copying the request for each case gives you a collection that's tedious to change. Data-driven testing keeps one request and a table of inputs and expected results, and runs the request once per row. Postman does this through the Collection Runner and a data file.

The data file

A data file is CSV or JSON. In CSV, the first line names the columns and each later line is one run. The tutorial's products.csvchecks one product from each category and one that doesn't exist:

id,expectedStatus,category
1,200,electronics
6,200,clothing
10,200,books
14,200,home
18,200,sports
9999,404,

Each column becomes a variable for that row, in the data scope from lesson 2. So the request URL can simply be {{baseUrl}}/api/Products/{{id}}, and on the first row {{id}} is 1. The last row has an empty category, because a missing product has none; an empty CSV field is an empty string.

Tests that read the row

The expected results come from the row too. pm.iterationDatagives a script the current row's values:

const expectedStatus = Number(pm.iterationData.get("expectedStatus"));

pm.test(`product ${pm.iterationData.get("id")} returns ${expectedStatus}`, () => {
  pm.response.to.have.status(expectedStatus);
});

if (expectedStatus === 200) {
  pm.test(`product ${pm.iterationData.get("id")} is in ${pm.iterationData.get("category")}`, () => {
    pm.expect(pm.response.json().category).to.eql(pm.iterationData.get("category"));
  });
}

Two details make this work well. The test names include the row's values, so the results read product 10 is in booksrather than six identical lines, and a failure says which row failed. And the category test only runs when a 200 is expected, so the 404 row isn't checked for a category it can't have. Every value from a CSV is text, so expectedStatus is converted with Numberbefore it's compared.

Running it

1. Open the Collection Runner from the collection or folder's menu, and pick the folder 06 Data-driven.
2. Select the data file. Postman counts the rows and sets the number of iterations to match.
3. Check the preview, which shows how Postman read the columns, and start the run.

Newman does the same with -d. This is the real output:

$ newman run testingapis-postman-collection.json -e testingapis-postman-environment.json \
    --folder "06 Data-driven" -d products.csv

Iteration 1/6
  √  product 1 returns 200
  √  product 1 is in electronics
Iteration 2/6
  √  product 6 returns 200
  √  product 6 is in clothing
Iteration 3/6
  √  product 10 returns 200
  √  product 10 is in books
Iteration 4/6
  √  product 14 returns 200
  √  product 14 is in home
Iteration 5/6
  √  product 18 returns 200
  √  product 18 is in sports
Iteration 6/6
  √  product 9999 returns 404

│              iterations │                 6 │                 0 │
│              assertions │                11 │                 0 │

Six iterations, eleven assertions: two for each product that exists, one for the missing one. Adding a test case is now a one-line change to the CSV, which someone who never opens the scripts can make.

When a row fails

Here a second row claims product 2 is a book:

Iteration 2/2

└ Product by ID, from the data file
  GET https://api.testingapis.com/api/Products/2 [200 OK, 553B, 38ms]
  √  product 2 returns 200
  1. product 2 is in books

 1.  AssertionError  product 2 is in books
     iteration: 2    expected 'electronics' to deeply equal 'books'

The iteration number, the row's values in the test name, and the expected and actual values. When a data-driven test fails, the first question is always whether the API is wrong or the data file is. Here it's the file: product 2 is a smartwatch. Keep data files under version control next to the collection, so a change to the expected results is reviewed like a change to code.

Running without a data file

Without a data file, {{id}} has no value and the request would fail. The request guards against that with a pre-request script:

// Without a data file there's no id to test, so skip instead of failing the whole run
if (!pm.iterationData.has("id")) {
  pm.execution.skipRequest();
}

A full run of the collection without data now skips this request rather than reporting a failure. Newman prints the request's name with nothing under it, counts zero requests for the folder, and the run still passes. That keeps one command, newman run collection.json -e environment.json, green for everyday use, with data-driven runs as a separate step.

JSON data files

JSON files hold the same thing as an array of objects, one per iteration. They're better when values aren't plain text: numbers stay numbers, and a row can hold a whole request body as a nested object.

[
  { "id": 1, "expectedStatus": 200, "category": "electronics" },
  { "id": 9999, "expectedStatus": 404 }
]

A field left out of an object, like category in the second row, has no value at all, whereas an empty CSV column is an empty string. pm.iterationData.has tells the two apart.

What makes good rows

• One row per equivalence class: one product from each category, not five from the same one.
• Boundaries: the first ID, a very large ID, zero, a negative number, text where a number belongs.
• Rows that should fail, with their expected status, next to rows that should pass.
• No rows that depend on data someone else might change. Product IDs from the seed data are stable; counts aren't.

The REST tutorial's lesson on writing test cases explains equivalence classes and boundary values, which are what decide the rows.