The API had a Postman collection with sixty requests in it. It lived in one person’s workspace, it was shared by a link that occasionally expired, and it was the only place anybody had written down what a valid request looked like. When that person was on holiday, the collection was unavailable and nobody could reproduce a reported bug.
The symptom
# what happened on the 14th
# we renamed contact_email to email in the order resource
# the mobile team found out on the 19th, from a crash report
#
# what would have caught it:
# the collection. which nobody ran, because running it meant
# opening an application and clicking sixty times.
$ git log --oneline --all -- api/
# nothing. the collection was never in the repository.Five days between the change and the discovery, and the discovery was a crash report from a user’s phone. The collection contained a request that would have failed immediately, and nothing ran it.
Why it happens
A manual test is a memory, and memories are per person. A collection that has to be opened and clicked is run when somebody remembers to run it, which is before a release and not after a merge — and the change that breaks a client is rarely the one somebody thought was risky.
The second reason is that the collection is not where the code is. A pull request renaming a field has no relationship to a document in a different application owned by a different account, so there is no moment at which the two are seen together.
The fix
Export it, commit it, review it
$ ls api/
collection.json
environments/
ci.json # committed. no secrets.
local.json # committed. no secrets.
$ git add api/collection.json
$ git commit -m 'add the refunds endpoint to the collection'
# and the diff a reviewer now sees alongside the controller changeExporting and committing means a change to the collection appears in the same pull request as the change to the endpoint, which is the entire point — and the absence of a collection change on a pull request that added an endpoint is now a visible omission rather than an invisible one.
The export format is verbose JSON with generated identifiers in it, so the diffs are noisy and re-exporting an unchanged collection can produce a diff. Normalising on export with jq -S before committing removes most of it, and nothing does that automatically.
Newman, which is what makes it a test
- name: Contract tests
run: |
npx newman run api/collection.json
-e api/environments/ci.json
--env-var "base_url=http://localhost:8000"
--env-var "token=${{ secrets.CI_API_TOKEN }}"
--reporters cli,junit
--reporter-junit-export newman-results.xml
- uses: mikepenz/action-junit-report@v2
if: always()
with: { report_paths: 'newman-results.xml' }The --env-var override is how a secret reaches the run without being in a committed file, and it is the mechanism that makes the committed environment safe to have empty values in. JUnit output means the results appear in the pipeline’s test report rather than only in the log, which is what makes a failure legible to somebody who did not write the collection.
if: always() on the report step is required, because a failing Newman run fails the step and the report would otherwise be skipped — leaving a red build with no indication of which request failed.
Assertions about shape, not about values
pm.test('the order resource has not changed shape', () => {
const body = pm.response.json();
pm.expect(pm.response.code).to.equal(200);
pm.expect(body).to.have.property('id').that.is.a('number');
pm.expect(body).to.have.property('email').that.is.a('string');
pm.expect(body).to.have.property('total_cents').that.is.a('number');
pm.expect(body.status).to.be.oneOf(['pending', 'paid', 'shipped']);
});
// NOT this, which fails whenever the seed data changes:
// pm.expect(body.total_cents).to.equal(4900);Asserting on presence and type rather than value is what makes the suite stable — the data changes every sprint and the contract must not. A test that fails for the wrong reason is a test that gets deleted, and a collection full of value assertions fails constantly.
The oneOf check on an enumerated field is the middle ground and is worth having: a new status value appearing is exactly the change a client needs to know about, and it is precisely the change that no type assertion would catch.
Environments as files, and the secret that must not be one
{
"name": "ci",
"values": [
{ "key": "base_url", "value": "http://localhost:8000", "enabled": true },
{ "key": "token", "value": "", "type": "secret", "enabled": true }
]
}An empty value with the secret type is the committed template and the real value arrives from the command line. Postman’s own secret type hides the value in the interface and still exports it, which is a distinction worth knowing before trusting it — a personal environment exported carelessly puts a real token in a diff.
Scanning the repository history for anything token-shaped is worth doing once at the start, because this is a file people commit before thinking about it.
Running it against a deployed environment as a smoke test
smoke:
needs: deploy
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- run: |
npx newman run api/collection.json
--folder 'smoke'
--env-var "base_url=https://staging.example/api"
--env-var "token=${{ secrets.STAGING_TOKEN }}"The --folder flag runs a subset, which is what makes one collection serve two purposes: the whole thing in CI against a local application, and a small read-only folder against a deployed environment. A smoke folder containing three GET requests catches a deploy that started and cannot serve traffic, which is the failure a health check sometimes misses.
Anything in the smoke folder must be safe to run against a live system, which means read-only — and enforcing that is a review habit rather than a mechanism, because nothing stops somebody adding a POST.
Verifying it worked
# reintroduce the change that started this
$ git revert --no-commit a3f9c11 # rename email back to contact_email
$ npx newman run api/collection.json -e api/environments/ci.json
┌─────────────────────────┬──────────┬──────────┐
│ │ executed │ failed │
│ assertions │ 184 │ 3 │
└─────────────────────────┴──────────┴──────────┘
1. the order resource has not changed shape
expected { id: 91204, contact_email: ... } to have property 'email'
$ echo $?
1Reintroducing the original breaking change and watching the suite catch it is the assertion that the whole exercise was for, and it is worth doing rather than assuming — a collection full of assertions that never fail is a collection asserting nothing. The exit code is what CI reads, and confirming it is non-zero is the second half.
What this costs
A second test suite in a second language. The assertions are JavaScript inside JSON strings, which has no syntax highlighting, no linting and no refactoring support — editing them in a text editor is unpleasant and editing them in Postman means re-exporting. That is a real ergonomic cost and it is the reason these suites decay.
The honest question is whether this duplicates the application’s own feature tests, and mostly it does. What it adds is that the collection is the artefact the client teams already have, so keeping it correct serves two purposes at once — and a schema validation test in PHPUnit would catch the same regression with better ergonomics and would not be shared with anybody. For an API with external consumers the duplication earns its place; for a purely internal one it probably does not.