A logistics supplier was acquired in October and their API was decommissioned in November. We continued sending consignments twice a day until February, receiving a successful response every time, because whoever wound the service down left a load balancer answering 200 with an empty body.
The symptom
what we sent: a consignment, twice a day
what we got: 200 OK, Content-Length: 0
what we did: logged success, marked the consignment
dispatched, moved on
what our monitoring alerted on:
4xx and 5xx none occurred
timeouts none occurred
error rate zero, correctly
queue depth zero, correctly
eleven weeks. found when a customer asked where their
order was.A successful response with no content satisfied every check we had, because every check was about failure and nothing was failing. The integration had stopped producing an outcome and had not stopped producing successes.
Why it happens
Monitoring is built after incidents, and incidents are things that fail loudly. An integration that fails silently looks identical to one with no traffic, and both look identical to one that is working.
The fix
Asserting on the response, not the status
// what it was
$response = $this->client->sendRequest($request);
if ($response->getStatusCode() !== 200) {
throw new SupplierRejectedConsignment($response);
}
// what it is
$payload = json_decode((string) $response->getBody(), true);
if (! is_array($payload) || ! isset($payload['consignmentRef'])) {
throw new SupplierContractBroken(
'a 200 with no consignment reference',
$response,
);
}One line asserting that the response contains the thing we asked for, which had never seemed necessary because a 200 means success. It would have turned eleven weeks into one failed dispatch, which is the whole finding and is four lines.
Alerting on absence
- alert: IntegrationSilent
expr: |
sum by (integration) (
increase(integration_success_total[18h])
) == 0
for: 1h
annotations:
summary: "{{ $labels.integration }} has produced no
successful outcome in 18 hours"
runbook: "https://wiki.internal/runbooks/integration-silent"the window has to match the cadence, which means one
rule per integration rather than one rule:
supplier dispatch twice daily → 18h
payment webhook continuous → 1h
address lookup on demand, bursty → 24h
accounting export monthly → not an
alert. a calendar entry and a
dashboard.
five alerts, one calendar entry, and the monthly one
is the case that shows the limit of the technique.An absence alert needs a window shorter than the interval it is watching, and some intervals are too long for an alert to be the right tool. The monthly export is a calendar entry with a person’s name on it, which is a worse mechanism and is the only one available.
What the integration had been doing
established by reading rather than by asking, because
there was nobody left to ask:
it sent a consignment and received a tracking
reference, which we stored and displayed to the
customer.
it did not book anything. the physical dispatch was
arranged by a separate process at the warehouse,
which had been true since 2023 and was not
documented.
so the eleven weeks produced orders with no tracking
reference and physical deliveries that happened
anyway.Reading the code to establish what an integration does is a poor substitute for asking somebody and is what is available when the supplier has been acquired and the person who built it has left. The finding — that the integration had stopped being load-bearing in 2023 — changed the decision entirely.
The decision not to replace it
the successor company offers an equivalent API.
what integrating with it would provide: a tracking
reference on the order page.
what we did instead: the warehouse's own system
already produces one, and it was reaching us in a
nightly CSV that a different job imports.
two lines to display it. no integration.The replacement was a fortnight and the data was already arriving through a path nobody had connected to the problem. Establishing what an integration is for, rather than replacing it because it existed, is the step that saved the fortnight — and it only happened because there was no supplier to renew with automatically.
Decommissioning: the list
the client code and its tests
the mapper and its fixtures
the scheduled job
the queue and its consumer
the API credentials, revoked at their end first
the credentials in our secret store
the outbound firewall rule
the DNS entry for their callback
the webhook endpoint and its signing secret
the monitoring dashboard and two alerts
the runbook
and the twelfth, found in April: a row in a
configuration table that a different job read to decide
whether to run.Eleven items and the code is one of them, which is the point — decommissioning is mostly not a code change. The configuration row found two months later is why this is now a checklist rather than a memory, and it had been causing a nightly job to skip silently.
Verifying it worked
$ ./bin/integration-inventory
active: 5
absence alerts configured: 5
cadence documented: 5
# a deliberate test: pause the address lookup
$ ./bin/pause-integration address-lookup --for=25h
...
IntegrationSilent fired at +25h04m
$ ./bin/orders-without-tracking --since=2025-11
1,204 orders affected
backfilled from the warehouse CSV: 1,204
$ grep -rn 'logistics-supplier' src/ config/ infrastructure/
# (nothing)Pausing an integration deliberately and confirming the alert fires is the step that distinguishes a configured alert from a working one, and it took twenty-five hours of waiting. The backfill was possible because the data had been arriving in a CSV the whole time.
What this costs
An alert per integration that will page somebody about a supplier’s public holiday, which has already happened twice. The window has to be generous enough to avoid that and short enough to be useful, and there is no window that satisfies both for an integration whose cadence is irregular.
The eleven weeks also produced twelve hundred orders with no tracking reference, which was recoverable and would not have been if the warehouse CSV had a shorter retention. The recovery was luck rather than design, and the honest summary is that the monitoring gap was found by a customer.