Billing was extracted into its own deployable in 2021 for independent scaling and independent deployment. Three years later it had never been scaled independently, forty-one of forty-four releases had touched both sides, and a team of four was maintaining four pipelines.
The symptom
$ git log --oneline --since=2021-04 --pretty='%H' |
while read sha; do
./bin/services-touched "$sha"
done | sort | uniq -c
41 app,billing
2 app
1 billing
$ ./bin/release-durations --since=2021-04 | tail -3
median release, single service 18m
median release, both services 41m
releases requiring a rollback 6 (all sequencing)and the six rollbacks, all the same shape:
billing deployed first, expecting a field the app
had not sent yet. or the app first, sending a field
billing did not accept. a contract change across a
network boundary, between two things released
together, by one team.Every one of the six rollbacks is a problem that does not exist when the two are one deployable. A contract between two services released simultaneously by the same four people is not a contract — it is a coupling with a network in it.
Why it happens
The boundary was drawn by subject matter, which is how everybody draws them, and subject matter is not what determines whether two things can be deployed independently. Billing is a coherent domain and it changes when orders change, which is the property that mattered and was not the property anybody measured.
The fix
The five questions, answered honestly
can one be deployed without the other?
once in three years, deliberately, as a test.
can one be rolled back independently?
no. the schema changes are coordinated.
do they share a schema?
they had separate databases and a sync job, which
is sharing a schema with extra steps.
can they be released on different days?
no, and the six rollbacks are the evidence.
does either have a separate team?
no. four people, both services, since 2021.
five noes. this is one service with a network in it.What the split had genuinely bought
one thing, and it was real:
the monthly invoicing run used 3 GB and had twice
exhausted the web tier's memory before the split.
separating it meant that could not happen.
which a separate process pool on the same codebase
also provides:
systemd unit: turkerdev-billing-worker
MemoryMax=4G
same image, same code, different limit.
the isolation was the requirement. the deployable was
not.Naming what the split actually bought, and then finding the cheapest thing that buys it, is the step that makes a merge defensible rather than a reversal. A memory limit on a systemd unit is one line and gives the same guarantee as a separate deployable.
Merging the codebases
# preserving history, which matters for blame
$ git remote add billing ../billing
$ git fetch billing
$ git merge --allow-unrelated-histories billing/main
$ git mv billing-src src/Billing
# 412 files moved, namespaces rewritten, and the
# composer.json entries merged.
$ vendor/bin/phpunit
Tests: 1,802 passed # both suites, one runKeeping the history is worth the awkwardness of an unrelated-histories merge — three years of commits explaining why billing code looks the way it does is the most valuable thing being moved. A fresh copy would have been simpler and would have made every line in src/Billing blame to one commit in February.
The database, which is the hard part
billing had its own instance, and a nightly sync
copying customers and orders into it.
the merge:
1 billing's tables restored into the main instance,
into the same schema, with a naming check for
collisions. two: `settings` and `logs`.
→ renamed to billing_settings, billing_logs.
2 the seven foreign keys that had been dropped in
2021 restored. one failed: 188 billing rows
referenced orders that had been deleted.
→ investigated, and they were test data from the
migration itself.
3 the sync job deleted. 3 incidents in 3 years, all
of them stale data during a failure.-- the check that had to pass before the keys went back
SELECT COUNT(*) FROM billing_invoices bi
LEFT JOIN orders o ON o.id = bi.order_id
WHERE bi.order_id IS NOT NULL AND o.id IS NULL;
-- 188, all created 2021-04-14, the migration windowRestoring the foreign keys is the moment the merge becomes real, because it is the point at which the database enforces what the sync job had been approximating. The hundred and eighty-eight orphans had existed for three years and nothing had ever noticed, which is what a boundary without referential integrity means in practice.
Deleting the resilience machinery
$ git show --stat HEAD
14 files changed, 8 insertions(+), 756 deletions(-)
src/Http/BillingClient.php | 188 -----
src/Http/RetryPolicy.php | 41 ---
src/Http/CircuitBreaker.php | 94 ----
src/Billing/FallbackInvoice.php | 31 ---
tests/Http/BillingClientTest.php | 402 -----
...
# replaced by:
- $response = $this->billingClient->raiseInvoice($dto);
+ $reference = $this->billing->raiseInvoice($order);Seven hundred and fifty-six lines existing to survive a network hop that did not need to exist. None of it was badly written and all of it was solving a problem the architecture had created — which is the clearest possible accounting of what a service boundary costs when it is not paying for itself.
The eventual consistency that went away
two bugs, both closed by the merge:
an invoice raised against an order that had been
cancelled in the two seconds between the HTTP call
and the write. reported 4 times in 3 years,
never reproduced.
a refund total that disagreed with the order total
for up to a minute after a line was adjusted,
because the sync ran on a schedule.
both are now one transaction.What we kept
layers:
- name: Billing
collectors: [{ type: directory, value: src/Billing/.* }]
- name: BillingApi
collectors: [{ type: className, value: App\Billing\BillingApi }]
- name: Rest
collectors: [{ type: directory, value: src/(?!Billing).* }]
ruleset:
Rest: [BillingApi]
Billing: [Rest]The module boundary from 2023 survives the merge unchanged, which is the point worth making: the boundary was never the problem. Enforcing it with a layer rule costs a configuration file and enforcing it with a network cost four pipelines, a retry policy and six rollbacks.
The one thing that got worse
a bad deploy now takes both down.
before: a broken billing release left the shop
running, with invoicing degraded and a
fallback response.
after: a broken release is a broken release.
which we accepted, because:
the fallback had produced stale data in 2 of the 3
times it activated, and
41 of 44 releases changed both anyway, so a broken
billing release had usually been a broken release.
mitigated by the health check and automatic rollback,
which now covers both.Verifying it worked
# six months after
pipelines 4 → 2
median release 41m → 19m
releases per week 2.1 → 3.4
rollbacks 6 in 3 years → 1 in 6 months
cross-service bugs 2 open → 0
$ vendor/bin/deptrac
0 violations # the module boundary holds
$ ./bin/memory-profile billing-worker
peak 2.9 GB, limit 4 GB # the isolation, preserved
# and the honest one
$ ./bin/incident-log --since=2024-02 --tag=deploy
1 incident: a migration that locked a table.
it would have happened either way.Releases per week going up is the outcome that mattered most, and it is a consequence of the median release halving rather than of anybody deciding to ship more. The memory profile is the check that the thing the split existed for is still true.
What this costs
An admission in public that the 2021 decision was wrong, in a decision record that supersedes one written by the same person. That is uncomfortable and it is the only way the reasoning survives — a reversal with no record reads to the next person as though nobody ever thought about it.
The merged deployable is also larger, and the layer rule is the only thing preventing it from becoming a single mass. That rule is a file somebody can edit, where a network boundary is a thing that has to be physically crossed — the enforcement is genuinely weaker, and the argument is that a weaker boundary that is respected beats a stronger one that produced six rollbacks.
And the split may have been right for a team that was going to grow and did not. The revisit conditions are written down — a separate team, or genuinely diverging release cadences — which is the same sentence as 2023 and is the honest position: this is the right shape for four people, and it is a shape with a stated expiry.