Two services, one database, and the year it took to separate them

The billing service was extracted in 2017 and it reached into the main application’s database because the data was there and building an endpoint would have taken a week. Two years later a nullable column could not be added to orders without a release meeting, and the two teams had a shared deploy calendar. That is what the shortcut cost, paid in instalments.

The symptom

# the migration that started the conversation
$ cat database/migrations/2019_03_04_add_tax_scheme.php
  $table->string('tax_scheme')->nullable();

# and the checklist it required
#   1. billing must deploy code that tolerates the column        (their release)
#   2. shop deploys the migration                                (our release)
#   3. billing deploys code that reads it                        (their release)
#   4. neither may roll back past the other
#
# three releases across two teams, for a nullable column.

Nothing was broken. Everything worked. The cost was entirely in coordination, which is the kind of cost that does not appear on any dashboard and is obvious to everybody doing the work.

Why it happens

The database is the most available integration point in any system: it is already running, it already has the data, and connecting to it takes a connection string. Every reason not to is about a future that has not arrived yet, and the person under deadline pressure is not wrong that an endpoint is slower to build.

What makes it structural rather than a mistake is that nothing marks the boundary afterwards. There is no compiler error, no linter, no failing test — the coupling is invisible in both codebases and shows up only as a meeting.

The fix

Deciding who owns the tables

This is the whole decision and everything else is implementation. Ownership means one service may change the schema and the others must not read it directly — and the interesting part is that it can be declared before any code changes.

shop owns      orders, order_lines, products, customers
billing owns   invoices, payments, credit_notes, tax_schemes

nobody owns    the shared tables billing reads today.
               those are the work.

written down, in both repositories, on day one —
before anything is built, so that new code stops adding to it.

Writing it down first stops the problem growing while the fix is being built, which is a year during which people ship features. Without that, the migration is chasing a moving target and never converges.

Enforcing it is a database grant rather than a convention. Revoking the billing user’s access to the shop tables, table by table as each one is migrated, turns a rule into a mechanism — and the failure is loud and local rather than a review comment somebody may not make.

Reads first: a read model, not a query

// billing maintains its own table, shaped for its own queries
final class CustomerProjection
{
    public function on(CustomerRenamed $event): void
    {
        DB::table('billing_customers')->updateOrInsert(
            ['customer_id' => $event->customerId],
            [
                'name'       => $event->name,
                'updated_at' => $event->occurredAt,   // the SOURCE time
            ]
        );
    }
}

// staleness is now measurable:
//   SELECT NOW() - MAX(updated_at) FROM billing_customers

Storing the event’s timestamp rather than the local one is what makes the lag a number that can be graphed and alerted on. Without it, “how stale is this” has no answer and every conversation about the read model is speculative.

The projection can be shaped for the queries that use it, which is frequently a larger win than the decoupling — a denormalised table serving one screen beats a join across a boundary. Rebuilding it from the event history is the recovery path, and it exists only if the history is retained, which is a retention decision to make deliberately rather than discover.

Writes: an API, an event, and which one each case needs

// billing needs an answer now → synchronous, and it accepts the coupling
$reservation = $this->shop->reserveCredit($customerId, $amount);

// billing is stating a fact → an event, and nobody waits
$this->events->publish(new InvoiceIssued($invoiceId, $customerId, $total));

// the test that decides:
//   if this arrives an hour late, is the system WRONG or just BEHIND?
//   wrong  → synchronous
//   behind → an event

Most of the writes turned out to be facts rather than requests, which is the usual result and is the reason this is tractable. The ones that genuinely need an answer stay synchronous and keep their coupling — the goal is not to remove every dependency, it is to remove the invisible ones.

The intermediate state, which lasts longer than anyone plans for

For most of the year, some tables were owned and some were not, and billing read a projection for two entities and the live table for four. That state has to be survivable rather than merely temporary.

final class Customers
{
    public function find(int $id): Customer
    {
        if (Feature::enabled('billing.customers.projection')) {
            return $this->fromProjection($id);
        }

        return $this->fromSharedTable($id);
    }
}

// one flag per entity, flipped independently, reversible in seconds.
// and a ticket per flag, with a date, to delete the losing branch.

A flag per entity rather than one for the whole migration is what makes the rollout incremental and the rollback instant. It is also what makes it possible to run both for a week and compare — reading from both and logging a difference is the cheapest possible verification and it caught two projection bugs before either reached a customer.

The flags are debt with a deadline, and the ticket to remove each one has to be created with it. On this migration four of them outlived their usefulness by months, which is the ordinary outcome and the reason the removal has to be scheduled rather than intended.

Verifying it worked

# the assertion the whole year was for
$ git log --oneline -1
  a3f9c11 add tax_scheme to orders
$ ./deploy.sh production
  deployed. billing was not consulted, notified or affected.

# and the grant that makes it structural
mysql> SHOW GRANTS FOR 'billing'@'10.%';
GRANT SELECT, INSERT, UPDATE ON `shop`.`invoices` TO ...
GRANT SELECT, INSERT, UPDATE ON `shop`.`payments` TO ...
-- no orders. no customers. it cannot regress.

$ SELECT NOW() - MAX(updated_at) FROM billing_customers;
0:00:04

A schema change deployed by one team alone is the outcome, and it is worth measuring as such rather than as a technical milestone — the metric that mattered was the number of cross-team release meetings, which went from roughly two a month to zero. The revoked grants are what stops it coming back, and they are the part most likely to be skipped because everything already works without them.

What this costs

A year, for a system that worked the whole time. That is the honest headline and it is why this rarely gets funded on its own — it has to be attached to something a business cares about, and on this occasion it was attached to a regulatory deadline that made the release coupling intolerable. Proposing it as a technical improvement would have failed, twice, as it had already.

The permanent cost is eventual consistency in the business rather than in the code. Somebody will ask what happens when a customer is renamed and the invoice generated four seconds later has the old name, and the answer — that it does, occasionally, under specific conditions — is a policy decision rather than a bug. Having that conversation explicitly is far better than the previous arrangement, where the two services were consistent and could not be released independently, and nobody had ever described that trade either.