The staging environment that was lying

The export feature was tested on staging four separate times over three weeks, worked every time, and failed in production on the first real use. Then it was fixed, tested on staging, and failed in production again. By the fourth cycle nobody trusted staging for anything, which is worse than not having it.

The symptom

# staging
$ time curl -s -o /dev/null /admin/exports/orders?year=2020
real	0m2.104s

# production
$ time curl -s -o /dev/null /admin/exports/orders?year=2020
real	0m30.002s
HTTP 504

$ mysql -Nse 'SELECT COUNT(*) FROM orders' # staging
8412
$ mysql -Nse 'SELECT COUNT(*) FROM orders' # production
412008

Fifty times the data is the headline difference and it was not the only one. Enumerating the rest took an afternoon and produced a list that surprised everybody, including the person who had built both environments.

Why it happens

Staging is built once and diverges continuously, because production changes are applied where they are needed and staging changes are applied where somebody remembers. Nothing compares them, so the divergence is invisible until it produces a false pass.

The fix

Enumerating the differences

found by comparing, not by remembering:

  data volume    8,412 orders vs 412,008
  memory_limit   512M vs 256M     ← nobody knew
  opcache        off vs on, with validate_timestamps=0
  MySQL          8.0.23 vs 8.0.19
  Redis          one instance, shared with dev
  TLS            a self-signed certificate
  cron           disabled since 2019
  queue workers  1 process vs 6
  the CDN        absent entirely

four of these can each independently cause a false pass.

The memory limit being higher on staging than production is the one that stings: a test that passes on staging and exhausts memory in production is a straightforwardly inverted signal, and it had been that way since somebody raised it to debug something in 2019.

A difference report, in the pipeline

// one endpoint, on every environment, behind auth
Route::get('/_env', fn (): array => [
    'php' => [
        'version'      => PHP_VERSION,
        'memory_limit' => ini_get('memory_limit'),
        'max_execution_time' => ini_get('max_execution_time'),
        'opcache'      => opcache_get_status(false)['opcache_enabled'] ?? false,
        'extensions'   => get_loaded_extensions(),
    ],
    'db'    => DB::selectOne('SELECT VERSION() v')->v,
    'redis' => Redis::info()['redis_version'],
    'queue' => config('queue.default'),
    'app'   => config('app.version'),
])->middleware('auth.internal');

# in CI, on every deploy to staging
diff <(curl -s "/_env" | jq -S .) 
     <(curl -s "/_env"    | jq -S .) > /tmp/env.diff || true

# an allowlist of legitimate differences; anything else fails
comm -13 config/expected-env-diff.txt /tmp/env.diff | tee /dev/stderr | 
  [ "$(wc -l)" -eq 0 ]

Allowlisting the legitimate differences rather than aiming for zero is what makes this maintainable — hostnames and credentials will differ, and a report that always fails is a report nobody reads. Everything else being a build failure is what stops the drift restarting.

It caught two more within a month: an extension enabled in production for a new feature and not on staging, and a PHP patch version that diverged because the images were built from a floating tag.

The dataset, which is the biggest lie

in increasing order of honesty and cost:

  a seeder             fast, reproducible, shaped like
                       whoever wrote it imagined
  a scaled generator   realistic volume, unrealistic
                       distribution — every customer has
                       exactly four orders
  anonymised copy      realistic volume AND shape, plus a
                       legal obligation and a real risk

what mattered was the DISTRIBUTION: one customer with
41,000 orders, which no generator would have produced.

The export failed on a customer with forty-one thousand orders, and no synthetic dataset would ever contain one because nobody writes a generator with that in mind. Realistic volume with a uniform distribution would have passed exactly as staging did.

-- applied DURING the copy, never after it
UPDATE customers SET
  name  = CONCAT('Customer ', id),
  email = CONCAT('c', id, '@example.invalid'),
  phone = NULL, address_line_1 = CONCAT(id, ' Test Street');

UPDATE payment_methods SET token = NULL, last_four = '0000';
TRUNCATE TABLE audit_log;
DELETE FROM users WHERE email NOT LIKE '%@ourcompany.example';

The obligations that come with it are real and are the reason this is a decision rather than a task: a copy of production data on a less-secured environment is the same data with weaker controls. The anonymisation has to be verified rather than trusted, and it has to run as part of the copy rather than afterwards — a window where the raw data sits on staging is the whole risk.

# verified, not trusted
$ mysql -Nse "SELECT COUNT(*) FROM customers WHERE email NOT LIKE
  '%@example.invalid' AND email NOT LIKE '%@ourcompany.example'"
0
$ mysql -Nse "SELECT COUNT(*) FROM payment_methods WHERE token IS NOT NULL"
0

# run as part of the refresh, so a schema change adding a
# personal-data column fails it rather than leaking

What staging cannot test, and admitting it

written down, in the deployment runbook:

  staging DOES cover: schema migrations, the happy path,
  integration with sandbox third parties, the admin UI,
  and anything data-volume sensitive (since the refresh)

  staging does NOT cover:
    concurrency — one user, no lock contention
    the CDN and its cache behaviour
    real third-party latency and failure
    real traffic distribution
    anything requiring production credentials

  → those are covered by a canary, or not at all.

Writing the list down is the part that changes behaviour, because “it passed on staging” stops being a sentence that ends a conversation. The items in the second column are what a canary deploy or a feature flag is for, and naming them makes the case for building those.

Verifying it worked

$ curl -s -o /dev/null -w '%{time_total}n' 
    "$STAGING/admin/exports/orders?year=2020"
30.001        # it now fails on staging too. that is the win.

$ ./bin/env-diff
2 differences, both allowlisted (hostname, cdn_enabled)

$ ./bin/staging-refresh --verify
copied 412,008 orders, anonymised 8,412 customers
verification: 0 unanonymised rows
elapsed: 22m04s

Reproducing the production failure on staging is the acceptance test for the whole exercise, and it is a strange thing to celebrate. The four false passes were the problem; a true failure is the fix.

Twenty-two minutes for a refresh is short enough to run weekly on a schedule, which is what keeps the volume and distribution current. A refresh that takes four hours becomes a quarterly event and the divergence starts again.

What this costs

A copy of production data living somewhere with weaker access controls, which is a genuine risk taken deliberately in exchange for a signal that is worth having. The anonymisation is the mitigation and it is only as good as its coverage — a new column holding personal data is anonymised only if somebody adds it to the script, which is why the verification query has to fail the refresh rather than report.

The environment is also more expensive to run: fifty times the data means fifty times the storage and a database that needs enough memory to be representative, which was most of the cost. A staging environment sized for a tenth of production tells you about a tenth of production, and paying for the honest version is the trade.

The difference report is the piece most likely to rot, because the allowlist grows every time somebody has a deadline. Reviewing it quarterly and deleting entries that are no longer justified is a chore with no visible benefit, and skipping it for a year returns the system to exactly where it started.