The error tracker had four hundred thousand events for the month and a browser tab that nobody had opened since August. A customer reported a checkout failure that had been in there since the third, at position eleven thousand, and that is the entire problem — the tool exists to prevent exactly what happened and had become the reason it happened.
The symptom
# events for the month, by issue, top five
188,412 ErrorException: Undefined index: utm_source
94,110 ClientException: 404 Not Found (GET /api/products/{id})
41,882 ValidationException: The given data was invalid
22,004 TokenMismatchException
14,201 ModelNotFoundException: No query results for [AppOrder]
3 PDOException: SQLSTATE[40001]: Serialization failure
1 TypeError: Argument 2 passed to Checkout::take() ← the oneThe top five are 360,000 events and none of them is an error. A missing query parameter, a 404 from an API that returns 404 for missing products, a validation failure, an expired CSRF token from somebody’s open tab, and a model lookup for a deleted record. All expected, all reported as exceptions, all drowning the two that matter.
Why it happens
The default handler reports every uncaught exception, and PHP applications throw exceptions for things that are not errors. A validation failure is a normal outcome of a form; an expired token is a normal outcome of leaving a tab open; a 404 from an upstream is frequently the correct answer to the question asked.
The distinction that is missing is between an exception — a control flow mechanism — and an error, which is something a person should look at. Nothing in the language or the framework makes that distinction, so it has to be made deliberately, once, and then maintained.
The fix
What is an error, what is an event, and what is neither
ERROR a person should look at this today
type errors, database failures, an upstream 500,
a queue job that failed its retries
EVENT worth counting, not worth reporting
a declined card, a rate limit hit, a login failure
NEITHER a normal outcome that happens to be an exception
validation failures, expired tokens, honest 404s
the test: if this fires 10,000 times, do I want 10,000
notifications, one number on a graph, or nothing at all?That last question is the whole classification and it can be answered per exception class in about twenty minutes for a mature codebase. The answer for most of them is “nothing at all”, which is uncomfortable to write down and is correct.
// app/Exceptions/Handler.php
protected $dontReport = [
ValidationException::class,
TokenMismatchException::class,
ModelNotFoundException::class,
AuthenticationException::class,
HttpResponseException::class,
];
public function report(Throwable $e)
{
// an event: counted, not reported
if ($e instanceof PaymentDeclined) {
Metrics::increment('payment.declined', ['reason' => $e->reason()]);
return;
}
parent::report($e);
}The report() method on the exception itself is the other place to put this, and it is better for anything domain-specific: a PaymentDeclined that knows it is not an engineering event keeps that decision next to the thing that throws it rather than in a growing list in the handler.
Grouping, and the one exception that appears as four hundred
// every one of these is a separate issue, because the message differs
throw new RuntimeException("Failed to fetch product {$id} from upstream");
// one issue, with the id as context
throw new UpstreamFailure('product.fetch_failed', ['product_id' => $id]);
// or, where the message must stay, tell the tracker how to group
SentryconfigureScope(function (Scope $scope) use ($id) {
$scope->setFingerprint(['upstream', 'product.fetch']);
$scope->setContext('request', ['product_id' => $id]);
});Interpolating a variable into an exception message produces a distinct message per occurrence, and every tracker groups on the message by default — so one bug appears as four hundred issues, each with one event, none of which crosses any threshold. That is how a real problem hides in a noisy tracker.
The fingerprint is the escape hatch for third-party exceptions whose messages you do not control, and it is worth setting for the two or three that dominate the issue list. Everything else is fixed by not putting values in messages, which is the same discipline as structured logging and for the same reason.
Release tracking, so a spike has a cause
// config/sentry.php
return [
'dsn' => env('SENTRY_LARAVEL_DSN'),
'release' => env('APP_RELEASE'), // the deployed SHA
'environment' => env('APP_ENV'),
'traces_sample_rate' => 0.0,
];
// and in the deploy script, after the symlink swap
// sentry-cli releases new "$SHA"
// sentry-cli releases set-commits "$SHA" --auto
// sentry-cli releases deploys "$SHA" new -e productionA new issue tagged with the release that introduced it answers the first question of every investigation without anybody having to correlate timestamps against a deploy log. The commit association goes further and suggests which change is responsible, which is right often enough to be worth the two lines in the deploy script.
It also enables the useful inverse: an issue that stops occurring after a release can be resolved automatically, which keeps the issue list from accumulating things that were fixed incidentally.
Alerting on the rate of new, not the volume of all
does not work: "more than 100 errors in 5 minutes"
fires on any traffic spike. muted within a week.
works:
a NEW issue appears in production → notify, always
an issue is seen by more than 50 users → notify
an issue regresses after being resolved → notify
the error RATE per request doubles → notifyA new issue in production is the highest-value signal available and it is worth notifying on unconditionally, because a first occurrence is either a new bug or a new code path — both of which somebody wants to know about. That is a manageable number of notifications precisely because the classification work has already removed the noise.
The rate per request rather than the absolute count is what makes an alert survive a traffic spike. Errors going up because requests went up is not an incident; errors per request doubling is.
Verifying it worked
# the following month
events: 400,112 → 1,884
issues: 892 → 41
unresolved: 892 → 6
median age of an unresolved issue: 34 days → 1 day
# and the check that the real ones still arrive
$ php artisan tinker
>>> throw new RuntimeException('deliberate, ' . uniqid());
# in the tracker within 4 seconds, tagged with the release
$ php artisan queue:work --once # with a job that throws
# reported, with the job class and the payload as contextThe median age of an unresolved issue is the number that says whether this worked, and it is a better measure than the event count — a tracker with 1,884 events that nobody reads is the same failure at a smaller scale. Thirty-four days to one day means people are opening it.
Throwing something deliberately after any change to the handler is worth doing every time. It is very easy to silence more than intended: a dontReport entry for a base class silences every subclass, and the way that presents is a tracker that is admirably quiet.
What this costs
A judgement call per exception, forever. Every new exception class needs somebody to decide whether it is an error, an event or neither, and there is no default that is right — reporting everything produces the situation this started with, and reporting nothing is worse. Putting the decision in the exception class itself, as a report() method, is what keeps it next to the code rather than in a list somebody maintains.
The real risk is silencing something that mattered. ModelNotFoundException is noise when it comes from a route model binding for a deleted record and is a genuine bug when it comes from a queue job looking up something that should exist — and the class is the same. Where that distinction matters, the answer is a domain exception thrown deliberately rather than a framework one filtered globally, which is more code and is the only version that is actually correct.