The system had three tracing mechanisms: a vendor SDK in the PHP application, a different vendor in the compiled sidecar, and a correlation id in the logs that connected neither. Consolidating meant choosing a vendor, and choosing a vendor meant instrumenting against their SDK — which is the decision everybody wants to defer.
The symptom
a request through three components, traced three ways:
nginx $request_id, in the access log
PHP a vendor SDK, spans to their collector
sidecar a different vendor, written by somebody else
so "why was this request slow" requires three tools, two
logins and manual timestamp alignment.
the obvious fix — pick one vendor — puts their SDK in the
application code, which makes the next change a rewrite of
every instrumented call site.The lock-in is the reason this had been deferred for two years, and it is a real concern rather than an excuse — an application instrumented against a vendor SDK has that vendor in every file that creates a span.
Why it happens
Tracing requires instrumenting the code, and every vendor shipped their own SDK because there was no standard. The instrumentation is the expensive part and it is the part that is vendor-specific, which inverts the usual relationship where the expensive part is portable.
The fix
What the standard actually standardises
OpenTelemetry, as of 2022:
the API stable for traces. what the code imports.
the SDK the implementation. PHP: BETA.
OTLP the wire protocol. stable.
the conventions attribute names. stable-ish.
the collector stable, and the piece that matters most.
the API being separate from the SDK is what makes this
worth adopting before the SDK is stable: the code imports
the API, and the SDK is swappable.// the application imports the API, not an implementation
use OpenTelemetryAPITraceTracerInterface;
public function place(Basket $basket): Order
{
$span = $this->tracer->spanBuilder('order.place')
->setAttribute('basket.line_count', $basket->count())
->startSpan();
try {
return $this->doPlace($basket);
} finally {
$span->end();
}
}A no-op implementation of the API ships with it, so instrumented code runs with tracing entirely disabled and costs almost nothing. That means the instrumentation can be added before the SDK is trusted, which is the sequencing that makes a beta dependency acceptable.
The collector as the abstraction
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
batch: { timeout: 5s, send_batch_size: 512 }
memory_limiter: { check_interval: 1s, limit_mib: 400 }
tail_sampling:
decision_wait: 10s
policies:
- { name: errors, type: status_code,
status_code: { status_codes: [ERROR] } }
- { name: slow, type: latency, latency: { threshold_ms: 2000 } }
- { name: baseline, type: probabilistic,
probabilistic: { sampling_percentage: 1 } }
exporters: { otlp/vendor: { endpoint: "..." } }
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlp/vendor]The processor order is significant and reads as a list: the memory limiter must be first so it rejects before anything is buffered, and the batch processor last so it batches what survived sampling. That order wrong produces a collector that works and drops spans under load.
The vendor is one line in a configuration file, which is the property that resolved the two-year deferral — changing vendors is a redeploy of the collector rather than a change to the application. That is worth the operational cost of running a collector on its own.
Tail sampling is the other reason and is the one that could not be done in the application: the collector buffers spans until a trace completes, then decides. Keeping every error and everything over two seconds, plus a one per cent baseline, is a policy that head-based sampling cannot express because the decision has to be made before the request runs.
The buffering is memory and the collector becomes a component with its own capacity planning — decision_wait of ten seconds means ten seconds of spans held for every trace in flight. The memory limiter is not optional.
Context across a queue boundary, by hand
// no auto-instrumentation for this framework's queue in
// 2022, so the propagation is explicit
$carrier = [];
TraceContextPropagator::getInstance()->inject($carrier);
dispatch(new SyncOrder($order, traceContext: $carrier));
// and in the worker
$parent = TraceContextPropagator::getInstance()
->extract($this->traceContext);
$span = $tracer->spanBuilder('SyncOrder')
->setParent($parent)
->setSpanKind(SpanKind::KIND_CONSUMER)
->startSpan();The W3C traceparent format carries the sampling decision as well as the identifiers, so a trace sampled at the edge stays sampled through the worker — which is the property that makes an end-to-end trace possible rather than a collection of fragments. Carrying a bare trace id loses it.
A middleware doing this for every job is the only version that survives, because a convention applied by hand is forgotten on the next job class. It is about forty lines and it is the single highest-value piece of instrumentation in the whole exercise.
Instrumenting a framework with no auto-instrumentation
written by hand, in 2022:
the HTTP kernel a middleware, 30 lines
the queue dispatch and handle, 40 lines
the database a query listener, 25 lines
the HTTP client a Guzzle middleware, 20 lines
the cache skipped — 90% of the span volume
about 120 lines. the semantic conventions are the part to
get right: a vendor's UI groups on db.system and
http.status_code, and a custom name is invisible.Following the semantic conventions is what makes a vendor’s dashboards work without configuration, and it is the least interesting part of the work to get right. Using db.statement rather than query is the difference between a database view that populates itself and one that requires a custom query.
Skipping the cache instrumentation was a deliberate decision after measuring: cache spans were ninety per cent of the volume and were almost never the answer to a question. That is the sort of decision that has to be made per component and is easier to reverse than to make in advance.
Verifying it worked
$ ./bin/trace-smoke
trace 9c1f4a7e3b2d4f81a6e011d0c8b3f204:
order.place 412ms php
├─ db.query 88ms php
├─ http.client 104ms php → pricing
│ └─ price.calculate 96ms sidecar
└─ queue.dispatch 2ms php
└─ SyncOrder 1.2s worker
# one trace, three components, two runtimes, one tool.The end-to-end trace crossing both the HTTP boundary and the queue boundary is the acceptance test, and it is worth building as a smoke command rather than checking by hand — a propagation change that breaks the chain is otherwise discovered during an incident.
What this costs
A beta dependency on the request path, which is the thing to be honest about. The SDK had two releases during the quarter that changed a public interface, and both required a small change to the instrumentation — pinned to an exact version, with the upgrade scheduled rather than automatic, which is not how a dependency should have to be managed.
The collector is also a new component with its own failure modes: it buffers, it can fall behind, and a memory limit reached means spans dropped silently. Monitoring the collector itself is a small piece of work that is easy to omit and produces a tracing system that appears to be working and is losing data.
The honest summary is that adopting this in 2022 is early, and it is early in a way that is recoverable — the API is stable, the instrumentation is the expensive part, and a beta SDK behind a stable API is a much better position than a stable SDK from a vendor.