Sampling traces at five per cent, keeping every error

Full tracing on a service doing forty thousand requests a day is a data volume problem; sampling uniformly loses the requests worth looking at.

the policy:

  a request that errored          keep, always
  a request slower than 1s        keep, always
  a request with a debug header   keep
  everything else                 5%

which requires a TAIL sampler — the decision is made
after the trace completes, so the whole trace has to be
buffered somewhere.

Head sampling decides at the start and cannot know the request will fail, which is why uniform sampling throws away exactly the traces that would have been useful. Tail sampling needs a collector holding spans until the trace ends, which is real infrastructure and is the cost of the policy. On a smaller service, keeping everything for seven days is simpler and cheaper.