A canary needs a metric to decide on, agreed in advance

A canary release without a decision rule is a slower deploy — somebody watches a dashboard for ten minutes, sees nothing obvious and promotes it.

agreed BEFORE the deploy:

  metric      error rate on /checkout, canary vs stable
  threshold   canary > stable × 1.5
  window      10 minutes, minimum 500 requests
  action      automatic rollback, no discussion

without the minimum request count, a canary with 3 requests
and 1 error is a 33% error rate and a false rollback.

The minimum sample size is the part that is always missing and is what makes automated canary analysis usable rather than infuriating. Comparing against the stable version rather than an absolute threshold is the other half — a background error rate of 0.4% is fine and an absolute threshold of 0.1% rolls back every deploy. Writing the rule down before the release is what stops it being renegotiated while the canary is live.