What a home-made canary controller caught on its first day

October 24, 2026canary · traefik · docker-compose · sre · deployment

Fleet Pulse's canary controller for Docker Compose and Traefik blocked three Job Radar releases on the day I built it, one deliberate failure and two real flaws in my own setup.

Context

My servers run Docker Compose and PM2 behind Traefik, not Kubernetes. Flagger and Argo Rollouts both need Kubernetes, so I wrote a small canary controller as part of Fleet Pulse, a reliability project (SLOs, deploys, incident records) that I built on 24 October 2026. It is about 500 lines of Python.

The first app on it was Job Radar. A deploy works like this:

Start the new version as a second container next to the stable one.
Check that Traefik can actually reach it.
Write a Traefik weighted route that sends 10% of requests to the canary.
Watch for 10 minutes, checking every 30 seconds, and compare the canary with the stable version over the same window.
If it is no worse, move 100% to the new version. If it is worse, remove the canary. Production never changed.

The rule for "worse" is in analysis.py: canary error ratio must stay under max(1%, 2 x stable), and p95 latency under 1.5 x stable + 50 ms. Comparing with stable at the same moment matters, because if a shared database is down both versions look bad and the new one should not take the blame.

Job Radar is a one-person tool, so it has almost no real traffic. The controller sends its own synthetic requests (3 per second to /health, / and /api/state, read-only paths) so there is something to measure.

What I saw

The deploy log is a JSON line per event. Three releases were stopped between 17:15 and 17:43 that afternoon:

rolled_back  canary 5xx 30.30% > allowed 1.00% (stable 0.00%)
rolled_back  only 37 canary requests, need 150 to judge
rolled_back  canary healthy but not reachable through Traefik

Finding the cause

Only one of these was a bad release in the usual sense.

The 30.30%. This was the planned drill. I built a Job Radar image where /api/state returns 500 and pushed it through. The canary served 156 requests and 47 failed, against 0 errors on stable. The controller removed it after about 8 minutes (17:15:55 to 17:24:09) and the running version never changed. That is the test I wanted to see pass.

"Only 37 requests." The third attempt was meant to be a good build. The canary was healthy and the error count showed 0%, yet the controller refused to promote it, because it had received 37 requests where 150 were needed. The stable version had seen 350 in the same time, and the canary's p95 latency was 5.0 s. The Traefik access log for the same ten minutes shows what happened: 38 requests to the canary logged as HTTP 499 (the client giving up) and one as 504, none successful. My analyzer only counted 5xx as errors, so a canary that answered nothing looked perfect, and only the minimum-sample rule stopped it. Without that rule, a canary that serves nothing would have been promoted with "0% errors".

"Healthy but not reachable." The fix for that was a check that calls the canary through Traefik, the same path real traffic takes, before any traffic is shifted. It fired on the next run and showed the root cause. I had attached the canary container to Traefik's network after creating it, so Traefik had registered an address on a network it could not reach. The container passed its own health check from inside Docker and was unreachable from the proxy.

I have no separate trace of why the third attempt got so few answers. The routability check failed on the very next run, and the same code was promoted once the network fix was in, so I treat it as the same defect.

So the honest summary: one drill, and two flaws in my own canary setup that the safeguards refused to wave through.

Root cause

Two causes behind the two real failures:

The canary was connected to the proxy network after creation, so the route pointed at an address Traefik could not use.
The analyzer counted only 5xx as failures. A 499 means the client abandoned the request, and from the user's side it is a failure.

The fix

Create the canary on the proxy network from the start.
Count 499 as an error.
Add wait_routable: a 200 through Traefik on the canary's own hostname before any weight is written.

After that, the same afternoon, a good build was promoted: canary 191 req, 5xx 0.00%, p95 266ms vs stable 235ms.

What stops it happening again

The controller has since promoted seven releases (three on 24 October, four on 25 October), each with 0 canary errors. All seven were clean: canary p95 ran from 246 to 400 ms against 235 to 369 ms for stable, always inside the limit, and none was flagged. I cannot claim it caught a real problem in any of them. An independent review of the code later that day found three high-severity gaps in the failure paths, for example a SIGTERM leaving 10% of traffic stuck on a canary. All were fixed with tests. The review also led to an early exit: a canary with at least 20 requests and half of them failing is removed at once instead of after the full wait.

One rule is deliberate: a release with a database migration is refused by the canary, because both versions share one database and a rollback would not be clean. Migration 018 was refused on 24 October (new migrations: ['018_task_interruptions.sql']). I took a backup and deployed with --skip-canary.

What I learned

A passing health check is not the same as being reachable. Test the path real traffic takes.
"Zero errors" with almost no traffic is a missing signal. The minimum-sample rule was the only thing between me and promoting a canary that served nothing.
Count client-abandoned requests as failures.
Test the failure paths. Most of the code that matters runs only when something is wrong.