Traefik stopped seeing new containers after Docker restarted

September 29, 2026docker · traefik · systemd · observability · live-restore

A restart of Docker's socket left containers holding a stale bind mount, so Traefik returned 404 for new routes and monitoring agents went blind; a small self-healing script on five nodes fixed it.

Context

Traefik discovers routes from Docker labels through a socket-proxy container, which reads the Docker socket. Monitoring and log agents (a metrics agent, a log viewer agent, a log shipper) and a CI runner read the same socket. On most of my nodes the Docker daemon runs with live-restore, so containers keep running when the daemon restarts.

What I saw

On 5 September, a new container on one node had no route and Traefik answered 404.
Between 18 and 29 September, on another node, new containers (including the relay for one of my products) again had no route.
The socket-proxy answered 503 with termination state SC-- for its /_ping health check.
At the same time, the metrics agent and log agent on two nodes had silently stopped seeing Docker, since 18 September. Nobody was alerted.

Finding the cause

The 404 looks like a labelling mistake, and that is where I started. The labels were fine. Asking Traefik directly through its API (/api/http/routers) showed no routers from the Docker provider at all, so Traefik had lost the provider, not the individual route.

The socket-proxy's 503 pointed at the socket. The date 18 September matched an apt upgrade of Docker across the fleet. A package upgrade restarts docker.socket, and that creates /run/docker.sock again with a new inode.

With live-restore, running containers survive, but a container that bind-mounted the single socket file still holds the old inode. Every Docker API call from it fails. The nodes without live-restore were not hit, because their containers were recreated on restart. This is the same family of problem as a sed -i replacing a file that is bind-mounted into a container.

I confirmed it by reproducing it on one node: stop and start docker.socket, watch the inode change (544871 to 1014638 in that test), and watch the socket-proxy, metrics and log agents lose the connection.

Root cause

A bind mount of a single file follows the inode, not the path. When the Docker socket is recreated, long-lived containers that mounted it keep pointing at a deleted file, and nothing tells them.

The fix

On 29 September I added a script, docker-sock-rebind, and installed it on five nodes. It compares the inode of the host's /run/docker.sock with the inode each running container actually sees at its mount point (through /proc/<pid>/root), and restarts containers that differ. It supports --dry-run, logs to the journal, and always exits 0 so it can never fail docker.service.

Two systemd units run it:

# docker-sock-rebind.service (oneshot, WantedBy=docker.service, after docker.service)
ExecStartPre=/bin/sleep 5
ExecStart=/usr/local/sbin/docker-sock-rebind

# docker-sock-rebind.timer
OnBootSec=5min
OnUnitActiveSec=15min

The service runs after every Docker start; the timer re-checks every 15 minutes. In the test, the socket-proxy, metrics and log containers restarted themselves within about 10 seconds, Traefik never returned a 503, and the relay and storage endpoints stayed at 200.

A quick manual check: if a new container has no route, look for a Docker-provider router in Traefik's API, then run docker-sock-rebind --dry-run.

What stops it happening again

The timer catches cases the post-start hook misses. And on 18 October I found the script had a hole. It resolved the mount path inside /proc/<pid>/root, but in most images /var/run is an absolute symlink to /run, and an absolute symlink inside that path resolves against the host's root. So the check was comparing the host's socket with itself and never found a stale mount. The CI runner on one node had held a stale socket since 15 October, and every job failed with runner_system_failure.

The fix: if /proc/<pid>/root/var/run is a symlink, map /var/run/... to /run/... before taking the inode. I did not use nsenter ... stat, because distroless images have no stat and the script would restart those containers forever. Images built from scratch have a real /var/run directory, so they are not remapped. I patched four nodes. The fifth, which does not run live-restore, was not patched and does not need it, because its containers are recreated whenever Docker restarts.

What I learned

When a check always passes, test it against a known-bad case. My first version was tested with a real stop/start, but the symlink case only showed up because a runner failed in production a month later.

"New container has no route" is not always a label problem. Check whether the provider is alive before editing labels.

The remaining gap is visibility. When docker-sock-rebind restarts a container it writes a line to the journal under its own tag, but there is no alert rule on it, so I would find out by reading the log or by the fact that things recovered. An alert on that tag is the next thing to add.

Agents that go blind do not complain. I now expect socket-dependent containers to need an explicit liveness check, which is why the monitoring gap mattered as much as the 404.