server-2 went slow and then died, and the apps were innocent: reading CPU steal before blaming the application

A VPS hit load 93 on 4 cores with 66% steal time; the cause was the host, not the workloads, and the useful work was in the alerts and in how I read the numbers.

Context

server-2 is a 4-core VPS that runs several of my internal apps, Postgres and Redis. Prometheus, Alertmanager and the probes run on server-3 and watch the whole fleet. server-1, server-2 and server-3 are separate VMs at the same VPS provider.

What I saw

At 23:38 (Ho Chi Minh City time) on 2026-08-30 server-2 was barely usable:

• load average 93 on 4 cores
• top: st 66%, us only 11%, and about 7.8 GB of RAM free
• dmesg: soft lockup on CPU#1 and CPU#3, stuck for 24-35 seconds
• rsyslog's file output suspended, Postgres and journald in D state, a kworker flush thread piling up

From outside, Prometheus on server-3 reported five server-2 targets timing out and EndpointDown for eight domains. Tailscale ping to the node timed out and SSH returned "No route to host" on and off.

Finding the cause

The reflex on a slow node is to look for the application that is misbehaving. The top header already answered it. Two numbers matter together:

• us 11% means my own processes were using little CPU.
• st 66% means the hypervisor was running something else on my vCPUs for two thirds of the time.

Free RAM ruled out memory pressure. The D-state processes and the lockups are what a kernel looks like when its CPUs stop being scheduled, not evidence of a bad query.

The comparison that made it solid: server-1 and server-3, same provider, showed steal at 0% at the same moment. Only this VM was affected, which fits a noisy neighbour on its physical host. After roughly ten minutes it eased on its own (steal 27%, load 15).

There was nothing to fix inside the VM, so I did not restart services. I did two small things: I silenced the server-2 alerts in Alertmanager for 3 hours, and I removed a probe for a lab I had shut down on 27 August, which had been firing since then.

Later I looked at 15 days of data across all three nodes. Steal was not exclusive to server-2: on 27 August server-1 reached 59% and server-3 23% for about four hours, and on 30 August server-2 reached 66%. But 13 of the 15 days were at 0%. It was real, intermittent and rare, so I did not recommend moving server-2 to another provider.

Root cause

Not in my VM. At 03:23 on 31 August the situation got worse: steal turned into soft lockups and then a hard stop, with server-2 unreachable by public address, tailnet and SSH from two vantage points. I rebooted it from the provider's portal and it came back at 09:11 with steal at 0%.

The provider said another customer on the same host was under attack, that this had overloaded the node, and that it had been dealt with. They also offered to migrate the VPS to another node if it happened again.

One local casualty: Redis had a damaged AOF tail from the hard stop. redis-check-aof --fix removed the last 3.6 KB, and everything else came up healthy, with 0 targets down.

The fix

I could not fix the cause, but I fixed how the fleet reacts to it. One steal episode had produced 12 alerts, and the noise hid the one that mattered.

1. Add a host label to every exporter and blackbox target in the Prometheus config, so each check says which machine it lives on.
2. Add an Alertmanager inhibit rule: a node-exporter ServiceDown suppresses ServiceDown and EndpointDown alerts for the same host.
3. Test it with a synthetic alert. The right host was suppressed and a different host was not.

When server-2 went fully down at 03:23 the rule worked: 11 of 12 alerts were suppressed and one ServiceDown was delivered.

One gotcha: sed -i or a script that rewrites prometheus.yml changes the inode, and a single-file bind mount then keeps serving the old file even though reload returns 200. Prometheus needed a docker compose restart.

What stops it happening again

• The inhibit rule means a host-level event is one alert, not twelve.
• The playbook for the next time: check st and wa in top, compare with another node at the same provider, and only then touch services. If steal is high, the action is a provider ticket quoting their offer to migrate.
• It did recur on 2026-09-08, in two episodes of about 11 and 16 minutes, with 21 soft lockups in the second one. The recommendation was to send a migration ticket quoting the provider's own offer. I do not have a recorded outcome for that, so I make no claim about it.

What I learned

• Check st before blaming the application. If us is low while load is huge, the CPU is not yours.
• A low reading can still mislead. On 8 September the measured steal was only 5.3-8.9%, but node_exporter had lost scrapes for about 10 minutes at exactly the worst moments. The peak was never recorded. A gap in data during an incident is not evidence of a mild incident.
• Missing reports are not "up". My status page showed 100% for server-2 on a bad day because its agent skipped cycles instead of reporting down. I now treat Prometheus alerts and the node's own dmesg as the source of truth.
• Alerts need topology. Without a host label and an inhibit rule, monitoring turns one outage into a pile of noise.
• Not every provider fault looks the same. On 5 October a different failure hit several VMs in the same minute with no steal and no soft lockups: ping worked, but SSH accepted connections and never returned a banner. That pattern means look at the node, not the guest.