A 22.8 MB homepage crash-looped my job crawler for almost 6 hours

One oversized company homepage ran Job Radar's crawler out of memory, and because the task came back after every restart, almost nothing was collected for nearly 6 hours and nothing alerted.

Context

Job Radar is a job-search crawler I run for myself. A Node.js crawler pulls postings and company career pages into Postgres through a work queue, and a separate web container shows the results at jobs.docaohieu.com. Both run under Docker Compose on one ARM node. Monitoring (Prometheus, Alertmanager, Loki) lives on a different node.

The date was 24 October 2026.

What I saw

I found it by accident at 17:24, local time. I had just finished a rollback drill for a deploy tool and ran docker ps. The crawler container said Up 5 seconds. It should have been up for days.

The container logs showed the cause:

FATAL ERROR ... JavaScript heap out of memory

The web site was fine the whole time. Only collection had stopped. No alert had fired.

Finding the cause

I went from symptom to data in about three minutes, using three sources.

• Logs. A heap out-of-memory error, not a network or database error. So some input was too big.
• The queue table. I looked for the task that was locked at the moment of each death. It was always the same one: a periodic career-discover task for one company website. Its attempts counter had reached 673.
• Loki and Prometheus. Loki, the log store on the monitoring node, shows the repeated fatal errors starting at 11:28 and stopping at 17:26, with one every 30 seconds or so: 54 to 57 in every half hour. That is almost 6 hours and roughly 110 restarts an hour, which fits the task's 673 attempts. Prometheus agrees on the start. The container's CPU counter resets on every restart, and it first shows several resets in the 15-minute window that begins at 11:30.

The homepage of that company site was 22.8 MB of HTML. The crawler read and parsed all of it; the heap reached about 636 MB and Node died.

One detour: a canary deploy at 17:14 had recreated the container and reset Docker's restart count, so docker inspect understated the loop. The metrics were the reliable record.

One thing stays unexplained. For about 14 hours before the loop, from 21:30 on 23 October, the container had been restarting once or twice an hour. In Loki that is 21 heap out-of-memory crashes with the same signature (the GC trace stops at about 636 to 639 MB) and one crash where the database server closed the connection. I did not trace which tasks were running, so I cannot say whether the same page was behind them. There have been no heap crashes since the fix.

I stopped the bleeding first, with a database backup and then a one-line change that pushed that task back a day. The crawler held steady at 75 MB.

Root cause

Two separate problems stacked:

1. fetchText read the whole response body into memory with no size limit.
2. A task that kills the process is claimed again after every restart. At startup the crawler releases all stale locks, so the same task was first in line again. Recurring tasks were designed never to die: after the last attempt they rest six hours. But a crash does not go through the failure handler, so that six-hour rest never ran.

The fix

Refuse large bodies while streaming. The reader stops as soon as the total passes 5 MB, and a declared Content-Length over the cap is refused before reading. No job source needs more than that.

export const MAX_BODY_BYTES = 5 * 1024 * 1024;
...
    size += value.byteLength;
    if (size > MAX_BODY_BYTES) {
      await reader.cancel();
      throw new ResponseTooLargeError(url);
    }

This went out the same evening through the canary, with three tests.

Quarantine a task that keeps getting cut off. Releasing a stale lock now counts as an interruption. On the third interruption in a row the task rests for a day, and finishing normally resets the count:

next_fetch_at = CASE WHEN interrupted + 1 >= $2 THEN now() + interval '1 day' ELSE next_fetch_at END,

This needed a new column (migration 018), so the canary controller refused it; I took a backup and deployed with its skip flag.

What stops it happening again

• The 5 MB cap, applied to robots.txt reads too.
• Quarantine, so one bad input costs at most three restarts a day, not a loop.
• A ContainerCrashLooping alert on the monitoring node. It counts counter resets, the signal that survives restarts:

sum by (instance, name) (resets(container_cpu_usage_seconds_total{name!="",name!~"lab-.*"}[15m])) >= 3

The rule has no for: delay, so it fires on the first evaluation that sees three resets. I did not stage a second crash loop to test it. Instead I replayed the expression against the stored metrics from 24 October. It returns 4 to 8 in every 15-minute window of the loop (cAdvisor's counter catches fewer resets than the roughly 28 crashes per 15 minutes in the logs, but still above the threshold) and 0 after it ended. On that data it would have fired within about 15 minutes of the loop starting. It has not fired since. The repo documents the cap, the quarantine and the alert under "Failure containment" in the README.

What I learned

• A health check on the container is not a health check on the work. ContainerDown only fires when a container disappears for a few minutes. A container that dies and comes back every 30 seconds is always "seen". My SLO measured the web page, not whether crawling happens. A crawler freshness check (completed tasks per hour above zero) was proposed in the postmortem but is not built.
• At-least-once queues need a poison-message rule. Retrying forever is fine until one input is lethal.
• Cap untrusted input at the edge. Anything fetched from the internet needs a size limit before it is buffered.