An unattended deploy script that refuses bad builds, swaps by rename and rolls back by itself

August 2, 2026deployment · bash · astro · strapi · reliability

After Jenkins stopped working for a client's Astro and Strapi site, I wrote a deploy script that checks the page count, stages the build while the old one serves, and rolls back on a failed health check.

Context

A client in New Zealand, a painting company, has a website built with Astro (the pages) and Strapi (the content). A new article is published every day by a script. The site is rendered at build time, so after an article is published, the site has to be rebuilt and deployed.

Jenkins had been broken since 2026-06-24, and I stopped relying on it. When I cleaned up the client's machine on 2026-08-27, its two Jenkins agents turned out not to have connected since the fleet moved to a new private network: the Jenkins server only accepts that network, and the client's machine is deliberately outside it. On 2026-08-02 I wrote build-and-deploy.sh to run right after the daily article, with nobody watching.

What I saw

This is a design post more than an outage, but a concrete danger drove the design. The Astro build needs Strapi to be reachable, because that is where it learns which pages exist. If Strapi is down, the build does not fail. It "succeeds" and writes only a handful of pages. A healthy build here is about 306 pages. A build without Strapi is under 10.

For a human running the command, that is obvious. For a script running unattended, a green exit code would copy a near-empty site over the real one.

The second problem showed up with the first version. It stopped the service, copied 59 MB of files into place, then started the service again. All of the copy time was downtime, and an SEO crawler was catching that window as 5xx errors and timeouts.

Finding the cause

Nothing here was a mystery, but there were choices:

Trust the build exit code? No: a "successful" build can be wrong, so the script checks the output instead.
What should the check measure? Counting index.html files in the build output is cheap and sees the failure directly.
Where is the time lost in the swap? In copying with the service stopped. A rename inside one filesystem is effectively instant.

I tested the guards rather than assuming them. I set the floor artificially high and confirmed the script refused and left production untouched. I pointed the health check at a URL that returns 404 and confirmed it detected the failure, restored the previous build to the same inode and left the older backup alone, with the site staying up throughout.

Root cause

The build tool's idea of success does not include "has the content". So the deploy step has to decide for itself that the artefact is fit to ship, before anything in production is touched.

The fix

The script has three stages and three exit codes: 0 deployed, 1 refused (site untouched), 2 rolled back.

1. Build and refuse.

MIN_PAGES=250          # a healthy build is ~306; a Strapi-less build is under 10
PAGES=$(find dist/client -name index.html | wc -l)
[ "$PAGES" -ge "$MIN_PAGES" ] || fail "only $PAGES pages (need $MIN_PAGES). Strapi probably unreachable — production left alone."

It also checks that both dist/client and dist/server exist.

2. Stage, then swap. The new build is copied next to production under a different name while the old one is still serving. Only then does the service stop:

cp -a "$SRC/dist/client" ./client.new
cp -a "$SRC/dist/server" ./server.new

sudo systemctl stop "$SERVICE"
mv client client.old
mv server server.old
mv client.new client
mv server.new server
sudo systemctl start "$SERVICE"

Because the renames happen on one filesystem, the outage shrinks to the service restart. I measured it at about 440 ms. This came in a second commit on the script, on 2026-08-16.

3. Health check and rollback. The first version slept six seconds and checked once. I changed it to poll every 250 ms for up to 40 tries, because the service is usually up in under a second and every wasted second is a second of errors if it is not. If the page does not return 200, the script stops the service, puts the old directories back and starts it again, then exits with code 2.

The previous build is kept for one cycle as client.bak and server.bak, so a manual rollback is still possible after a successful deploy.

The daily-article script calls the deploy only when an article was actually published, so quiet days touch nothing.

What stops it happening again

A Strapi outage during a build now ends in exit code 1 and a log line, not a stub site.
A bad deploy that passes the page count but does not serve is reverted automatically (exit 2).
Reporting is thin, and I would rather say so. The deploy script only prints. Its caller, daily-article.sh, appends that output to a log file and sends no alert on exit code 1 or 2. Because the caller runs with set -e, a refusal or rollback also stops it before the Facebook share, so the article is live but not shared. The log is the only record. An alert on those exit codes is the obvious next step.

What I learned

A tool saying "success" is not the same as the result being correct. For anything unattended, validate the output, not the exit code.
Pick a cheap, direct signal that matches the failure. A page count is crude, but it catches exactly the dangerous case.
Measure where the downtime really is. I assumed the restart was the cost; it was the copy. Doing the slow work before the stop shrank the outage to the restart alone.
Make the safe path the automatic one. A script that can only end as "deployed", "refused, nothing changed" or "rolled back" is easy to reason about when nobody is watching.
Test failure paths on purpose: a forced high floor and a fake 404 proved the guards.