Two months of silent failure in a social posting pipeline

August 5, 2026observability · alerting · automation · incident-review · cron

A shared API key was restricted, two posting bots kept running without errors anyone saw, and a client's social channels went quiet or repeated themselves for about two months.

Context

I look after the website and SEO for a client in New Zealand, a painting company. As part of that I had two small bots on one server, both started from the owner's crontab on New Zealand time:

A Google Business Profile poster (since May 2026). It drove a browser to click "Add post", wrote the caption with Gemini Vision from the photo, and kept state in SQLite. It ran twice a week with a random delay of up to two hours.
A Facebook Page poster (since June 2026). It used the Graph API, took posts from a queue.json file, and ran a generator script that asked Gemini for new posts whenever the queue dropped below four. It ran 14 times a week.

Both read the same Gemini API key. Neither one sent me anything when it failed.

What I saw

On 14 October 2026 I looked at the bots for an unrelated reason and found that nothing had gone wrong in the places I would have noticed, and a lot had gone wrong everywhere else:

Facebook had not posted anything since 6 August. The queue was empty: 83 posts marked posted, 23 skipped, and 141 logged 403 responses.
The Google Business Profile had 19 posts, from early August onward, that all carried the same caption. They included photos of interiors, spray work and plastering, under a caption that began "Another exterior repaint finished...".
Roughly a quarter of Google Business runs had failed outright: 12 failed out of 51, with an "Add post" click blocked by an overlapping DIV. The code did not retry.

So the client's page had been silent for two months and their business profile had been repeating itself for just as long. No alert, no email, no log anyone read.

Finding the cause

I started from the 403s because there were 141 of them in one place. They all came from the Gemini call, and the response body said API_KEY_HTTP_REFERRER_BLOCKED. That error means the key has a website (HTTP referrer) restriction, and requests from a server have no referrer, so they are refused. The shared key had been restricted to websites in early August, which is also when the identical Google Business captions begin.

That explained Facebook: the generator could not produce posts, so the queue drained and was never refilled. The poster itself was healthy, it just had nothing to send.

It did not obviously explain the Google Business duplicates, because that poster kept posting. Reading the caption function showed why. When the Gemini call raised an error, the code caught it and returned a hard-coded fallback caption, the "Another exterior repaint finished..." text. The post still went out, with a green status, and the fallback was identical every time.

Root cause

Three things had to be true for this to last two months:

A key shared by two bots was restricted without anyone checking what used it.
One bot failed by running out of work (a queue draining), which is not an error anywhere.
The other bot turned an error into a plausible-looking success by falling back to fixed text.

There was no monitoring of outcomes. The only signal I would have had was an exception, and the fallback code removed the exception.

The fix

I did not repair the two bots. I replaced them with a self-hosted Postiz scheduler (a public open-source tool) that posts through the official APIs: the Google Business Profile API and the Facebook Pages API, rather than clicking through a browser.

Content generation moved into scheduled jobs that I control. The Facebook post is built from each new blog article and scheduled for a morning slot. The Google Business post uses a real project photo from the profile and a caption written by a model that must look at the picture. If the generated caption fails my checks (no links, phone numbers, hashtags or emoji, and none of the old canned text), it retries up to three times and then skips that run and alerts me instead of posting anything.

I also removed the damage. I deleted all 19 duplicate Google Business posts through the API, saved their IDs to a file, and left the other 25 posts on the profile untouched. The old bot directories and cron entries were removed after the new path had published real posts.

What stops it happening again

Every scheduled job reports failure to a chat alert, including "I skipped this run".
The Facebook job checks after the fact that the post it scheduled actually exists in the scheduler, not just that the call returned.
There is no static fallback text any more. If the model cannot write a good caption, nothing is posted.
The gap that remains is a job that never starts at all. It would report nothing, so a check for "nothing published in N days" is the next alert to add.

What I learned

Alert on silence, not only on errors. A bot that posts nothing and a bot that posts the wrong thing both exit cleanly. The useful questions are "did the expected thing happen in the expected window?" and "did we take a fallback path?".

A fallback that hides a failure is worse than the failure. I now treat any except: return default in an output path as a bug unless the default is clearly marked, counted and alerted on.

Shared credentials need an owner and a list of consumers. Restricting one key broke two systems and told neither.