Building a reliable agent taught me not to trust the word 'success'

I built a small agent to dogfood my own job board. It pulls new agent-engineering roles from buildagentic.ai, has an LLM pick the relevant ones, and posts a digest to a Telegram channel on a schedule. Simple on paper: fetch, filter, post.

I didn't want a demo. I wanted something that runs unattended and doesn't quietly rot. That turned out to be the hard part — and every hard part was the same shape: something reported success while being broken.

Four of them, in order.

1. The workflow that "ran" but didn't. GitHub Actions reported "no jobs were run" — no error, just nothing. The cause: a ${{ }} expression inside a shell comment in my workflow file. GitHub's expression parser scans the whole run body and doesn't honor #, so it tried to evaluate a comment as code and rejected the entire file before any job started. The comment explaining a rule was the thing breaking the rule. No error surfaced. The job simply never existed.

2. The LLM that returned garbage under a strict schema. I constrained the model's output with a JSON Schema, felt safe, and moved on. Then a run failed on "no parseable JSON." A schema shapes a well-formed reply — it does nothing about an empty string, a refusal, or prose where structure should be. Structure is not truthfulness. I added retries with a repair instruction, and a short-circuit on the errors that don't self-heal. The schema made me feel covered. It wasn't.

3. The alert that said "delivered: true" — and delivered nothing. My pipeline's health check reported delivered: true every day. It was lying. The code returned true when there were zero alerts to send — "nothing attempted" and "sent successfully" had collapsed into the same value. My alert webhook had been dead for six days, and every health run cheerfully reported green. The system I built to catch silent failures was itself failing silently. I split the states — nothing to send, delivered, rejected — so "true" means one thing, and made a delivery failure loud, because the one thing a broken alerter can't do is alert you that it's broken.

4. The credentials that "synced." A rotated key that was live in the env-var store but not in the running build. A subscriber the API accepted as "synced" but who was never actually subscribed. "The POST returned 200" kept getting mistaken for "the thing happened."

The pattern across all four: a green light is only meaningful if the code path that would turn it red actually executed. "No error" often means "the failing branch was never reached." A 200 means the server received bytes, not that your intent came true. "Delivered: true" meant nobody checked.

Building a demo agent, you never hit this — you're watching it run. Building a reliable one, the whole game is distrust: assume every success signal is lying until you've made it prove otherwise, and make failure impossible to miss. The agent itself is maybe 200 lines. The reliability is most of the work, and none of it shows in the happy path.

It runs every Tuesday now, posts a clean digest, and costs a fifth of a cent a run. And I trust it — not because it says it works, but because I made it prove it does.

The full code is on GitHub: github.com/taboxdev/telegram-role-alerts — clone it, point BOARD_URL at your own feed, and it's yours.