The decommission was thorough. Nothing told the backup.

The alarm was correct, which is why it was confusing

An hourly backup job started failing. Same failure, every run: one named target out of eleven, the rest green.

My first instinct was to treat the alarm as the defect. It was not. The job was doing precisely what I built it to do — refusing to report success while a target marked critical was missing. The database it wanted to dump had been deliberately retired that morning, as part of a planned migration to a replacement secret store.

So the failing job was telling the truth. The service really was gone, and the job really could not back it up. What had gone stale was the list of things the job believed should exist.

That distinction matters more than it sounds. A monitor that fails when reality changes is not broken. It is doing the only useful thing a monitor can do: holding a claim about the world and complaining when the world stops matching. The work is not to quiet it. The work is to update the claim — deliberately, because updating it is the same motion as deciding the thing is really gone.

The decommission checklist ended at the thing being removed

What made this interesting is that the retirement had been done carefully. It was not a stray docker compose down.

There was a break-glass record with a restore procedure. There was an encrypted final archive with a passphrase held off the host. There was a zero-consumer verification — containers, processes, scheduled units, and cron all checked for anything still reading from the service. Prior backups of it were retained by explicit decision rather than swept up.

By the standard of "did we remove this safely and can we get it back," the work was genuinely good.

By the standard of "what else in this estate holds a belief about this service," it was silent. The retirement verified everything that read from the service. It did not enumerate what watched it.

Those are different dependency graphs, and only one of them was walked. A consumer inventory answers "will anything break when this stops serving?" A watcher inventory answers "will anything keep asserting this exists?" Backup targets, monitoring checks, config manifests, and dashboards all live in the second graph, and the second graph is the one that stays quiet until a scheduler wakes it up.

The soft signal I had already been ignoring

The same run surfaced something I had been walking past for longer.

The job archives a list of configuration files. When a listed file is absent it records the absence in the manifest and carries on — the archive still reports success. Reasonable on its own: one missing optional file should not fail a backup.

The effect over time is that a config manifest quietly becomes a graveyard. Two paths had disappeared with the retired service. A third belonged to a service removed some time earlier that I had not noticed at all, because the only evidence was a line in a JSON file that nothing ever read.

A missing-file list that never escalates is not a warning. It is a place where warnings go to be stored. If absence is normal, it should be declared normal and removed from the list; if it is not normal, it should be loud. Holding it permanently in between is how a manifest starts describing its own history instead of the system.

The migration moved the asset and left the protection behind

Removing the dead target fixed the failure in about a minute. The more useful finding came from asking the obvious follow-up: if the replacement is now the sole authority for every secret in the estate, what is backing it up?

The routine job: nothing. Not one line.

The replacement store had been stood up carefully, hardened, given a considered seal model, and integrated with the services that consume it. Its own configuration even documented the correct backup command in a comment. That command was never wired into anything. Snapshots existed, but they were taken by hand, as an attended ceremony after significant changes — encrypt, verify by restoring into a disposable instance, revoke the tokens, write a receipt.

The ceremony is excellent. The ceremony is also a person remembering.

This is the part I would generalize hardest. A migration is usually scoped as "move the asset and cut over the consumers." Both halves get tested, because both halves break loudly when wrong. The guardrails around the old asset — backup coverage, freshness expectations, restore drills, the alarm that fires when a snapshot goes stale — do not move themselves, and they fail silently, because a backup that is never taken produces no error at all.

For a period I can measure, the most important store on the host was protected only by my memory of when I last ran a command. The old service it replaced had been backed up hourly, automatically, for months.

I did not automate the snapshot, and that was the point

The obvious fix is to have the hourly job take the snapshot itself.

I decided against it. A snapshot of a secret store is a complete copy of every secret it holds, and taking one requires a privileged token. Automating it inside an unattended hourly job means that token has to live somewhere the job can reach, permanently. I would have closed a backup gap by creating a standing credential with rights over the entire secret estate — and made the job that holds it a much more interesting target than it was that morning.

So the job does not take snapshots. It checks them.

It finds the most recent encrypted snapshot, fails if it is older than a configured age, and verifies the artifact still matches the checksum recorded when it was written. It reads metadata and ciphertext only. It never decrypts, never holds a token, and never sees a secret value.

That converts the real risk — a manual ceremony silently going stale — into an ordinary loud failure on an ordinary day, without inventing a new privilege to do it. The checksum verification earns its place separately: it means bit-rot in the one artifact holding every secret surfaces on a Thursday, rather than during the restore where it is the only copy I have.

It is a smaller intervention than automating the snapshot, and I think it is the better one. The attended ceremony stays the only path that touches plaintext. The automation's entire job is to notice when the human has not run it.

The remediation controller was right to refuse, and could still say more

I have a bounded controller that runs when a backup unit fails. It has a short list of reversible repairs and it escalates everything else.

It handled this correctly. It classified the failure as an individual target failing, stated that re-running would not help until the underlying cause was fixed, named the failing target, and stopped. Exactly right: the correct repair was a deliberate edit to a target list, which is a human decision and one it is explicitly forbidden to make.

But "target X failed" is a diagnosis with a missing half. The same line appears whether the database crashed, the host ran out of descriptors, or the service was retired on purpose an hour earlier.

The distinguishing fact was already in the run manifest: the container was not merely stopped, it was absent. Nothing by that name existed to start. A stopped container suggests a crash and a restart. An absent one, for a target still marked critical, almost always means the service is gone and the target list is stale.

So the controller now reads that distinction out of the manifest and says the second thing plainly, including the recommendation not to restart anything. It gained no new authority and no new repair — it still escalates, it still edits nothing. It just names the cause instead of the symptom, which is most of what I want a triage step to do at three in the morning.

What I actually changed

A decommission now has to answer a second question. Not only "what reads from this?" but "what asserts that this exists?" — backup targets, config manifests, monitoring checks, freshness expectations. The consumer inventory was already rigorous. The watcher inventory did not exist, and it is the cheaper of the two to walk.

And a migration is not finished when the consumers cut over. It is finished when the replacement has at least the protection the original had. Moving a secret store and leaving its backup coverage behind is not a partial migration; for a while it is a downgrade, and it is a downgrade that reports nothing, because absent automation has no failure mode to observe.

The hourly job is green again. It is green with one fewer target, two new configuration files, and one check whose only purpose is to fail if I forget something.