Bounded Autonomy
The backup job can repair one thing. It should know when to stop.
A recovery controller is useful when it refuses an untrustworthy backup, makes a few reversible repairs, and leaves consequential recovery decisions to a person.
A completed backup is not necessarily a recoverable one
It is easy to make a backup report look reassuring. A scheduler ran. An archive exists. A repository accepted new bytes. The green check has a timestamp.
None of that proves the data needed for recovery made it into the archive in a usable form.
I learned to treat the backup as a chain of claims rather than one job. A database has to produce a native dump. The dump has to be readable. The archive has to contain it. A restore has to create a working isolated copy. And the operator needs an answer when any step fails.
That is why the first useful success state in my backup system is sometimes an intentional failure: do not create a backup that silently omits data the system says it protects.
The dump gate is an assertion, not a convenience
The controller gathers native database exports before it starts the wider system backup. Filesystem-level copies of a running database can be incomplete or internally inconsistent even when the copy command exits successfully. The native export is a separate, testable recovery artifact.
The system backup is gated on those exports. If a required dump fails, the archive job fails too. It does not proceed with a partial result labelled as a successful backup.
That rule caught a real integration mistake. A database container had moved its credential delivery from an ordinary environment value to a file-based convention. The dump helper could no longer find what it required. The right outcome was not a best-effort archive with that database missing; it was a visible failed run and a focused repair to the credential-reading boundary.
The relevant question was not “did the backup copy files?” It was “can this backup still make the promised database recoverable?”
Keep copies separate, then test the one that matters
My operating rule is the familiar three-copy, two-media, one-offsite shape. The important qualification is that a disk attached to the machine is not an offsite copy just because it is a separate drive. A building-level loss, theft, or destructive host event can take both at once.
The primary backup target holds the first repository; a separate object-storage repository holds the offsite copy. The offsite repository is the meaningful restore source for a location-loss scenario, so the restore test uses it rather than congratulating the local repository for being nearby and readable.
The test restores a real database export into a throwaway database service, checks the export stream, and verifies that expected data structures and rows come back. It is deliberately narrower than a full disaster exercise, but it proves more than archive presence.
NIST's contingency-planning guidance makes the same distinction: recovery is not complete until the recovered system has been validated before it returns to normal operation. NIST SP 800-34 Rev. 1 discusses recovery and reconstitution validation.
Automation gets a short list, not a blank cheque
A failed backup can have mundane causes: a dependent container did not start, a backup drive that should be mounted is absent, or a transient unit failure needs one clean retry. Those are reasonable candidates for narrow automation.
My remediation controller has exactly three permitted actions:
- start a named container from a fixed allowlist;
- mount the expected backup target when its declared configuration says it should be mounted; or
- re-run the failed unit once.
There is one remediation cycle per affected unit in a day. That limit matters. A controller that keeps restarting a broken job can erase the evidence of the failure, create alert noise, and consume the time needed to investigate the actual cause.
Everything outside those three actions becomes diagnosis and escalation. The controller deliberately does not unlock, prune, initialize, or otherwise mutate a backup repository. It does not free disk space. It does not edit scripts, unit files, or credentials. Those are consequential decisions with a much larger blast radius than “start the known dependency once.”
The useful boundary is not “the system never touches anything.” It is “the system can only touch what it can reverse and explain.”
Failure needs a receipt, not a storm of email
Repeated failures should be correlated as one incident. Otherwise an hourly backup failure produces hourly messages, and the operator learns to ignore a real signal.
The notification layer tracks state transitions instead:
| Transition | Outcome |
|---|---|
| Healthy → failing | Open one incident with the failure context and a proposed next step. |
| Failing → failing | Update the incident without repeating the alert. |
| Failing → healthy | Send one resolution notice on the same incident. |
| Healthy → healthy | Send, at most, a low-frequency health confirmation. |
Each remediation attempt writes a receipt: what it observed, whether an action was permitted, what it attempted, and why it stopped. A receipt is much more useful than a vague “auto-heal succeeded” message when a later restore review needs to reconstruct the event.
There is a local-model assist in the notification flow, but it is intentionally proposal-only. It receives a constrained diagnostic summary and can suggest a cause, a change for human review, or a next command. It has no execution path. The deterministic controller enforces the allowlist; the model is not an authority boundary.
Stopping is part of recovery
The controller did not auto-fix the credential-delivery failure described above. That was correct. Its allowed actions did not include changing how a database credential was resolved, so it classified the incident, retained the evidence, and escalated it.
That is the behaviour I want from a recovery system. It should resolve a small set of understood, reversible conditions quickly. It should refuse to turn uncertainty into write access. And it should leave a clear trail when a person needs to take over.
“Autonomous” is only a useful description when the boundary is named. In this case, the automation owns scheduled verification, constrained retries, and evidence collection. The operator still owns retention changes, destructive actions, recovery-policy changes, and the decision to make a novel repair.
The backup job is not valuable because it can do everything. It is valuable because it can prove when it should not.
If this overlaps with something you are working on
Send me a short note describing the workflow, what is frustrating about it today, and any data, timing, or approval constraints that matter. Start a conversation.
Three commentaries
These are invited responses to the article's argument and its possible evolution. They are viewpoints, not independent reviews, endorsements, or evidence for the claims above.
Codex
The recovery controller is compelling precisely because its narrow repair set is paired with a record of why it stopped. Its next maturity step is governance, not more autonomy: periodically review the escalation receipts, decide which recurring failures truly deserve a safe new primitive, and reject the rest. That keeps the allowlist earned by evidence rather than expanded by convenience.
Qwen, local model
This article wisely defines the limits of automated recovery, emphasizing that a backup job’s value lies in its ability to refuse untrustworthy states rather than force repairs. The three-action allowlist prevents automation from obscuring root causes or creating dangerous side effects. To evolve this pattern, the system should incorporate machine-readable confidence scores for its diagnoses, allowing operators to prioritize interventions based on the controller’s certainty level. This would transform simple receipts into actionable risk assessments, helping humans decide when to intervene versus when to trust the automated containment.
Claude
Only touch what it can reverse and explain is a better boundary than any autonomy adjective, and gating the archive on native dumps caught a failure that a file-copy check would have missed. The allowlist is empirical, though—three actions drawn from failures already seen. I'd close that loop deliberately: review escalated incidents on a fixed cadence and decide whether a fourth action has earned its place.