Each control was correct. Together they built a job only I could do.

It started as a pricing decision, not a technical one

I want to be honest about the trigger, because the engineering write-ups of migrations usually begin with a technical failure and this one did not.

My self-hosted secrets manager worked. It had been running the stack's Docker services without complaint for months. What it did not have, on the tier I was willing to pay for, was multi-factor authentication on the administrative account.

The record of my own indecision is unflattering and worth including. In late July I asked whether there was an alternative with MFA included for free, and concluded I should "either accept it or figure out a different tool." Two days later I found MFA was in fact available and noted that the concern could be dropped. And then I migrated anyway, because the version I could actually use long-term still meant an enterprise key I had decided not to buy.

So the real driver was a licensing boundary drawn through a security feature. That is a legitimate reason to move. It is also a reason that generates none of the urgency a technical failure would, which matters later, because a migration with no outage behind it competes against everything else for twenty-five days.

I committed by making it impossible to stop

On 27 July I did something I still think was right. I attached a condition to my agent session — install the new store and migrate all secrets to it — that blocked the session from ending until the condition held.

It is a strange kind of commitment device: automation used not to do the work but to stop me from wandering off it. It worked in the narrow sense. The store was installed and initialized that day.

It did not work in the sense I actually needed, because "install the store and move the secrets" turned out to be perhaps a fifth of the job. The rest was consumers, evidence, rotation, and proof — and none of that is a thing a hook can hold you to, because none of it has a clean finish line.

Every control I chose was individually correct

The design principles I picked were, and are, defensible. I would recommend each of them to a client:

Value-blind tooling. Scripts never print a secret value. Output is fingerprints, counts, and digests. A transcript can be pasted anywhere without leaking material.

Digest-pinned applies. Every write is a two-step: a dry run emits a plan hash, and the apply refuses unless you pass that exact hash back. You cannot apply a plan you did not just read.

Attended-only decryption. Recovery material can only be opened at a real terminal with a human at it. No passphrase arguments, no cron, no agent.

Ephemeral privilege. Tokens are created for one operation and revoked at the end, recorded in the receipt.

Chained evidence. Each snapshot references the digest of the evidence before it, so the history cannot be silently rewritten.

Individually: correct. I still believe that. Read them as a set and you can already see the shape of the problem.

The compound effect is a process only a present human can execute

Put those five together and you have defined a job that cannot be delegated, to a person or a machine, by construction.

I tried to delegate it twice. In early August I asked an agent, in as many words, whether I could hand over all the passphrases so it could finish the migration itself. Ten days later I asked for instructions to issue it a write token. Both times the honest answer was the design's answer: no, because attended-only means attended, and a standing write token to the secret store is the exact thing the ceremony exists to avoid.

The controls were doing their job. Their job, in aggregate, was to guarantee that a specific human being had to be awake, at a terminal, with access to the recovery passphrase, for every consequential step. On a team you solve this with separation of duties — two people, either of whom can run the ceremony. Solo, you have built a system whose throughput is capped by your own calendar.

The timestamps make this concrete in a way I did not enjoy reviewing. Two in five of my messages about this migration fall between 6pm and midnight. The ceremonies did not fit in the working day, because the working day already had client work in it — so the attended steps, the ones that by design could not be handed to anything, consistently landed in the evening.

The failure I did not design for was not being able to find my own key

This is the part I would most want another operator to take seriously, because it is not in any threat model I wrote down.

Three times across the migration, in my own words, I asked where my own material was. Where the store's passphrase was. Where my token was. Where a specific recovery file lived. On one occasion, after a preflight check refused to continue, I asked whether the store was now simply inaccessible.

It was not. Every time, the material existed, was correctly protected, and was recoverable. The failure was not custody. It was retrievability under pressure — knowing, at the moment you need it, which of several protected artifacts holds the thing you want and what opens it.

There was also a genuine circularity to break. The most convenient copy of the passphrase protecting the sealed store's recovery material lived in a password vault running on the same host. That is fine on an ordinary Tuesday and useless in the scenario the material exists for, which is the host being unavailable. The mitigation is unglamorous and mandatory: an off-machine copy, and a written note saying plainly that the on-host copy must not be relied on alone.

A related moment: an automated preflight for the boot ceremony failed with the vault client locked, and correctly refused to proceed — "no passphrase, nothing further can be tested." Correct behaviour. It also meant the readiness check for my unattended recovery path could only be run attended.

Some friction was pure tax with no safety benefit

Not all of the twenty-five days bought security. A meaningful slice was the controls rejecting me for reasons that had nothing to do with risk.

Twice I pasted a plan digest that was sixty-three characters instead of sixty-four — one character lost in a copy — and the apply refused. It was right to refuse; it could not know the difference between a typo and a substituted plan. But the operator experience is a hard stop with no hint that the input was merely short. Separately, a multi-line command with a wrapped hash split across lines and the shell tried to execute the fragments as commands.

Then there is the cost of value-blind output during an actual failure. A provisioning run failed with, in full, "unexpected provisioning failure; details suppressed." Debugging that required writing a bespoke diagnostic that printed a traceback of file, line, and function only — deliberately no values and no message. Suppression is the right default when output may contain secrets. But a system that cannot tell you why it failed has moved the cost from the attacker to you, and on a solo operation you pay it out of your evening.

Two separate incidents also required triage after secret material reached agent session output. Both were caught and both resulted in rotation. That is the system working. It is also two more attended ceremonies added to the total.

What I would keep, and the one thing I would change

I would keep all five controls. The evidence chain in particular earned itself: when I needed to confirm what had actually happened weeks earlier, the receipts were authoritative in a way my memory and my transcripts were not. My conversation record does not even cover the full period. The receipts do.

What I would change is that I designed the controls and never designed the operator path through them. Specifically, before starting a migration like this again, I would write down — before the first ceremony, not after the twelfth:

• Which artifact holds which recovery material, in one index, in plain language.
• What opens each one, and where the off-machine copy of that lives.
• Which steps require an attended terminal, so the calendar cost is visible up front rather than discovered at the end of a long evening.
• What each refusal actually means, so a rejected input is distinguishable from a dangerous one.

None of that weakens a single control. It is documentation, and it is the difference between a safe process and a safe process a tired human can execute.

The lesson I am taking into client work is narrower than "security is hard." It is this: when you evaluate a control, you evaluate it alone, but you operate your controls as a set. The compound requirement — how many of them demand a specific present human simultaneously — is a real property of the system, and nobody I know measures it. On a team it shows up as friction. Alone, it shows up as elapsed weeks, and as an operator asking his own notes where the key is.