The files copied, but the workflow did not complete

What I observed

A long-running data job appeared to have finished. The destination had the files. The sizes looked right. Nothing had errored, and nothing in the output suggested a problem.

Then four downstream stages refused to run. Each one checked for the same thing before starting: a completion marker written by the job as its final act. The marker was not there.

So I had a state that reads as a contradiction: the data arrived and the workflow did not complete.

What the obvious explanation suggested

The obvious readings, in the order I actually considered them, were all comfortable ones:

• the marker path was wrong, or permissions blocked the write;
• the downstream stages were checking the wrong location;
• a race condition meant they checked before the job's last step landed;
• or the marker logic was simply buggy and had never worked.

Every one of those makes the marker the problem. Three of them end with me deleting or relaxing the check. That is the direction the reasoning naturally runs, because the visible evidence — the files are there — feels like the stronger signal.

It was the weaker signal.

Why that was wrong

I had edited the script while it was running.

This is a specific and slightly notorious property of shell scripts. A shell does not load the whole script into memory and then execute it. It reads from the file, executes what it has, and comes back for more, tracking its position as a byte offset into the file.

Edit the file in place while it is running and the offset is now pointing into different content. The shell resumes at the same numeric position in a file whose bytes have shifted. Depending on how much the length changed, it will resume mid-line, skip a block entirely, execute something twice, or die on a syntax error in code that is perfectly valid on disk.

In my case the edit shifted the offset past the finalization block at the end of the script. The main body — the part that copied the data — had already been read and was running normally. The tail, which verified the result and wrote the completion marker, was never read at all.

That is the whole failure. The job did not crash. It did not report an error. It ran a shorter program than the one I had written, and the difference was exactly the part that produces the evidence.

Two details make this worse than an ordinary bug:

• It is invisible in the output. The skipped block never ran, so it never logged anything, including its own absence.
• The file on disk is correct. Every post-incident inspection of the script shows finalization logic that plainly works. The bug lived in the gap between the file and the running process.

What the control got right

The four downstream stages did exactly what they were built to do. They had no way to know why the marker was absent, and they correctly declined to assume.

This is worth stating clearly, because the shape of the incident is unusual: the control did not fail. The control was the only component that noticed anything was wrong.

Had those stages instead checked "do the expected files exist at the destination" — the check that feels more direct, more real, closer to the actual goal — they would all have run happily on top of a job that never verified its own output.

The one-line fix that would have forged evidence

The fastest way out of that state is to create the marker by hand and let the pipeline continue. The files are there. I know why the marker is missing. The job "really did" finish.

That reasoning is wrong in a way that matters more than the hour it saves.

A completion marker does not mean the files appear to be present. It means the finalization block ran to the end and its checks passed. Those checks are the difference between "bytes exist at the destination" and "the transfer was verified, the count reconciled, and the result is safe to build on."

In this incident the finalization block never executed. I did not know whether its checks would have passed. Creating the marker by hand would not have been a shortcut to the truth — it would have been me asserting, in the system's own evidence format and indistinguishably from the real thing, a fact I had not established.

And the forgery is durable. Six months later the marker is just a marker. There is nothing in it that says a human made it because they were confident. Every downstream consumer, every audit, every future me reading the state would treat it as proof that a verification ran.

A control that can be satisfied by hand, in the same form the system writes, produces evidence that cannot be trusted afterward. That is a design problem, not a discipline problem.

What I changed

Four things, in increasing order of how much they generalize.

The immediate one. Re-run the job from a clean state and let it write its own marker. This is the boring answer and it is the correct one. The cost was time, and the thing I got back was a true statement about the data.

Never edit a running script in place. Edit a copy and move it into position, so the running process keeps its original inode and finishes the program it started. Any editor that writes to a temporary file and renames it over the original is already safe; an editor that truncates and rewrites in place is not. This is a one-line habit change that eliminates the entire failure class.

Make the marker carry evidence, not just existence. A marker whose content is empty can be reproduced by anyone with a keyboard. A marker containing what the finalization step actually observed — counts, sizes, a checksum of the manifest, the start and end timestamps, the version of the job that ran — cannot be casually forged, because forging it requires doing the work it attests to. The check then verifies the contents, not the filename.

Make skipped finalization loud. A job that can end without running its own final block should treat that as an error condition rather than a silence. A trap on exit that asserts finalization ran, and complains if it did not, turns an invisible truncation into a visible failure.

What this did not prove

It did not prove the data was bad. As far as I could tell it was fine — that was never the question. The question was whether I could demonstrate it was fine, and for a period of time the honest answer was no.

It also did not prove that markers are the right control for every pipeline. They are a good fit where a stage has a meaningful notion of "verified and complete." They are a poor fit where completion is continuous or partial, and bolting one on there produces ceremony rather than evidence.

What it means operationally

The transferable lesson is not about shell scripts. It is about which direction you reason when a control disagrees with your intuition.

When a check blocks work and the surrounding evidence suggests everything is fine, there are two possible worlds. In one, the check is broken. In the other, the check is the only thing that has noticed a real problem. Those worlds look identical from the inside, and only one of them is safe to act on.

The tell is what the control actually asserts. If it claims something narrower and more specific than "everything looks okay" — this exact step ran, these exact counts reconciled — then it knows something you do not, and overriding it means discarding the one piece of information you were missing.

What to do differently tomorrow

• Check whether any of your controls can be satisfied by hand in the same form the system produces. Every one that can is a control that produces unfalsifiable-looking evidence. Give it content that requires the work.
• Stop editing running scripts. Edit a copy, then move it into place.
• When a gate blocks you, work out what it asserts before you work out how to get past it. If it asserts something you have not independently established, it is doing its job.
• Treat "the output looks right" as a hypothesis, not a result. It is evidence about appearance. Completion markers exist precisely because appearance and correctness come apart, and the moment they do is the moment you most want to have been strict earlier.

Further reading

• Bash Reference Manual: Shell Operation — how the shell reads and executes input incrementally.
• POSIX shell command language — the specified reading and execution model behind this behaviour.
• NIST SP 800-34: Contingency Planning Guide for Federal Information Systems — recovery testing and the difference between a completed procedure and a verified one.