The stack was green and no prompt worked

What I observed

Every service in an AI stack reported healthy. Container health checks passed. The orchestrator showed the full set running. Restart counts were zero. The monitoring dashboard was uniformly green, and had been for some time.

Not one prompt returned an answer.

There was no error banner, no crash loop, no resource exhaustion, and nothing in the logs that announced itself as a failure. The system was, by every signal I had built to watch it, working perfectly.

What the obvious explanation suggested

When everything is green and nothing works, the first instinct is to distrust the part you can see least. So I went looking at the model backend: was it actually loaded, was it out of memory, was inference silently timing out?

That is a reasonable place to look, and it was wrong, in an instructive way. The backend was fine. It answered when I asked it directly.

So was every other component, individually. Each one, tested on its own, behaved exactly as designed.

Why that was incomplete

The chain looked like this — synthetic, but structurally what matters:

  user
    │
    ▼
  chat interface ──► gateway / router ──► model backend
       health: ok        health: ok           health: ok

Three components, three health checks, three green lights. And here is what each health check actually asserted:

Every check was a liveness check. Liveness is a statement about one process: am I running, did I bind my port, does my own status endpoint return success.

Not one of them was a contract check — a statement about the boundary between two processes: when I ask my neighbour for the thing I will actually ask it for in production, do I get back the thing I expect?

The failure lived entirely in those boundaries. The identifier the interface requested did not resolve to anything the router could route. The router returned an error the interface rendered as an empty response rather than a failure. Nothing in that path was unhealthy. The path simply did not exist, and no check had ever been asked to look at a path.

This is what makes the class dangerous: the number of things that can be individually healthy while collectively useless grows with every component you add. Adding more health checks of the same kind does not help. It makes the dashboard greener.

What I changed

One synthetic end-to-end request became the real health check. A single scripted transaction that enters where a user enters, traverses every hop, and asserts on the content of what comes back — not the status code. If a real question does not produce a real answer, the system is down, regardless of what the component checks say.

This is the whole fix, and it is deliberately unsophisticated. It costs one inference per interval. It replaces a dashboard that could not fail with one that fails when the product fails.

Identifier drift became a startup assertion. The names by which components refer to each other — model identifiers, route names, aliases — are a contract in the most literal sense, and they were being maintained by hand in more than one file. Now the deployment asserts at startup that every name one component requests resolves in the component that must serve it, and refuses to come up otherwise. A mismatch fails at deploy time, loudly, instead of at request time, silently.

Empty responses stopped being rendered as success. A downstream error that arrives as an empty body is a failure, and the interface now says so. This was the reason the outage was invisible rather than merely present: the one place a human was actually looking had been designed to degrade quietly.

Health check semantics got named. Every check in the stack is now explicitly one of three things — liveness (the process is running), readiness (this component can serve requests), or contract (the boundary to my dependency works end to end). Written down, the gap was obvious: there had been eleven of the first kind and none of the third.

The positive and negative tests

A fix that is only ever tested when it passes is not tested.

• Positive: the synthetic transaction returns a well-formed answer, checked on content, on a schedule.
• Negative, and more important: deliberately break each link in a staging copy — rename an identifier so it no longer resolves, stop the backend, make the router return an error — and confirm the check goes red for each one.

Only the negative test proves the check can detect anything. A monitor that has never been observed to fail is not a monitor; it is a decoration that happens to be green. Running that exercise is what turned "we added a check" into "we know what this check catches."

What this did not prove

It did not prove the stack is correct. It proves one representative path works, which is a much smaller claim, and deliberately so. Other paths, other identifiers, and other request shapes remain untested unless someone tests them.

It also did not prove that per-component health checks are useless. They are good at what they do: telling you which component to look at once you already know something is broken. The mistake was letting them answer a question they were never able to answer — is the system working?

What it means for a buyer

This one translates almost too neatly.

An AI system's components can each be operating exactly as specified while the system delivers nothing. If your assurance that a system is working comes from a dashboard of component statuses, you have assurance that the components are running. That is not the same as assurance that anyone can get an answer, and the gap between those two claims is invisible until someone tries.

The question worth asking a vendor, an integrator, or your own team is not "is the system monitored?" It is: when did a real request last traverse the whole path and return a correct answer, and what would have gone red if it hadn't?

If the answer is a dashboard, the answer is that nobody knows.

What to do differently tomorrow

• Count your contract checks. Not health endpoints — checks that cross a boundary and assert on content. If the count is zero, your monitoring cannot detect the failure described here.
• Add exactly one synthetic transaction through the full path, asserting on the response body. One is enough to change the category of thing you know.
• Break it on purpose, in staging, once per link. Confirm each break turns the check red. Anything that stays green is a blind spot you now have written down.
• Audit where errors become silence. Empty states, default fallbacks, and optional-chaining that swallows a null are where outages go to hide from the only human who is looking.

Further reading

• Google SRE Book: Monitoring Distributed Systems — symptom-based versus cause-based monitoring, and why black-box probes matter.
• Google SRE Book: Testing for Reliability — probes, canaries, and testing the monitoring itself.
• Kubernetes: Configure Liveness, Readiness and Startup Probes — the precise semantic difference between the probe types.
• NIST AI Risk Management Framework — measuring whether an AI system achieves its intended purpose, not merely whether it runs.