A perfect evaluation score stopped being trustworthy

What I observed

I built a detector for a classification task, generated a test corpus for it, and evaluated it. It scored perfectly. Not "strong" — perfect, across every category in the corpus.

My first reaction was satisfaction. My second, arriving a beat later and much less welcome, was that I could not think of a single realistic system that should score perfectly on its first honest evaluation.

What the obvious explanation suggested

The comfortable readings are all available:

• the task is genuinely easy;
• the detector is genuinely good;
• the corpus is small enough that perfect is unremarkable.

Any of those could be true. The problem is that a fourth explanation predicts exactly the same score, and I had no way to tell it apart from the others.

Why the score carried no information

Here is how the evaluation had actually been assembled, in the order it happened:

1. A model helped me enumerate the cases the detector should catch.
2. I used a model from the same family to generate the test corpus from that enumeration.
3. I tuned the detector until it handled the corpus.
4. I evaluated the detector against that same corpus.

Step four is not an evaluation. It is a consistency check on steps one through three.

The corpus contained the cases that model conceived of when asked to imagine the problem. The detector was tuned until it caught the cases in that corpus. Anything the model did not think of was absent from the corpus — and therefore absent from the tuning, and therefore absent from the score.

The blind spots did not cancel out. They were shared, and a shared blind spot is invisible to the metric by construction: a case that neither the corpus author nor the detector considers is not a case the detector fails. It is a case that does not exist as far as the measurement is concerned.

That is what makes the perfect score worse than a mediocre one. A detector scoring 80% tells you about 20% of something. A detector scoring 100% against a corpus that shares its conceptual origin tells you the two artifacts agree, which you already knew, because you built them that way.

The score was not wrong. It was answering a different question than the one I thought I had asked. I had asked "how well does this detector cover the problem?" and the number I got back answered "how well does this detector match this corpus?" — and the corpus was never an independent description of the problem.

The general form

This is broader than one detector, and it will get more common rather than less.

Wherever a model participates in both producing a system and producing the evidence used to judge it, correlated error is the default outcome and the metric silently loses its power to discriminate. The pattern appears in:

• test cases generated by the model whose output they will grade;
• LLM-as-judge setups where the judge and the generator are the same family;
• retrieval evaluations using questions written from the documents that were indexed;
• synthetic training data evaluated against synthetic test data from one source;
• and prompts refined against a fixed example set until the examples pass.

The mechanism is the same every time. The measurement and the thing being measured share a parent, so they share a distribution, and agreement between them stops being informative about the world.

What I changed

Provenance became a required field. Every evaluation artifact now records where it came from: who or what generated it, when, from what source, and whether the system under test — or its ancestors — had any role. An artifact whose provenance overlaps the system's is marked as such, and results from it are labelled a consistency check rather than an evaluation. This is bookkeeping, and it is the change that does the most work, because it makes the failure visible at a glance instead of requiring someone to remember the history.

Independence became a property of the corpus, not an aspiration. A holdout set is only a holdout if its cases were derived without the system's involvement — from a different model family, from a published dataset, from real (permitted) examples, or from cases written by hand specifically to be awkward. The headline number now comes from an independent set. The original corpus still runs, as a regression check, which is a real and useful job that is simply not the job I was asking it to do.

Adversarial cases got written by hand, on purpose. The most valuable part of this exercise turned out to be sitting down and asking: what would a case look like that a model asked to imagine this problem would not produce? Boundary conditions, deliberately ambiguous inputs, near-misses, cases that should be refused rather than classified, mixed-language and malformed input, and things that are legitimately hard to adjudicate. Those cases came from a person, and they immediately stopped the score being perfect — which was the point.

A perfect score became an alarm. In this system, 100% now triggers an investigation rather than a release. Either the task is trivial, the corpus is too small, or the evaluation shares an origin with the thing it measures. All three are things I would rather know than not.

What this did not prove

It did not prove the detector is bad. It may well be reasonable. The finding is strictly about the evaluation: the number I had could not support the claim I wanted to make with it.

It also did not establish a validated accuracy figure. Producing one would require an independent corpus, a documented sampling method, and ideally someone other than me adjudicating the disagreements. I have not done that, so I do not publish a number.

And it is not a claim about any specific vendor or model family. The mechanism is structural. It applies to every model that helps build the thing that judges it, including the ones I use.

What it means for a buyer

If someone shows you an evaluation of an AI system — your team, a vendor, a consultant, me — there is one question that separates a measurement from a demonstration:

Who or what produced the test cases, and did the system being tested have any role in producing them?

If the answer is "the same model," the score describes internal consistency. That is worth something for catching regressions, and it is worth nothing as evidence of real-world coverage. It should not be the basis of a purchase, a go-live decision, or a claim to a customer.

The follow-up is equally simple: what does this evaluation not cover? A team that has genuinely thought about independence can answer immediately, because they had to decide what to exclude. A team that cannot answer has not yet discovered that the question exists.

What to do differently tomorrow

• Write down the provenance of every test set you rely on. If any of them shares an origin with the system under test, you have a consistency check labelled as an evaluation. Relabel it — that alone is progress.
• Build one small independent set, from a different source, by a different method. It does not have to be large to be more informative than a large correlated one.
• Hand-write the awkward cases. Ask what a model imagining this problem would fail to imagine, and write those down. It is uncomfortable and it is the highest-yield hour in the whole exercise.
• Treat a perfect score as a defect report against your evaluation, not a result from it.

Further reading

• NIST AI Risk Management Framework — the Measure function, and documenting what an evaluation does and does not establish.
• NIST AI RMF Core — measurement validity, independence of assessment, and documented limitations.
• Datasheets for Datasets — recording dataset provenance, composition, and intended use.
• Model Cards for Model Reporting — reporting evaluation conditions alongside performance figures.