Whitepaper · WP-3
Grading an AI that cannot see the answers
Abstract
An AI graded on data it can see is not being graded; it is being flattered. Control Desk evaluates its single AI call — diagnosis — against a secret answer key that the diagnosis path is structurally unable to read, enforced by an import-boundary guard and a serialization-leak check rather than by trust. It measures two things on two layers: recall, driven entirely by deterministic detectors, with an acceptance floor of 0.95; and precision-over-detected, the AI-quality gate, with an acceptance floor of 0.70. The separation is deliberate, so a flaky detector cannot silently distort the AI number and failure attributes to the layer that caused it. When the AI is uncertain it degrades loudly to a safe default rather than guessing, and the workspace reports the observed fallback rate rather than the configured intent. The suite is 279 automated tests, green as of 2026-07-16, making zero network calls, with AI behavior exercised through injected fakes. The intellectual contribution is the stance itself: these floors are acceptance gates, not achievements, and no live accuracy figure is reported. This paper builds on WP-1's invariants and evaluates the AI call whose cost is characterized in WP-2.
1. Why measure recall and precision on different layers
Control Desk does two very different jobs. Deterministic detectors decide what to flag; the single AI call decides why a flagged class of problem occurred, choosing from a fixed taxonomy of causes. These are separate competences, and folding them into one score hides which one failed. So they are measured separately.
- Recall asks: of the divergences that were really there, how many did detection find? It is driven entirely by the deterministic detectors, with no AI involved, and its acceptance floor is 0.95.
- Precision-over-detected asks: of the problems that were detected and then diagnosed, how often did the AI assign the right category? This is the AI-quality gate, and its acceptance floor is 0.70.
The word over-detected is load-bearing. The AI is scored only on the problems detection actually surfaced, so its number reflects its own judgment and nothing else.
This is the reason for the split. If a single blended accuracy dipped, you could not tell whether the detectors missed things or the AI mislabeled them — and you would tune the wrong layer. By separating recall from precision-over-detected, a detection regression moves recall and leaves the AI-quality gate untouched, while an AI regression moves precision and leaves recall untouched. Each number accuses its own layer. This clean attribution is exactly what WP-2's architecture makes possible: because the build forbids any AI in detection, clustering, or triage, recall is provably an all-deterministic measurement with no AI to blame or credit.
2. An answer key the AI cannot read
Grading requires knowing the truth. The evaluation harness plants known breaks in the corpus and mints a secret answer key describing them. The obvious risk is that the thing being graded gets to see the key — that the diagnosis path, somewhere along its data flow, reads the answers it is supposed to be inferring. If that were possible, every reported number would be worthless.
Control Desk makes it impossible rather than improbable. Two mechanisms, described at the level of what they enforce:
- An import-boundary guard makes it structurally impossible for the diagnosis path to reach the code that holds the answer key. The separation is a boundary the build refuses to let anyone cross, not a coding convention.
- A serialization-leak check ensures the answer key cannot ride along inside the bounded case file — the only view of the data the AI is ever given. The AI sees evidence assembled for it and nothing more; the answers are not smuggled in as a field it could read.
Together these make the grading honest by construction. The AI is graded against answers it structurally cannot see, so a high precision number can only mean the AI reasoned to the right cause, never that it copied it. This is the answer-key-separation invariant introduced in WP-1, described here in full because it is what makes every other number in this paper meaningful.
3. Safe degradation and honest status
An AI that must always answer will fabricate one when uncertain, and a fabricated diagnosis is worse than an admitted unknown because it looks like knowledge. Control Desk's diagnosis step therefore never halts the pipeline and never guesses to save face. On any failure or uncertainty it returns "unknown" with a conservative default action, and it reports the degradation loudly rather than hiding it.
That loudness feeds the workspace's AI status honestly. The status the workspace shows is the observed diagnosis fallback rate — how often the AI actually degraded to the safe default — not the configured intent or a hoped-for figure. If the AI is falling back more than expected, the operator sees the real rate, not a reassuring constant. Honest status is the runtime companion to honest grading: the evaluation says what the AI's quality is under test, and the workspace says what it is doing right now.
4. Gates, not results
The most important sentence in this paper is about what the numbers 0.95 and 0.70 are. They are acceptance gates — thresholds a run must clear to be considered passing — not recorded achievements to be advertised. We publish the floors and refuse to publish the achieved accuracy above them.
This is a deliberate epistemic stance, and it is the paper's real contribution. A single reported accuracy number invites two dishonesties: it tempts the publisher to quote the best run as if it were typical, and it tempts the reader to treat a figure measured on one corpus as a guarantee on their own. A gate avoids both. It states the minimum quality a run must demonstrate to pass, on a corpus whose answers the AI could not read, and it says nothing about a specific achieved figure that would not transfer honestly to your data anyway. Holding the line here — publishing the acceptance floor and never an achieved accuracy — is how the evaluation stays trustworthy instead of merely impressive.
5. What the test suite proves
The evaluation design is only as good as the discipline that runs it. As of 2026-07-16 the suite is 279 automated tests, all green. Two properties matter as much as the count. First, the suite makes zero network calls: nothing in it depends on a live external service, so a passing run reflects the system's own behavior and not the weather on someone else's servers. Second, AI behavior is exercised through injected fakes — stand-ins that let the pipeline's handling of any AI response, including failures and degradations, be tested deterministically and repeatably. The tests prove that the deterministic scaffolding behaves correctly around the AI under every response it might give, which is precisely the scaffolding this whole paper argues should be doing the load-bearing work.
What this does not claim
This paper reports no achieved live accuracy figure, by design; the acceptance floors are 0.95 for recall and 0.70 for precision-over-detected, and those are gates a passing run must clear, not scores we assert we reached. It does not claim the AI is correct on data unlike the seeded evaluation corpora — a gate cleared on one corpus is evidence of method, not a guarantee on yours. It does not claim "unknown" is always the right answer; safe degradation is a floor on harm, not a substitute for a real diagnosis, and a high fallback rate is a signal to investigate, not a success. It does not claim 279 green tests prove the absence of all bugs — no finite suite can — only that the behaviors those tests assert hold. And it does not claim the answer-key separation defends against threats outside its two mechanisms; it guarantees the diagnosis path cannot read the key, which is the specific dishonesty it was built to prevent.
6. What this means for your data operations
The practical consequence is that you can trust the one AI call in the loop because its quality is gated, its grading is honest by construction, and its uncertainty is reported rather than hidden. When the AI is unsure, your pipeline degrades to a safe default and tells you so; when it is confident, its confidence was earned against answers it could not see. See the honest AI-status chip and the diagnosed cases in Control Desk, read the evaluation method stage by stage in the documentation, and understand the architecture whose invariants make this separation enforceable in WP-1. The cost of that single, carefully graded AI call — and why it stays flat as your data grows — is the subject of WP-2.