Evaluation
Evaluation
Control Desk separates two distinct quality concerns: whether the detectors find every problem (recall), and whether the AI correctly categorizes the problems it is shown (category accuracy). Conflating them would make failures impossible to attribute.
Two metrics, two layers
The evaluation harness tracks two acceptance gates, each measuring a different layer of the system:
- Recall floor: 0.95. Recall measures whether the deterministic detection layer found every break that was planted in the evaluation corpus. A recall failure means a detector missed something — no AI is involved in this number, because detection is fully deterministic. The acceptance gate is 0.95.
- Category-accuracy gate: 0.70. Category accuracy measures whether the AI, given correctly detected findings, assigned the right cause category. This gate applies only over findings that were detected; if a finding was missed (a recall failure), it cannot contaminate this number. The acceptance gate is 0.70.
Keeping the two metrics separate is the key design decision. A flaky detector lowers recall but cannot silently drag down the AI-quality number. An AI that consistently mis-categorizes a class of problem shows up in category accuracy without inflating or deflating the recall result. Each failure attributes to the right layer.
These gates are acceptance criteria, not recorded achievements. The system passes evaluation when both gates are met; no further numerical claim is published.
Answer-key separation
For evaluation to be honest, the AI must be graded against answers it cannot read. Control Desk enforces this structurally.
The evaluation harness plants known breaks in the evaluation corpus and mints a secret answer key at that time. Two enforced boundaries prevent the diagnosis path from seeing that key:
- An import-boundary guard makes it structurally impossible for the diagnosis path to import from the evaluation harness's answer-key module. The guard is automated and fails the build if the boundary is crossed.
- A serialization-leak check verifies that the answer key cannot reach the diagnosis path through an indirect route — for example, embedded in a shared data structure. This check also runs automatically.
The result is that the AI is evaluated blind. It cannot read the expected answers, directly or indirectly. The grade it receives reflects its actual performance on the evidence it was given.
The two-domain proof
The substrate is claimed to be domain-neutral. The evaluation makes that claim verifiable: one unchanged substrate engine ingests two unrelated domains — each through nothing but a field mapping — and surfaces every seeded divergence in both. If the substrate contained any domain-specific logic, the second domain would fail.
This is not a theoretical argument. The two-domain proof runs as part of the automated test suite: 279 tests as of 2026-07-16, exercising both domains, all passing.
Zero network calls, injected fakes
The test suite makes zero network calls. AI behavior in tests is exercised through injected fakes — implementations that return controlled outputs deterministically — rather than through live AI calls. This makes the suite fast, stable, and runnable without external dependencies.
Testing AI behavior with fakes is sound because the diagnosis step has a defined interface: it receives an evidence file and returns a structured response from the fixed taxonomy. A fake that implements the same interface exercises every part of the pipeline that handles AI output — the structured parse, the fallback path, the degradation reporting — without requiring a live model.
Evaluation isolation
The evaluation corpus and the governance configuration are kept structurally separate. Structural detection rules (from the governance path) are proven silent on the evaluation corpus, which means adding or changing governance rules cannot move the benchmark numbers. The gate results reflect the underlying detection and AI behavior, not the rule configuration of any particular deployment.
For how the pipeline produces the outputs that evaluation grades, see Pipeline. For how governance rules interact with detection, see Governance. The technical grounding for these measurement choices is developed in full in the whitepaper Grading an AI that cannot see the answers.