Detection
Detection
Detection runs in step 4 of the pipeline. It is entirely deterministic: every finding is produced by evaluating an explicit rule against the data, with no AI involvement.
Two detection axes
Control Desk organizes detection along two independent axes. The axis that applies to a given deployment depends on how many sources are configured.
Cross-system detection
Cross-system detectors operate on two or more sources simultaneously. They identify divergences between systems — cases where the same entity appears differently (or not at all) in different sources. A cross-system detector cannot run with a single source; it requires at least two.
Structural detection
Structural detectors operate on a single source. They validate individual records against constraints defined in the detection rules — independently of what any other source says. A single-source deployment runs structural checks only; a multi-source deployment runs both structural and cross-system checks automatically.
The seven detection capabilities
Cross-system: value mismatch
Identifies cases where the same entity is present in two or more sources but carries different values for a mapped field. The detector compares the canonical representation of that field across sources and produces a finding for every divergence that exceeds the configured threshold.
Cross-system: missing upstream
Identifies entities that are present in one source but absent from a source that should contain them. This pattern is missing-upstream detection. It surfaces gaps in replication, late arrivals, and deletion events that were not propagated.
Cross-system: schema drift
Identifies cases where the canonical shape of records from one source diverges from the expected schema — fields appearing or disappearing, type changes, or unexpected nullability. Schema drift detected early prevents downstream normalization failures.
Structural: not-null
Flags records where a field declared non-nullable contains a null or empty value. The rule specifies which fields are required and what severity to assign when the constraint is violated.
Structural: format
Validates field values against a declared pattern. A field expected to carry a date in a specific format, a code with a fixed structure, or a reference in a known form will produce a finding whenever a record deviates from that pattern.
Structural: range
Validates numeric or date fields against declared bounds. Records where a value falls outside the permitted range — below a floor, above a ceiling, or outside a window — produce a finding at the configured severity.
Declarative, rule-driven configuration
Every detector is controlled by rules that declare what to check, the severity of a violation, and the scope of application. Rules are authored through the governance intake flow (validate, preview, approve, then activate as a separate step). No custom code is required to configure a new check.
Single-source deployments run only structural checks. When a second source is added, cross-system checks become available automatically — no configuration change to the detection rules is needed.
Source connectors
Detection begins with data ingestion. Control Desk supports four connector types. Each connector handles reading the source; a per-source field mapping then declares how source fields align to canonical ones.
- CSV — delimiter-separated flat files.
- JSONL — newline-delimited JSON records.
- Sample-data — a built-in connector for seeded synthetic corpora, used in evaluation and benchmarking.
- SQL — a relational database source, proven at approximately one million records.
Adding a new source type requires only a new connector and a field mapping. The detection logic, clustering, triage, and all downstream steps are unchanged.
Per-source field mapping
Because different source systems use different names for the same concept, each connector is accompanied by a field mapping. The mapping is declared in configuration: it lists which source field corresponds to which canonical field. After the map step, all records — regardless of origin — share the same canonical field names. Detection rules are written against canonical fields and therefore apply uniformly across all sources.
For the governance flow that creates and activates detection rules, see Governance. For how detected findings are clustered and routed to diagnosis, see Pipeline.