Detection
97%
62 of 64
Engine calibration record
Measured accuracy against known answers
The engine is run against applications built with known answers — planted weaknesses and clean controls whose correct verdict is recorded in advance. This page publishes the counts from those runs, single-pass and repeated, along with the model and index version each figure was produced under.
The control applications and their answer keys were authored by us. This is an internal calibration, not an independently-verified benchmark. The counts, model, index version and date are published so the arithmetic can be checked.
Method
How each run is produced
EngineThe production engine, unmodified. No test harness substitutions.
Check setThe entire published check set for the stated index version — not a reduced subset.
Model contextThe model reads the code and the framework requirements only. The answer key is not in context; it is applied afterwards, in scoring.
DirectionMeasured both ways — weaknesses caught, and clean controls wrongly flagged.
Unlabelled flagsA flag on a check not pre-labelled is reviewed and locked by a person before it counts either way.
Repeated runsThe same code is assessed several times, so a single lucky pass cannot inflate the result and verdict instability cannot hide.
Acceptance thresholds used to gate releases: detection ≥ 0.80, precision ≥ 0.70, false-positive rate ≤ 0.30.
Latest run
Overall result
· index · 3 control applications · 1 pass each · published 02 August 2026
Precision
87%
62 of 71 flags
False positives
27.3%
9 of 33 clean
Stability
—
single pass
MeasureSingle runRepeated (k-of-1)
Detection
62 of 64 · 97%
—
Precision
62 of 71 · 87%
—
False positives
9 of 33 · 27.3%
—
Stability
—
—
Post-run review
What this run showed
Every verdict that disagreed with our expected result was reviewed against the control application's source code before this record was published. The findings below are the outcome of that review.
The disagreements
In each disputed case the engine's verdict was correct and our expected
result was wrong. Our applications had been described against a single piece of
confirming evidence while the published control required more than that one thing —
a control was recorded as met because part of it was. Seven descriptions were
corrected across two reviews (CHK-INFRA-056,
CHK-AUTH-024, CHK-VALID-055,
CHK-COMMS-006, CHK-SESSION-007).
No planted weakness was altered, and no correction was made to improve a figure.
One application change
A clean control (CHK-APISEC-001) depended on an
unsettled reading of its standard. Rather than argue the label, the application was
changed so the control is unambiguously satisfied. A clean control whose correct
answer is debatable measures nothing.
Repeatability
Two runs of the same engine over identical inputs agreed on 65 of 73
verdicts. All disagreement fell in controls that call for judgement; every objective
category — cryptography, key handling, input validation, session, access control,
dependencies — returned identical verdicts in both runs.
Across both runs
All 64 known weaknesses were found in at least one run, and only one of
34 clean controls was flagged in both. A weakness found in either pass reaches the
report; a flag raised in only one pass is what the multi-pass escalation in a real
assessment resolves.
Summary
The engine performed as designed. Where it disagreed with us, it was
right. Our control applications continue to be extended so the testing stays
meaningful as the engine improves.
Breakdown
Detection by difficulty
Planted flaws are graded by how hard they are to catch. Easy is a surface pattern; hard requires reasoning across the code. Single-pass figures; the repeated-run column is the k-of-1 result.
COLUMNS: single-pass · k-of-1
Detection by severity
Calibration over time
Previous runs
Detection across every published run, oldest to newest. The run detailed above is the most recent point. Earlier runs are shown as configuration and outcome only — each was produced against the index and control-application revision current at its date, so the points are a record, not a like-for-like series.
PublishedIndexPasses
DetectionPrecision
2026-08-02
2026.09.1-master
1
62/64 · 97%
87%
2026-08-02
2026.09.1-master
1 · queued
61/61 · 100%
94%
2026-08-01
2026.09.1-master
1
55/61 · 90%
100%
2026-08-01
2026.09.1-master
1
53/58 · 91%
95%
2026-07-28
latest
1
55/58 · 95%
96%
2026-07-25
latest
3 · queued
42/56 · 75%
93%
Coverage
What these figures cover
These figures cover checks that manifest in code a small application can contain — injection paths, cryptographic choices, authentication and session handling, secrets, configuration and dependency posture.
Governance, process and documentation checks in the framework — policies, inventories, attestations — are assessed by the engine but cannot be scored by a planted-weakness corpus, so they are outside the scope of these numbers.
How much of the framework these figures speak for. The corpus exercises 85 of the 429 checks the engine assesses from source, and 55 of those carry a planted weakness — the only ones a detection figure can be measured on. The remaining 344 run on every assessment but have no control case here, so nothing on this page reports on them either way.
Building a control case per check is deliberate, incremental work: each one is an application written to contain a specific weakness. The corpus grows with the method rather than ahead of it, and this figure moves with it.
Definitions
How to read these figures
Detection (recall)Of the planted weaknesses, the share the engine caught. TP / (TP + FN).
PrecisionOf everything flagged, the share that was a real weakness. TP / (TP + FP).
False-positive rateOf the clean controls, the share flagged on a single pass. Rarely zero — which is why repeated runs are also measured.
Cumulative detectionAcross repeated runs, weaknesses caught in at least one pass (k-of-n).
Systematic false positivesFlags that persist across most or all runs — a repeatable misread rather than one-off noise. Tracked separately from the single-pass rate.
StabilityHow often the identical verdict returns when the same code is re-assessed.
Scope
Controlled measurements of the engine against applications authored and labelled by us. Not a statement about any customer's application, not a third-party audit, and not a regulatory certification. Every figure is tied to a named model, a published index version and a date, and is derived from the counts shown beside it.