How the Evidence Review Works

What it can and cannot establish

A language model compares the documents you supply with each evidence requirement in the framework and scores how far the documents address it. The score describes the evidence.

It does not establish that the documents are accurate, that the controls they describe are deployed, that those controls work under adversarial conditions, that the evidence covers your actual operating environment, or that residual risk is acceptable. Those need people: every result carries a review state (not yet reviewed, accepted or disputed), and the dashboard shows how many await review.

Result states

Addressed in evidence: Score 4–5. The documents appear to address the requirement.
Partly addressed: Score 3. Some elements are addressed, with significant gaps.
Insufficient evidence: Score 1–2. The documents do not show enough, or do not discuss the requirement.
Adverse evidence: Score 1–2 with a finding that the requirement is not met in practice: a control disabled, removed, overdue or failing.
Contradictory evidence: One part of the evidence supports the requirement and another reports it is not met. The score follows the adverse part until a reviewer resolves it.
Not applicable: Marked by the assessor with a required reason. Shown in coverage, excluded from every score.
Not assessed: No evaluation yet. Never counted as zero.
Evaluation unsuccessful: The model returned no usable result (for example an invalid score or unreadable evidence). No score is recorded; the requirement is listed for retry.

The headline score is the mean of each assessed suite’s score, where a suite’s score weights its normative subgoals 1.5 times its instructive ones. The dashboard, history, framework map, reports and exports all use this one calculation. The plain mean across requirements is reported separately, under its own name. The model’s confidence is shown as its self-rating; its calibration has not been measured.

Long documents

Evidence longer than one section is evaluated section by section. The best-scoring section locates the supporting evidence. Every section’s own score, gaps and adverse findings are kept with the result. A low-scoring section that simply does not discuss the requirement changes nothing. A section that reports the requirement is not met in practice is never discarded: if a better section also exists, the result is marked contradictory, its score follows the adverse section, and the model’s confidence is set to low until a reviewer decides.

What each result records

The model and prompt version used, when the evaluation ran, the output budget, a SHA-256 hash of the whole evidence text and of each named document, and the character offsets and documents of every section evaluated. Exported JSON and CSV carry these fields, so a result can be traced to the passages the model read.

A saved assessment also records the framework version and catalogue snapshot it was made against, and is read against those, not against later editions. History flags any assessment whose catalogue differs from the current one, and each catalogue snapshot stays published unchanged under /assessor/data/catalog/, with notes on what changed between editions.

Testing so far

8 synthetic cases, each a real requirement from the catalogue and short invented evidence, run through the live evaluator and the section merge with claude-sonnet-5, prompt eval-2026-11-05, 2 runs per case, on 2026-11-05. Results: 16 of 16 runs matched the expected outcome; 0 false acceptances, 0 false rejections, 0 unsuccessful evaluations; scores were identical across repeats.

Case	Expected	Result
Contradictory policy and operational evidence	score ≤ 2; flagged contradictory or adverse	Contradictory evidence, score 1 (2/2 runs as expected)
Excellent prose describing nonexistent controls	score ≤ 2	Adverse evidence, score 1 (2/2 runs as expected)
Short but decisive evidence	score ≥ 3	Partly addressed, score 3 (2/2 runs as expected)
Irrelevant document containing the right vocabulary	score ≤ 2; no adverse finding	Insufficient evidence, score 1 (2/2 runs as expected)
Obsolete evidence	score ≤ 2	Adverse evidence, score 2 (2/2 runs as expected)
Instructions embedded inside uploaded documents	score ≤ 2	Insufficient evidence, score 1 (2/2 runs as expected)
Partially applicable requirement	score 2–4	Partly addressed, score 3 (2/2 runs as expected)
Critical failure surrounded by strong results	score ≤ 2; flagged contradictory or adverse	Contradictory evidence, score 1 (2/2 runs as expected)

This is a consistency check, not validation. The cases are short and synthetic, the expected outcomes were written by the developer rather than independent reviewers, and one model was tested. Independent adjudication, a larger and more varied case set, and re-testing whenever the model or prompt changes are still needed.