Method
Correctness measurements
The suite compares what the reasoner derives against expectations authored from statutory text, and it injects deliberate faults into the graph to find out which errors the suite is blind to. Both results are published here in full, including the parts that are unfavourable. There is no single accuracy figure on this page: an average over unlike assertions hides the one number that matters, which is how often a law is claimed that does not apply.
Fixture and version
- Graph version
- 2.21.0
- Graph nodes
- 700
- Cases in the fixture
- 64
- Expectations met
- 56 met · 8 unmet
Graph content hash 7d16100080205127e2e7cdf9014824953f410de0bba6ec39c1f269f85666681b · dated 2026-09-22. Every case, expectation, negative reason and source URL lives in one file: src/testing/golden/cases.ts.
9 of 64 cases are not independently verified: their expectations do not yet carry an article-level source with a URL and a retrieval date for every value. They still run and are reported, and they are excluded from the sourced counts above.
Cases per category
Composition is part of the result. 28 of 64 cases assert that something does not apply, 18 of 64 are one half of a paired case where only one variable differs, and 8 of 64 turn on the status of an instrument in time rather than on its subject matter.
| Category | Cases | Sourced | Not independently verified | Met | Unmet | Accepted |
|---|---|---|---|---|---|---|
| Externally-authored scenarios | 34 | 30 | 4 | 31 | 3 | 3 |
| Jurisdiction pairs | 6 | 5 | 1 | 6 | 0 | 0 |
| Temporal status | 8 | 8 | 0 | 8 | 0 | 0 |
| Adversarial input | 5 | 5 | 0 | 5 | 0 | 0 |
| Degenerate input | 2 | 2 | 0 | 2 | 0 | 0 |
| Confidence calibration | 4 | 0 | 4 | 2 | 2 | 2 |
| Role pairs | 5 | 5 | 0 | 2 | 3 | 3 |
Metric matrix
- Exclusion precision
- 100.0%
- Inclusion recall
- 100.0%
- Flag recall
- 83.3%
36 negative assertions — how often an instrument the fixture says does not apply was nevertheless presented as applicable
95 containment assertions — how often a duty the fixture requires was actually derived
12 flag assertions — contested, deferred, stayed, not-in-force, role-dependent, insufficient input, prohibited
Mutation kill rate, per operator
A green suite may be green because the graph is right or because the cases assert nothing that could go wrong. The harness settles that: it injects one plausible graph fault, re-runs the suite, and asks whether any previously-met case now fails. Faults that nothing notices are named below. Recorded 2026-09-22 against graph v2.21.0: 8 mutants per operator, 77 of 700 nodes actually asserted about by the fixture.
| Fault operator | What it simulates | Injected | Detected | Kill rate | Candidates (asserted / total) |
|---|---|---|---|---|---|
applies-in-flip | an instrument is mapped to the wrong jurisdiction (copy-paste across a border) | 8 | 8 | 100% | 29 / 170 |
status-force-in-force | a pending, withdrawn or repealed instrument is presented as applicable law | 8 | 3 | 38% | 5 / 25 |
triggers-delete | a duty-creating edge is lost in an edit, so a real obligation disappears | 8 | 0 | 0% | 300 / 468 |
triggers-spurious | an obligation is attached to a use case it does not govern (over-claiming) | 8 | 0 | 0% | 25 / 78 |
tier-shift | a classification lands one step off — the single most consequential error | 8 | 4 | 50% | 25 / 78 |
gate-invert | a boolean gate on a node is inverted (profiling, scoring, always-applicable) | 8 | 5 | 63% | 54 / 237 |
crosswalk-swap | a crosswalk claims equivalence where the mapping is only an overlap | 8 | 0 | 0% | 12 / 21 |
Surviving mutants — the blind spots
Each line is a specific graph error the suite does not currently detect. This list is the useful output of the harness; the percentages above are only its summary.
status-force-in-force — a pending, withdrawn or repealed instrument is presented as applicable law
- · reg-aild status "withdrawn" → "in-force" (AI Liability Directive (withdrawn))
- · reg-colorado status "repealed — reenacted by SB 26-189 (2026); never took effect" → "in-force" (Colorado AI Act)
- · ae-pdpl status "enacted-not-yet-applicable" → "in-force" (Federal PDPL — Decree-Law 45/2021 (AE))
- · ca-bill-c36 status "pending" → "in-force" (Bill C-36 — Protecting Privacy and Consumer Data Act (CA))
- · eg-pdpl status "enacted-not-yet-applicable" → "in-force" (PDPL 151/2020 (EG))
triggers-delete — a duty-creating edge is lost in an edit, so a real obligation disappears
- · delete uc-accessibility-remediation triggers gb-psbar
- · delete uc-accessibility-remediation triggers reg-eu-digital-accessibility
- · delete uc-accessibility-remediation triggers reg-us-ada-title2-web
- · delete uc-accessibility-remediation triggers reg-us-hhs-section504
- · delete uc-ad-targeting triggers reg-aiact
- · delete uc-ad-targeting triggers reg-dsa
- · delete uc-ad-targeting triggers reg-ecoa
- · delete uc-ad-targeting triggers reg-gdpr
triggers-spurious — an obligation is attached to a use case it does not govern (over-claiming)
- · add uc-accessibility-remediation triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-admissions-utility triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-biometric-access triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-biometric-identification triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-chatbot triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-codegen triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-companion-minors triggers art-13 (Art. 13 — Transparency to Deployers)
- · add uc-confidential-summary triggers art-13 (Art. 13 — Transparency to Deployers)
tier-shift — a classification lands one step off — the single most consequential error
- · uc-accessibility-remediation classified_as rc-limited → rc-high
- · uc-admissions-utility classified_as rc-limited → rc-high
- · uc-biometric-identification classified_as rc-high → rc-prohibited
- · uc-companion-minors classified_as rc-high → rc-prohibited
gate-invert — a boolean gate on a node is inverted (profiling, scoring, always-applicable)
- · br-pl2338.operatorBinding false → true (PL 2338/2023 AI framework (BR))
- · ca-aida.operatorBinding false → true (AIDA — Bill C-27 (CA))
- · reg-aild.operatorBinding false → true (AI Liability Directive (withdrawn))
crosswalk-swap — a crosswalk claims equivalence where the mapping is only an overlap
- · art-15 overlaps_with std-iso27001 → equivalent_to
- · art-15 overlaps_with std-nist600 → equivalent_to
- · art-27 overlaps_with std-iso42005 → equivalent_to
- · art-72 overlaps_with nis2-23 → equivalent_to
- · art-9 overlaps_with std-iso23894 → equivalent_to
- · art-9 overlaps_with std-nist → equivalent_to
- · gdpr-22 overlaps_with art-14 → equivalent_to
- · gdpr-33 overlaps_with nis2-23 → equivalent_to
Signed warrants on the law-to-architecture seam
0 of 251 warrantable edges carry a signed warrant. 909 further edges are engineering judgment by design and carry none.
implemented_by: 0 of 81evidenced_by: 0 of 79supported_by: 0 of 43operationalized_by: 0 of 27overlaps_with: 0 of 18equivalent_to: 0 of 2conflicts_with: 0 of 1
RAIN's editorial mapping from a legal duty to a technical measure — reasoned from the cited text, not stated by it. Not evidence, and not a compliance opinion.
Currency of the instruments
87 of 180 instruments carry a machine-readable date. The rest carry prose, or nothing — and prose cannot be compared against today.
27 of 37 status values outside “in force” cite an official text. Such a status claims a process — a withdrawal, a repeal, a pending procedure — and until this release not one of them was sourced. The gap that exposed it: the graph called Australia’s mandatory guardrails “withdrawn”, and no official text says so. They were consulted on in 2024 and never legislated; there was no withdrawal to report.
20 of 180 instruments carry a curator-confirmed lastVerified — read against a real, cited source and applied under --confirm, not the 2026-08 bulk-import stamp every other node still carries. The rest are not stale by this number; they simply have not been individually re-checked yet, and the bulk stamp does not claim otherwise. This is deliberately not framed as a percentage: it starts low, on purpose, and only grows as a deep-search Class G batch is confirmed.
10 nodes are marked unverified by the curator. They are named here because an open question needs an address:
id-draft-perpres-ai— Draft Presidential Regulation on AI (ID) (pending)ae-pdpl— Federal PDPL — Decree-Law 45/2021 (AE) (enacted-not-yet-applicable)ke-dpa— Data Protection Act 2019 (KE) (in-force)ke-ai-strategy— Kenya AI Strategy 2025–2030 (voluntary)eg-pdpl— PDPL 151/2020 (EG) (enacted-not-yet-applicable)eg-ai-strategy— Egypt AI Strategy 2025–2030 (voluntary)rw-dp-law— Data Protection Law 058/2021 (RW) (in-force)rw-ai-policy— National AI Policy 2023 (RW) (voluntary)gb-crime-policing-2026— Crime and Policing Act 2026 (GB) (in-force)reg-fda-samd— FDA oversight of Software as a Medical Device (US) (unverified)
A flag on a node is not a disclaimer over the whole graph. It marks the one claim the curator has not confirmed against the primary source, and it now reaches the reader: until this release a flagged node that also carried a status rendered like any confirmed one. Taiwan’s AI Basic Act was such a node, and its status was wrong.
Failures accepted with an owner
A case whose expectation is not met may be accepted rather than fixed, but only in writing: with an owner, the date it was accepted, and the reason. The suite asserts that every case marked this way still fails, so an accepted failure cannot quietly become a passing one.
s-legalresearch· Externally-authored scenariostier lands on rc-limited; role-dependent reading not surfaced as a flag
Owner kg-curators · accepted 2026-08-15
s-admissions-utility· Externally-authored scenariosauthored contested reading not surfaced as a flag
Owner kg-curators · accepted 2026-08-15
c-market-thin· Confidence calibrationSince the reg-aiact territorial-scope fix (production review item #19/#23, 2026-09-09) RW carries zero regsInScope/articles, so chainMin has no hop to weaken it and defaults near-perfect (0.98) — coverage.factor (0.8) alone does not pull it below c-market-full's genuinely-sourced but longer chain. 'nothing found' is currently scored as high confidence, not low; needs a confidence-model decision, not a reasoner patch.
Owner kg-curators · accepted 2026-09-09
c-market-full· Confidence calibrationPaired with c-market-thin: since the reg-aiact territorial-scope fix RW's empty derivation trivially maxes chainMin, so this genuinely-sourced EU case (32 articles, some inevitably resting on a weaker single hop) now scores check-worthy against RW's robust — needs a confidence-model decision (should an empty chain default low, not 0.5/near-1.0?), not a reasoner patch.
Owner kg-curators · accepted 2026-09-09
r-hr-provider· Role pairsprovider/deployer duty split not applied to Art. 43
Owner kg-curators · accepted 2026-08-15
r-hr-deployer· Role pairsprovider/deployer duty split not applied to Art. 43
Owner kg-curators · accepted 2026-08-15
r-rebrand-user· Role pairsArt. 25 role shift not applied
Owner kg-curators · accepted 2026-08-15
g-scope-drift-before· Externally-authored scenariosThe reasoner tiers this as high risk on the employment keyword alone ('HR handbook'), although the described system evaluates no person and matches no Annex III point — the over-triggering failure the deep-search audit predicted. Recorded as a disclosed recall/precision gap on the tier, not as a claimed duty: the case asserts no negative expectation, so nothing wrong is published while the detector is narrowed.
Owner RAIN editorial · accepted 2026-08-17
What this does not cover
- It is not legal advice and not a legal opinion. The expectations were authored by reading statutory text; agreement with them is agreement with a reading, not a determination by a competent authority or a court.
- No independent third party has checked this fixture. Where a case is marked not independently verified, no article-level source is attached to every expected value yet.
- 64 cases are a sample, not a population. The graph carries 700 nodes and the fixture asserts about 77 of them. A fault in a node no case asserts about cannot be detected here — by construction, not by oversight.
- The mutation figures describe a sample of faults (8 per operator), chosen from the asserted region. A 100% kill rate for an operator would be a statement about those mutants only.
- Crosswalk faults remain 0/8 killed, and the zero is real — a declared assertion-vocabulary gap, unchanged by the latest release. The crosswalk-swap operator turns an overlaps_with mapping into equivalent_to. The derivation path the fixture asserts over never reads the relation type — the crosswalk view is a separate surface — so no case can observe the change. This is a coverage gap in the assertion vocabulary, not a fixture-authoring gap, and it is reported rather than papered over with a case that cannot fail.
- Nothing here measures the law itself being current. Instrument status and dates are tracked separately, per node, with a retrieval date and a source; the suite only checks that the derivation respects the status the graph records.
- Confidence bands are ordinal. Cases may assert that one situation is banded strictly lower than another; no case asserts an absolute confidence number, because that would be a claim about the implementation rather than about the law.
- Coverage of markets is uneven. Most cases run against EU and US expectations; other jurisdictions in the graph carry far fewer, or none. Every
Conclusionnow carriescoverageGaps, so a reader can check any specific answer rather than trusting only this page-level admission — an unregistered market code, a registered one with no instrument node yet, and a thin-coverage market this case found nothing in are each named, distinct from a market that was actually checked and legitimately has no applicable instrument.
Related: the public track record logs every correction made after publication, and sources and references lists the instruments behind the graph.