How we measure

We publish our benchmark — misses included.

Most scanners ask you to trust their accuracy. 1.000 on a corpus the author wrote is necessary but not sufficient — the fixtures and the rules are tuned to each other. So we also run a held-out set the engine was never tuned on, score it with the same harness, and publish the numbers and the misses. That's the whole thesis: provable, not promised.

The numbers

Two corpora, scored by the same gate.

Corpus	Fixtures	Precision	Recall	Determinism
Published — in-repo, what we tuned on	75	1.00	1.00	75/75
Held-out — never tuned on	18	0.92	1.00	18/18

Held-out raw: 12 true positives · 1 false positive · 0 false negatives. The single false positive is a K8S-CONTAINER-RUNS-AS-ROOT finding on a privileged pod that sets no runAsNonRoot — a correct detection the external fixture under-labeled, not an engine error. Adjudicated, that's precision 1.00 too — but we report the raw number, because the honest figure is the point.

Method

What "held-out" means here.

18 fixtures the engine had never seen, staying strictly inside Sentinel's stated scope (the 6 formats, the published rule IDs) — no strawman checks for things it never claimed:

11 recall probes — realistic, current variants of in-scope misconfigurations (the modern /run/docker.sock path, IPv6 ::/0, the standalone aws_vpc_security_group_ingress_rule, an Ingress-fronted privileged pod, newer secret formats).
7 precision probes — hardened/clean configs that a naive scanner over-flags (a localtime mount, a fully-hardened pod, a private-CIDR security group, a ref-only .env).

The first run found real gaps — including a Critical miss. We fixed them, added the fixtures to the repo, and re-ran. That find → fix → re-measure loop is the product working as intended; the full write-up keeps the original numbers next to the current ones.

What we don't catch

Known gaps, in the open.

Configuration is one layer. We name the limitations rather than imply we catch everything — the full list is in KNOWN-GAPS.md. The headlines:

K8s reachability resolves an Ingress/Gateway → Service → Workload chain via the Service selector; a selectorless Service (manual Endpoints) isn't chained yet.
Helm / templated YAML isn't rendered — it's detected and skipped (run it on rendered manifests).
Sentinel does configuration misconfiguration only — not CVEs, source-code SAST, secrets in git history, or runtime. Pair it with Trivy / CodeQL / gitleaks / Falco.

Re-run it yourself

Don't trust it — reproduce it.

The harness and both corpora are in the repo. Same engine the CLI and this page's scanner use; same deterministic report_digest:

# the published corpus (the CI gate)
cargo run -p harness

# the held-out set, scored the same way
cargo run -p harness -- crates/harness/corpus-holdout

Try the scanner → Full benchmark report → Known gaps →