the published accuracy record

Benchmark

A small, v0 pilot: 25 clean libraries, 11 vulnerable training apps, 20 deployed web apps. Every caveat published, not just the flattering numbers.

Clean-pass rate, libraries

100%

▲ higher is better

25 of 25 clean repos passed clean, both tiers

Small sample: n=25, exact 95% CI on the underlying false-critical rate is 0–13.8%. Libraries only — applications are measured separately below. IN-SAMPLE: the scanner was fixed in response to these repos, so read it with the held-out number beside it, never alone.

Clean-pass, held-out libraries (2026-08-11)

55.6%

▲ higher is better

5 of 9 — against 100% on the in-sample corpus

n=9, 95% CI 21.2–86.3%. All 4 false alarms were deterministic — 3 secret-scanner, 1 dependency — the pure model false-critical rate was 0%. Four precision rules were then written against these four failures, which BURNS this corpus: re-scoring it now returns 0%, and that is in-sample, not a generalization result. Held-out generalization has been untested since, pending a second set frozen after the fixes.

Verdict-level recall, Deep tier

100%

▲ higher is better

11 of 11 vulnerable apps verdicted unsafe

Verdict-level, not finding-level — parity with a union of three open-source scanners, not detection superiority. 91.7% under strict pre-registration.

Secret clean-pass rate, applications

100%

▲ higher is better

20 of 20 deployed apps passed clean, free deterministic tier

Up from a raw 65% via general precision rules, never per-repo allowlists — but the rules were tuned on these same 20 apps, so this is in-sample. Held-out validation is an open v1 item.

Application clean-pass, Deep tier

80%

▲ higher is better

16 of 20 passed green — the other 4 were later-fixed false alarms, not missed vulnerabilities

Model false-critical rate 0% — no critical on model opinion alone. All 4 not-green verdicts were adjudicated over-escalated on dependency drift; root causes since fixed, re-run outstanding.

Free vs. Audit

nittim ships two tiers. Free scan runs deterministic checks only — a secret scanner, a dependency-CVE lookup — no model in the verdict. Audit adds Claude Opus reasoning across the full framework, including semantic bugs (SQL injection, auth bypass, SSRF) no secret or CVE points to.

Tier	Clean-pass rate	Recall
Free — deterministic scan	100% (25 of 25)	63.6% (7 of 11)
Audit — Opus	100% (25 of 25)	100% (11 of 11)

On 2026-08-11 the free tier's secret scanner produced a false critical on 3 of 9 held-out repos — 33.3% (95% CI 7.5–70.1%), against 0% in sample. The deterministic scanners, not the model, are where every false alarm in this benchmark came from. Rules were written to close those cases and each corpus now scores 0% — but those rules were written against those repos, so that 0% is in-sample. The second frozen set has since been run: on 12 fresh libraries, pre-registered at the public tag corpus-heldout2-frozen-2026-08-19 before anyone ran them, the secret-scanner false-critical rate was 8.3% (1 of 12, 95% CI 0.2–38.5%). The point estimate is four times better and the sample cannot prove it: those bounds overlap the earlier 7.5–70.1% across nearly their whole range, so the honest claim is that the fixes did not fail on fresh code, not that they measurably improved. The one failure was the same defect class as before — every verdict-forcing hit sits under t/unit/ — a test root spelled in a way the non-production path vocabulary has not seen. Separately, a manifest-level re-verification on 2026-08-19 found 3 of 9 repos satisfy the corpus's own dependency predicate, 2 fail it, and 4 cannot be evaluated at all because they ship no lockfile — which weakens the corpus, not the secret figure above: that predicate is about dependencies and these are secret findings.

Free-tier mean hard-evidence findings per vulnerable app: 4.27. The free tier missed 4 of 11 apps — digininja/DVWA, juice-shop/juice-shop, stamparm/DSVW, OWASP/Vulnerable-Web-Application — because none of them had a committed production secret or a runtime high/critical CVE for the scanners to find. This is the precise gap Audit exists to close, not a defect in the free tier.

Score separation, Deep tier

On the Deep tier, the readiness/safety score (0–100, higher is safer) separates the two populations clearly: a mean of 89.3 on clean libraries versus 30.3 on intentionally-vulnerable apps. The two populations are maximally contrasted by construction — audited libraries against deliberately broken training apps — so this gap describes the corpus, not how well nittim discriminates a real app with two or three embedded issues from clean code, which this benchmark does not yet measure.

Applications — 20 mature, deployed web apps

nittim audits applications, not just libraries. This axis measures both tiers against 20 mature, deployed open-source web apps (Ghost, n8n, and others) across several stacks. A real dependency tree means a not-green verdict here is not automatically a false alarm — every one was checked by hand against OSV and the real lockfile.

Metric	Result
Secret clean-pass rate (free tier)	100% (20 of 20)
Runtime-CVE catches (reported separately — several driver findings were later adjudicated down, see below)	3 of 20
Clean-pass rate (Deep tier, Opus)	80% (16 of 20)
No critical on model opinion alone (Deep tier)	100% (20 of 20)

The unflattering number, published on purpose: of the five dependency findings that drove the 4 not-green verdicts, 2 were sound, 2 overstated, and 1 flatly false (a critical raised against an already-patched version because the scanner read the declared range instead of the installed lockfile version). The model never invented a critical — the failure mode was a deterministic layer escalating ordinary dependency drift too aggressively. Both root causes are fixed; the table reports the run as measured, before those fixes.

Application-axis caveats

Same rule as everywhere on this page: the qualifier ships with the number.

Mature apps are a lower bound, not the target class

The 20 apps are well-maintained, widely-deployed open-source projects — not the younger, thinner-reviewed, AI-assisted code nittim is built to audit. A convenience sample of the best-run codebases on GitHub understates real-world error in both directions. Read every number on this axis as a lower bound.

One flatly false critical — found, disclosed, fixed at the source

Of five dependency findings driving the four not-green verdicts, two were sound, two overstated, and one flatly false: a critical raised against a package version the lockfile had already patched, because the scanner read the declared version range instead of the installed version. That scanner bug is fixed (installed versions are now resolved from pnpm/yarn lockfiles), so the false critical no longer fires — but the table above reports the run as measured, not as it would look after the fix.

Over-escalation was the real failure mode — since retuned

All four not-green verdicts penalized ordinary dependency drift rather than a reachable defect in the app's own code. The pure model never invented a critical; the deterministic dependency floor was too aggressive. It has since been retuned so ordinary dependency drift no longer forces the harshest verdict, while a genuine critical still does. A post-fix re-run on held-out apps is an outstanding v1 commitment.

Single pass per app — verdict variance unmeasured

Each app was audited once. One model-only not-green (Metabase) flipped to green on an independent re-audit, so run-to-run variance is real and these figures understate it. The v1 commitment is to report verdict variance across N≥3 runs; until then, treat any single verdict near a band boundary as potentially unstable.

The adjudicator shares a model family with the system under test

Every adjudication was performed by an Opus-family model — the same family nittim ships — because no strictly stronger independent judge exists. Verdicts were decided against OSV and real lockfile ground truth rather than model opinion, which blunts but does not eliminate shared-family blind spots. Discount the soundness numbers accordingly.

Read this before the numbers above

This is a v0 pilot on a small, hand-built corpus. Every limitation we found is listed here, next to the numbers it qualifies — not in a footnote.

Small sample — point estimate, not a tight bound

n=25 clean repos, n=11 vulnerable apps. At n=25 with zero observed false criticals, the exact 95% confidence interval for the true false-critical rate is 0%–13.8% — a single false critical would move the observed rate to 4%. This is a v0 pilot, sized to catch gross failure modes, not to certify a precise rate.

Library FCR and application FCR are different measurements

The 0% false-critical rate above is measured on mature open-source libraries, frameworks, and CLI tools. Web applications — the code nittim is built to audit — are measured separately on a 20-app corpus (see the application axis below), where a deployed app's real dependency tree makes "false alarm" a genuinely harder call. Neither corpus covers the young, AI-assisted codebases nittim actually targets; both are lower bounds.

Recall is verdict-level, not finding-level

A 100% recall figure means every vulnerable app in the corpus was verdicted unsafe — not that every documented vulnerability in each app was individually located. On this corpus, verdict-level recall is parity with a one-line rule unioning the outputs of three open-source scanners (Gitleaks, Trivy, Semgrep), not evidence of detection superiority over them.

The juice-shop-ctf disclosure

A 12th vulnerable-corpus entry was the only miss in the initial run, then removed afterward on the grounds that it is a CTF-challenge generator, not a vulnerable application. The removal is defensible but happened after the result was known, so we publish both numbers: 100% (11 of 11) on the corpus as currently defined, and 91.7% (11 of 12) under strict pre-registration with that entry kept in.

The juice-shop-ctf entry, both ways

Re-frozen 11-app corpus: 100% (11 of 11). Strict pre-registration with the disputed 12th entry kept in: 91.7% (11 of 12).

Confidence interval, not a point claim

The exact 95% confidence interval on the clean-corpus false-critical rate is 0%–13.8% at n=25. Treat 0% as a small-sample point estimate, not a guarantee.

The full corpus and methodology are public: github.com/ilanwolberger/fis-benchmark. For real public repos, see Verified — latest audit came back green. See Data & Trust for what nittim stores, or return to the home page.