Verify it yourself

Proof — verify our claims yourself

We sell falsifiability, so the selling is machine-checked. Every public claim binds to resolvable evidence; a skeptic can reproduce every check below on a fresh checkout. This page is itself generated from those sources and fails CI if it drifts.

See it run (< 60s, real output)

The recording is of the exact commands in § 5, nothing staged. Its commands are re-executed in CI (.github/workflows/proof-demo.yml), so a demo that showed something broken would turn CI red — the recording cannot drift from current behavior.

1. Every public claim binds to evidence

Rendered from docs/CLAIMS.md. A <!-- claim:ID --> marker in README/docs must resolve to a backed entry here with a resolving evidence pointer, or task check-claims fails the build.

Claim	Kind	Evidence	Resolves
The release process is documented as an inheritable runbook + succession doc, and the project’s bus-factor (trailing-90-day distinct human reviewers) is tracked and reported truthfully — currently 1, not implied to be more.	qual	docs/succession.md#trailing 90 days	✅
177 commands.	quant	src/scripts/check_artefact_count_messaging.ts#Artefact-count messaging gate	✅
The retrieval economy cuts always-loaded context tokens ~65.6% measured against the FULL always-loaded projection — 98,529 → 33,897 tokens (eager rule load + skill/command descriptions + MCP schemas, thin-flipped). Method: agent-config benchmark over the pinned token baseline; the baseline is the honest “what the user pays if everything loads eagerly”, NOT a synthetic full-corpus strawman (council Q4).	quant	internal/bench/reports/token-baseline.json#eager_rule_load	✅
On a weak host (claude-haiku-4-5) the package produces a significant, placebo-controlled discipline lift on scope/downstream traps; on a strong host the same measurement is a published null — the package transplants discipline a weak model lacks, not model intelligence.	quant	docs/benchmark.md#weak-host-specific	✅
The non-coding domain skills (finance/founder/ops/content) are forged on TS/PHP and labeled unvalidated until they pass a sourced domain-truth fixture; no public prose implies proven domain correctness, and the validated count is CI-ratcheted.	qual	src/scripts/domain_soundness_status.ts#checkRatchet	✅
On the READ-ONLY FAN-OUT slice family, tier-downshifted subagent dispatch (lite/haiku vs session-tier-proxy sonnet) nets a ≥30% USD-weighted token-cost reduction at held quality — measured 2026-07-08 (n=10 paired live dispatches, 20 telemetry lines): 10/10 exact-match on BOTH arms, 29.4% fewer raw tokens, 76.5% USD-weighted cost reduction at the 3x haiku↔sonnet price ratio. FAMILY-SCOPED — the mechanical-edit family is unmeasured and its downshift (incl. the deferred tier downgrades of existing units) stays gated. Negative control held: an open-ended synthesis/unknown slice never resolves below the session tier (inferSliceTier → medium/inherit, never lite).	quant	internal/bench/routing-downshift/results-2026-07-08.md#FAMILY-SCOPED PROVE	✅
The lift-carrying essential cut (kernel + downstream-changes) keeps a significant weak-host discipline lift at a fraction of the full load’s tokens, and the lift is FAMILY- and HOST-SCOPED — measured on three hosts: claude-haiku-4-5 (weak) shows the family-scoped lift (trapE 0.533→1.000, 7/7 discordant, corpus cost 1.71x); claude-sonnet-4-6 (strong) is a ceiling null; gpt-5-mini (non-Claude weak, codex prompt-prepend surface) FAILED replication with headroom (corpus Δ=+0.024 p=0.70, capability trend n.s. — no harm claimed, injection-surface confound documented). Therefore discipline_profile: auto enables the lift only where measured (vendor-granular unknown_defaults). Non-claims — the balanced router profile was removed after a NULL measurement (p=0.81, n=24); no full-tier recommendation exists; no cross-vendor lift is claimed.	quant	docs/benchmark.md#REPLICATION FAILED	✅
Behavioural-eval coverage is measured per tier and CI-ratcheted so it can only rise; the current coverage and its gap are published, never implied as “264 evaluated skills”.	qual	src/scripts/skill_eval_coverage.ts#checkRatchet	✅
A hand-rolled, dependency-free BM25 + trigram lexical index resolves the “recalls but does not rank” gap: on the retrieval-precision corpus (9 keyword-overlapping-confuser tasks) it drives the mean top tie-set from 3.333 (the _score bucket scorer) to 1.0 — every needed decision uniquely top-ranked — with precision@1 and precision@5 unchanged at 1.0. Method: deterministic, model-free re-ranking of the SAME retrieved entry set; both scorers measured over the identical store via measure_lexical_ranking.ts.	quant	internal/bench/reports/lexical-ranking.json	✅
The whole layer is compiled into host agents with zero runtime daemon.	qual	docs/contracts/no-runtime-boundary.md#file-first, no-runtime suite	✅
103 governed rules.	quant	src/scripts/check_artefact_count_messaging.ts#Artefact-count messaging gate	✅
On a deterministic multi-session recall corpus, the memory substrate produces a measured, placebo-controlled recall lift — memory-on 27/27 vs no-memory 10/27 and vs equal-byte placebo 9/27 (claude-haiku-4-5, n=9 tasks x 3 seeds, sign test p=0.031 for BOTH pairings). Scoped honestly: this is the context-value upper bound (perfect retrieval on a one-fact-per-task corpus), not retrieval precision under a large store.	quant	internal/bench/reports/second-brain-delta.json	✅
Removing the perfect-retrieval assumption, the substrate’s REAL keyword retrieval recalls the needed decision into the top-5 under keyword-overlapping confusers (precision@5 9/9) and the model disambiguates it from the co-injected confusers — retrieval-on 27/27 vs no-memory 5/27 and vs equal-count placebo 5/27 (claude-haiku-4-5, 9 tasks x 3 seeds, sign test p=0.008 both). Named limit: retrieval RECALLS but does not RANK (mean tie-set 3.3, ties broken by store order, not relevance) — the discrimination gap that motivates the SQLite-FTS5 activation path (ADR-116) at larger scale.	quant	internal/bench/reports/second-brain-retrieval.json	✅
Every artifact the package ships — source AND the condensed projection that reaches consumers — is machine-scanned in CI for hidden-Unicode, mixed-script-confusable, and instruction-smuggling payloads (the rules-file-backdoor class); a finding blocks the release before npm publish, not just the merge.	qual	.github/workflows/publish-npm.yml#lint_agent_security	✅
270 skills (README hero + feature list).	quant	src/scripts/check_artefact_count_messaging.ts#Artefact-count messaging gate	✅
Removes only its own keys from a shared host config (matched by JSON-pointer + SHA-256), never a neighbour tool’s entries.	qual	docs/contracts/install-layout.md#JSON-pointer	✅

16 backed claim(s) — all evidence pointers resolve in CI.

Artefact counts in public prose (skills, commands, governed rules, guidelines, personas) are generated from source and CI-drift-checked: update_counts.ts writes the numbers, check_artefact_count_messaging.ts fails the build on any count-shaped prose mention that drifts from the source count — or on two different numbers for the same artefact kind.

We also publish our debt: 1 claim(s) are logged as unbacked inventory in the ledger — not yet bound, and therefore not allowed to carry a marker in public prose. Hiding them would be the opposite of the point.

2. We publish honest nulls

Benchmark results — including the runs where the package changed nothing — live in docs/benchmark.md. We do not delete a measured null to make a number look better; the null is the evidence of honesty.

Behavioural-eval coverage — the honest baseline. Skill quality is only as good as its measurement. Today 2 of 264 skills carry a behavioural evals.json, and the highest-traffic / highest-cost tiers (default-surface + rich + routers, 0 of 35) are not yet covered. We publish that gap rather than imply “264 evaluated skills”: coverage is measured per tier (./scripts-run src/scripts/skill_eval_coverage) and CI-ratcheted so it can only rise — a merged change can never lower it. Authoring the priority-tier evals is gated on per-case human ratification (a generated assertion that checks the wrong property is worse than none), so the number grows deliberately, not overnight.

Non-coding domain soundness — scoped, not proven. The finance / founder / ops / content profiles sell concrete domain value (DCF, runway, RICE, incident command, messaging), but the skills are forged on TS/PHP — “promising, not proven” off those stacks. A disclaimer floor bounds liability, not correctness: a skill can be format-correct, disclaimered, and still embed a wrong domain assumption. Today 9 of 20 default-surface domain skills carry a sourced domain-truth fixture (./scripts-run src/scripts/domain_soundness_status); the rest are labeled unvalidated and the validated count is CI-ratcheted. The fixtures landed so far are the deterministic targets (runway, unit-economics, DCF, forecasting, scenario band/sensitivity), whose answer keys are computed from cited standard formulas — never the skill’s own output; the rubric targets (incident command, messaging, fundraising, editorial) need domain-competent grounding and remain unvalidated, so validation lands deliberately — the gap is published, never implied away.

Second-brain substrate — measured recall lift, honestly bounded. On a deterministic multi-session recall corpus, the memory substrate beats a no-memory baseline AND an equal-byte placebo: memory-on 27/27 vs no-memory 10/27 vs placebo 9/27 (claude-haiku-4-5, 9 tasks × 3 seeds, sign test p = 0.031 for both pairings) — a real, placebo-controlled lift (internal/bench/reports/second-brain-delta.json). Scoped, not oversold: this is the context-value upper bound (perfect retrieval on a one-fact-per-task corpus), NOT retrieval precision under a large store; the lift concentrates exactly where memory is the only source and ties where the prompt self-contains the fact. Boundary vs a human PKM, and why the Obsidian export stays declined, are in docs/second-brain-scope.md.

A follow-up removed the perfect-retrieval assumption: against a store of keyword-overlapping confusers the REAL retrieval recalls the needed decision into the top-5 (9/9) and the model disambiguates it — retrieval-on 27/27 vs no-memory 5/27 and vs placebo 5/27 (p=0.008 both). The honest limit: the keyword scorer recalls but does not rank (mean tie-set 3.3), which is the SQLite-FTS5 activation signal (ADR-116) at scale.

3. Known limits (published, witness-tested)

Each limitation below is stated by the skill itself and carries a witness test that reproduces it. If the limitation is ever fixed, its witness goes red — so a “Known limit” can never quietly become false.

skill	known limit	witness
check-refs	Only validates references to known-root paths (docs/, skills/, rules/, commands/, contexts/, personas/, …). A relative-path link such as ./sibling.md or ../foo.md is not matched, so a broken relative link is never reported.	tests/scripts/witness/check_refs_relative_gap.test.ts

4. What is checkable — us vs. the category

This is not a takedown — the point is the last column. For each claim, our evidence is a pointer you can resolve on a fresh checkout; the wider category is described only by what is publicly observable, never a named competitor and never a counter-claim to anyone’s headline number. A claim is “checkable” only when its our evidence pointer resolves — CI enforces that (task check-comparison), so this column can never lie.

Claim	Our evidence	The category	Checkable?
No runtime — no background daemon, no state database, no auto-write memory.	docs/contracts/no-runtime-boundary.md#file-first	Swarm-runtime tools in this category ship a background process and/or a state store by design; that is an architectural fact of the runtime approach, not a defect.	✅
Capability/benchmark results are published — including the runs where the package changed nothing.	docs/benchmark.md	A category headline figure (e.g. an ‘84.8%’ score) appears across marketing surfaces with no reproducible methodology published — so it cannot be verified either way.	✅
Every public claim binds to machine-checked evidence.	docs/CLAIMS.md	Marketing claims in this category are not bound to a machine-checked ledger a reader can reproduce.	✅
Skills publish their own known limits, each with a witness test.	tests/scripts/witness/check_refs_relative_gap.test.ts	Published known-limitation surfaces backed by reproducing tests are not a standard artifact in this category.	✅

5. Verify it yourself

On a fresh checkout, reproduce the claims above:

task check-claims   # every markered public claim binds to resolvable evidence
task check-refs     # no broken internal references
task check-skill-gaps    # every logged known-limit cites a real witness test
task check-comparison    # every comparison-table "our evidence" pointer resolves
./scripts-run src/scripts/skill_eval_coverage         # behavioural-eval coverage, per tier
./scripts-run src/scripts/skill_eval_coverage --check # the ratchet: coverage may not drop
task build-proof-check   # this page is in sync with its sources

If a claim ever loses its binding, or this page drifts from the ledger, CI goes red. Reproducibility is the proof.