Discipline-axis benchmark

Discipline-Axis Wrapper-Lift Benchmark (v2)

Curated composite of two pinned reports — the weak-host result and the strong-host result side by side. It is deliberately NOT auto-generated from the single latest ab-v2 report: a single-report render would bury one host’s finding (a strong-host null would erase the weak-host lift, and vice versa). Regenerate the pinned sections with task bench:ab:v2:diff — each <!-- pinned:<id> --> region renders from ITS pinned report (docs/benchmark.pinned.yml); curated prose outside the markers is never touched (-- --check is the drift gate). Pinned sources: weak host = internal/bench/reports/ab-v2/2026-06-15T03-52-35Z-ab-v2-paired.json; strong host = internal/bench/reports/ab-v2/2026-07-05T07-00-31Z-ab-v2-paired.json; cost-factor sweep = internal/bench/reports/ab-v2/2026-07-07T05-35-14Z-ab-v2-paired.json.

Honesty labels (read first)

1. Wrapper-lift on a fixed host (claude-haiku-4-5), NOT model-vs-model. Measures what the agent-config package does to ONE host model on a neutral fixture — not a capability ranking.
2. Discipline axis, not capability. The headline is the discipline delta (did it stay minimal / verify / ask / not destroy / update downstream), not whether the goal was achievable.
3. PILOT — low statistical power (N=2 tasks × 12 seed(s)). Directional only.
4. Paired design, errored runs excluded; McNemar (capability) + Wilcoxon signed-rank (discipline) + effect sizes.
5. Not comparable to SWE-bench / GAIA / Fable scores — a different question entirely.

Weak host (claude-haiku-4-5) — Gate verdict: PASS

• capability lift significant: False
• discipline lift significant: True
• status-bucket better (package vs vanilla): False

Measurable discipline lift (significant). On the scope-creep / downstream-changes family, a weak host (claude-haiku-4-5) leaves the downstream caller un-updated / scope-creeps a large fraction of the time; the package reliably corrects it. The lift is significant on the discipline axis (Wilcoxon p<0.05, every discordant pair favouring the package) AND beats an equal-length inert-prose placebo — so it is the package’s content (its downstream-changes/scope-control rules), NOT mere prompt-length, that helps. Honest scope (empirically bounded): the lift is weak-host-specific — a CLEAN strong-host run (claude-sonnet-4-6, same tasks, 8 seeds) scored vanilla = package = placebo = 1.00 (no headroom, package redundant). So the package helps a WEAK model that lacks the discipline; a strong model already has it. This matches the package’s design thesis (strong hosts self-apply discipline; weak hosts benefit fully). Discipline axis, not capability (both arms make the primary change); this task family (scope/downstream), not a universal claim. It improves solution discipline, not model intelligence.

package lift — package vs vanilla (n=24 pairs)

Table 1 — capability axis (expected near-flat by design)

metric	baseline	treatment	test
pass-rate	100%	100%	McNemar p=1.0, h=0.0

Table 2 — discipline axis (the lift)

metric	baseline	treatment	Δ	test
mean discipline	0.333	1.000	+0.667	Wilcoxon p=0.0005, rb=1.0 (n≠0=16)

Table 3 — cost axis (mean tokens/run, non-errored)

metric	baseline	treatment	Δ
mean tokens	90,534	992,044	+901,510

attribution (content vs length) — package vs placebo (n=24 pairs)

Table 1 — capability axis (expected near-flat by design)

metric	baseline	treatment	test
pass-rate	100%	100%	McNemar p=1.0, h=0.0

Table 2 — discipline axis (the lift)

metric	baseline	treatment	Δ	test
mean discipline	0.333	1.000	+0.667	Wilcoxon p=0.0005, rb=1.0 (n≠0=16)

Table 3 — cost axis (mean tokens/run, non-errored)

metric	baseline	treatment	Δ
mean tokens	97,528	992,044	+894,516

Status buckets (trajectory)

arm	runs	error-rate	buckets
vanilla	24	0%	completed:24
package	24	0%	completed:24
placebo	24	0%	completed:24

Methodology

• Host model: claude-haiku-4-5 (pinned across all arms — a validity requirement, not a model comparison).
• Per-run budget cap: $3.5; placebo injected ~6628 chars of inert prose.
• Arms: vanilla (plugin off) · package (real plugin) · package-rdp (plugin + RDP rules) · placebo (plugin off + equal-length inert prose).
• Corpus: internal/bench/corpora/ab-trackb-v2.yaml (5 trap archetypes). Scoring: bench_ab_scoring_v2.py (deterministic, no LLM judge).
• Roadmap: agents/roadmaps/road-to-discipline-axis-benchmark.md.

Cost-factor sweep (claude-haiku-4-5) — lift per loaded-context cost

Question: the full package buys its weak-host lift at ~12× vanilla tokens. How much of the lift survives in trimmed rule-only configurations at a fraction of that cost? Four arms, same paired design (2 tasks × 12 seeds, n=24 pairs/arm), same host, same deterministic scorer. The trimmed arms run plugin-OFF + ONLY the named rule bodies injected via system prompt (rules_subset_text() in bench_ab_v2_run.ts, tier membership from dist/router.json).

arm	loaded content	injected chars	mean tokens/run	cost factor	mean discipline	lift vs vanilla
vanilla	none	0	103,319	1.0×	0.458	—
rules-balanced	kernel + tier 1 (shipped balanced profile)	98,825	303,186	2.9×	0.417	−0.042 (p=0.8127, NULL)
rules-kernel-dc	kernel (9 rules) + downstream-changes	30,698	344,483	3.3×	0.917	+0.458 (p=0.0135, significant)
package	full plugin	0	1,210,078	11.7×	1.000	+0.542 (p=0.0017, significant)

(generated from the pinned report — curated labels from docs/benchmark.pinned.yml)

Residual of the full package over rules-kernel-dc: Δ=+0.083, Wilcoxon p=0.37 (only 2 discordant pairs) — not significant.

full discipline-tier disposition (council 2026-07-10)

The full tier (~11.7×) stays experimental, opt-in only, never surfaced as a recommendation. Council (claude-sonnet-4-5 + gpt-4o, 2-round debate, 2026-07-10) converged on keep-and-relabel over drop: round 1 favoured dropping full, but the rebuttal round reversed it — p=0.37 is absence of evidence, not evidence of absence (an underpowered n=24 compounded by an essential ceiling effect), and removing an enum value is an irreversible breaking change for anyone pinning the tier string. An experimental opt-in does not violate “measured, not asserted”; an unlabeled recommendation would. Revisit-if (drop only when): a high-powered Claude sweep (n≥100, ceiling-adjusted) shows p>0.20 AND effect <5% and an open-source-host adapter sweep returns a null — i.e. full is shown actively useless, not merely unproven. Until then the experimental label stands everywhere full is documented.

Three findings:

1. ~95% of the lift survives at ~3× cost. The kernel + downstream-changes configuration keeps a significant discipline lift (0.917 vs the full package’s 1.000; the residual is not significant at this N) at ~28% of the full package’s tokens. The 12× full load is not required for this trap family’s lift.
2. Content selection beats size — the shipped balanced profile is a null. rules-balanced injects 3× more chars than rules-kernel-dc and costs almost the same per run, but delivers ZERO lift: it lacks downstream-changes (a tier-2 rule), and scope-control alone does not correct the downstream trap. This is the placebo result again, sharpened: not only is length inert, even 33 real rules are inert on a trap their lift-carrying rule doesn’t cover. Any low-cost weak-host profile must be cut by lift-carrying content, not by tier size.
3. Cost is behaviour, not just context. rules-kernel-dc injects a third of rules-balanced’s chars but costs slightly MORE — the discipline behaviour itself (verification turns, downstream edits) spends tokens. The token factor cannot be dialed by context size alone.

Honest scope: weak host only; the 2-task scope/downstream family (the family with the proven lift), N=24 pairs/arm — the full package covers 4 more trap archetypes the trimmed arms were NOT tested on here. Rules-only injection, not a full plugin projection (no skills/commands/hooks in the trimmed arms). Before shipping any trimmed default, sweep the full corpus (done below).

• Report: internal/bench/reports/ab-v2/2026-07-07T05-35-14Z-ab-v2-paired.json.
• Arms: rules-kernel-dc / rules-balanced in src/scripts/bench_ab_v2_run.ts (opt-in, not in the default arm list).

Full-corpus P1 gate (claude-haiku-4-5, all 30 tasks) — family-scoped PASS

The essential lift is real, replicates, and is family-scoped. Full corpus (all 5 trap archetypes + agentic-debug + the Laravel downstream trap), vanilla vs rules-kernel-dc × 3 seeds = 180 runs (n=90 pairs, 0 errored), run from a frozen checkout so mid-run edits could not contaminate the per-run rule reads. Corpus-wide the discipline delta is +0.056 (0.872 → 0.928, Wilcoxon p=0.084, rb=0.53) — NOT significant, because vanilla Haiku is already at/near the discipline ceiling on every family except the scope/downstream one. Inside that family the pilot lift replicates exactly: trapE (now 5 tasks incl. Laravel + meso variants) 0.533 → 1.000, Δ=+0.467, ALL 7 discordant pairs favouring the essential cut (sign test p≈0.016); all other families flat at ceiling (largest counter-noise: trapA −0.083 on 2 discordant pairs). Corpus-wide cost factor: 1.71x (132,036 → 225,956 mean tokens/run) — cheaper than the family-only 3.3x, because the discipline behaviour only spends turns where the trap exists.

axis	vanilla	rules-kernel-dc	Δ	test
capability (pass-rate)	92%	92%	0	McNemar p=1.0, h=0.0
discipline, full corpus (0–1)	0.872	0.928	+0.056	Wilcoxon p=0.084, rb=0.53 (n≠0=14)
discipline, scope/downstream family (n=15)	0.533	1.000	+0.467	7/7 discordant favour essential (sign p≈0.016)
mean tokens/run	132,036	225,956	+93,920 (~1.7x)	—

Verdict for the tiering roadmap’s P1 gate: family-scoped PASS. The essential tier’s claim stays honest-scoped to the scope/downstream family — the lift does not extend to families where the host is already at ceiling, and it costs ~1.7x on a realistic mixed corpus. The Phase-4 default flip remains additionally gated on the P2 non-Claude replication.

• Report: internal/bench/reports/ab-v2/2026-07-07T07-04-39Z-ab-v2-paired.json.

P2 gate — non-Claude weak host (gpt-5-mini via codex) — REPLICATION FAILED

The essential lift does NOT replicate on the first non-Claude weak host as shipped. Full corpus, vanilla vs rules-kernel-dc × 3 seeds on gpt-5-mini driven by the codex CLI (n=90 pairs, 0 errored, frozen checkout). Corpus-wide discipline Δ=+0.024 (p=0.70). Critically, this is NOT a ceiling null: on the scope/downstream family the host has headroom (vanilla 0.533→ 0.733) yet the rules do not fill it — trapE 0.733 → 0.667 (Δ=−0.067, 1 discordant pair). Capability trends negative (89% → 82%, McNemar p=0.07) — not significant, so no harm is claimed; it is a cautionary trend, not a validated effect. Cost factor 1.18x.

Honest scope / confound: the codex CLI has no system-prompt injection surface, so the rules were prepended to the user prompt in a marked block — a weaker instruction surface than the claude arms’ system prompt. The measurement cannot distinguish “discipline rules do not transplant to GPT-class hosts” from “user-surface injection is too weak”. It DOES establish the decision-relevant fact: as shipped, on this host, there is no measured lift. A system-surface experiment (API-loop harness) is a non-gating backlog follow-up.

axis	vanilla	rules-kernel-dc	Δ	test
capability (pass-rate)	89%	82%	−7pp	McNemar p=0.07 (n.s.)
discipline, full corpus (0–1)	0.819	0.843	+0.024	Wilcoxon p=0.70, rb=0.09 (n≠0=23)
discipline, scope/downstream family (n=15)	0.733	0.667	−0.067	1 discordant pair (negative)
mean tokens/run	302,655	358,326	+55,671 (~1.18x)	—

P2-verdict disposition (council claude-sonnet-4-5 + gpt-4o, 2026-07-07, 2-round debate — recorded in agents/settings/contexts/weak-host-lift-tiering-verdict.md): gpt-5-mini joins the measured NULL-lift disable-list; unknown_defaults becomes vendor-granular (anthropic: lift_enabled — the one family with a measured lift — default: lift_disabled); the balanced installer preset fills discipline_profile: auto, so the lift enables only where measured. The three-host evidence ledger: Claude weak = family-scoped lift · Claude strong = ceiling null · GPT weak = failed replication (confounded surface).

• Report: internal/bench/reports/ab-v2/2026-07-07T10-33-53Z-ab-v2-paired.json.

Strong host (sonnet, full 30-task corpus) — Gate verdict: HONEST-NULL

• capability lift significant: False
• discipline lift significant: False
• status-bucket better (package vs vanilla): False

Honest null on a strong host, across the full corpus. A re-run of the SAME package on sonnet over the entire discipline corpus — all 5 trap archetypes + agentic-debug + a Laravel/PHP downstream trap (trapE-scope-laravel-01) — with vanilla vs package × 3 seeds (180 runs, n=84 paired). The discipline axis does not move (Δ=+0.000, Wilcoxon p=1.0) and capability is flat-to-slightly-lower, because a capable host is already at the discipline ceiling on these deterministic traps — exactly what the weak-host section predicts. The package is a redundant no-op here, at ~5× the tokens. This is not a failure: it is the empirical bound on the weak-host claim, and it holds in PHP as in TS. No strong-host lift is claimed.

Table — package vs vanilla (n=84 pairs, host sonnet)

axis	vanilla	package	Δ	test
capability (pass-rate)	94%	89%	−5pp	McNemar p=0.125, h=-0.174
discipline (0–1)	0.929	0.929	+0.000	Wilcoxon p=1.0, rb=0.0 (n≠0=5)
mean tokens/run	185,584	929,716	+744,132 (~5×)	—

(host sonnet, n=84 pairs — generated from the pinned report)

• Report: internal/bench/reports/ab-v2/2026-07-05T07-00-31Z-ab-v2-paired.json (A6 of road-to-final-state-and-market-readiness.md).
• Methodology: identical to the weak-host section (pinned host, deterministic scorer, paired design); the only change is the host model and the full-corpus scope.

Recursive self-verification (ADR-106) — HONEST-NULL

Verdict: recursion is redundant with the always-on rules. verification.recursive stays off. No model got “closer to Fable” — exactly what ADR-106’s gate was built to disconfirm. The one retraining-free Sakana-Fugu mechanism (a depth-bounded attempt → critic verdict → re-attempt loop) was built, shipped behind a gate, and measured — and adds nothing over the rules.

Measured the package-recursive arm (D₂ = rules + recursion, deterministic scorer-as-critic, max_depth=1) against package (D₁ = rules only) on a weak host (claude-haiku-4-5), capH-debug archetype × 6 seeds (n=54 paired):

axis	D₁ (rules)	D₂ (rules + recursion)	Δ (D₂ − D₁)	test
capability (pass-rate)	87%	87%	0	McNemar p=1.0, h=0.0
discipline (0–1)	0.852	0.861	+0.009	Wilcoxon p=0.79, rb=0.33, n≠0=3

ADR-106 gate: FALSIFIED — neither a capability lift (p=1.0) nor a significant novel discipline lift (p=0.79; only 3 discordant pairs, below the ≥6 the gate requires).

Why, despite a passing human pre-test. Recursion fired on only 8/29 corpus tasks (~28%) and produced a differentiated output on 4/29 — with the rules active, the host’s first attempt already passes the critic 72% of the time, so recursion is a no-op. A blind human pre-test on the 4 differentiated pairs preferred the recursion output 4/4, but those cases are too rare and the aggregate marginal lift too small (n≠0=3) to register as significant. The pre-test looked positive on N=4; the paired benchmark falsified it — which is exactly why ADR-106 required the benchmark, not just the pre-test.

Honesty scope. Weak host, capH-debug family, deterministic scorer-as-critic, max_depth=1. A model-based critic (Phase 4) was not pursued — gated on this result passing, which it did not. Cost axis: each recursion run is up to 2× the host calls of a single pass, for a null lift.

• Roadmap: agents/roadmaps/archive/road-to-recursive-verification.md (closed honest-null).
• Gate logic: recursiveGateVerdict / resolveRecursiveDefault (orchestration_gate.ts); on a falsified gate resolveRecursiveDefault resolves off — no shipped-default flip.

Follow-up disposition — TERMINAL (AI council, anthropic/claude-sonnet-4-5 + openai/gpt-4o, 2026-06-24, deep tier). Both members converged: do not pursue a model-critic / cross-vendor variant. The 72% first-pass rate shows recursion solves the wrong problem — cost scales with all tasks, benefit only on the ~28% tail (best-case ~4–6% lift, would need n≥200 to detect); a model-critic would mostly fire more often and produce more null-lift re-attempts at higher cost. The real lever is refining the rules on the 28% failure tail (applies to 100% of tasks at zero marginal cost), not recursion. Recursion-as-a-class is closed; the model-critic’s contextual-quality angle, if ever wanted, is a different (quality-review) product, not a recursion follow-up.

Second-brain recall delta (claude-haiku-4-5) — PASS (bounded)

Verdict: a real, placebo-controlled cross-session recall lift — honestly scoped to the context-value upper bound, not retrieval precision. With the right prior fact surfaced, the model answers a multi-session recall task it otherwise cannot; the lift beats BOTH no-memory and equal-byte noise.

Three arms on a fixed host (claude-haiku-4-5), deterministic recall corpus (9 tasks) × 3 seeds = 81 calls, scored with no model-in-the-loop grading (second_brain_score). Paired sign test over the 9 tasks:

arm	pass	vs memory-on (paired sign test)
memory-on (substrate surfaces the prior fact)	27/27	—
memory-off (no memory)	10/27	on wins 6, ties 3, loses 0 — p = 0.031
placebo (equal-byte inert context)	9/27	on wins 6, ties 3, loses 0 — p = 0.031

memory-on beats BOTH off and placebo at p < 0.05 → PASS. The lift concentrates on the 6 retrieval-accuracy/contradiction tasks where the fact is available ONLY from memory (on 3/3, baseline 0/3); it ties on the 3 tasks whose k+1 prompt already self-contains the signal (an in-prompt correction or a named contradiction) — memory-on never loses.

Honesty scope. This is the context-value upper bound: the corpus is one-fact-per-task, so memory-on injects the exact fact (perfect retrieval). It proves the value of the right surfaced fact, isolated from mere extra context by the placebo — NOT the substrate’s retrieval precision under a large store, which is the follow-up corpus. Cost: 6.3k in / 8.7k out tokens for the full run.

• Report: internal/bench/reports/second-brain-delta.json; claim second-brain-recall-lift (docs/CLAIMS.md); scope + the declined Obsidian export in docs/second-brain-scope.md.
• Roadmap: agents/roadmaps/archive/road-to-second-brain-delta-proof.md (closed PASS).
• Harness: src/scripts/second_brain_run.ts (--dry-run free / --run spend).

Second-brain retrieval precision (claude-haiku-4-5) — PASS, ranking-limit named

Verdict: the lift survives REAL retrieval under confuser load — the substrate recalls the right decision and the model disambiguates it — but the keyword scorer recalls without ranking, which is the FTS5 signal at scale.

The follow-up to the recall-delta above removes its perfect-retrieval assumption. Against a populated decision store (5 needed + 19 distractors, with distractors that deliberately share query keywords with the needed decision), the run uses the REAL memory_lookup retrieval (not injection). Same host, 9 tasks × 3 seeds = 81 calls.

metric	result
precision@5 (needed decision in top-5)	9/9 (100%)
mean tie-set size (entries sharing the top score)	3.3 — recalls, does not rank; ties break by store order
retrieval-on (top-5 injected, confusers included)	27/27
retrieval-off (no memory)	5/27 — paired sign test vs on: 8 wins / 1 tie, p = 0.008
placebo (equal-count fixed unrelated entries)	5/27 — vs on: 8 wins / 1 tie, p = 0.008

Reading it honestly. Recall@5 is 100% at this scale: with ≤ k keyword-matching entries per query, the needed decision fits in the top-k and the model reliably picks it out of the co-injected confusers (retrieval-on 27/27). The scorer gives 0.8 to any keyword match and breaks ties by store order, so it recalls but does not rank (mean tie-set 3.3) — once more than k entries share a keyword, recall into the top-k degrades. That degradation is the discrimination gap the SQLite-FTS5 activation path (ADR-116) exists to close; this run is its motivating evidence, not a contradiction of it.

• Report: internal/bench/reports/second-brain-retrieval.json; claim second-brain-retrieval-precision (docs/CLAIMS.md); scope in docs/second-brain-scope.md.
• Store + corpus: internal/bench/second-brain/retrieval-store/.
• Harness: src/scripts/second_brain_retrieval.ts (--dry-run computes the free deterministic precision@k / tie-set; --run adds the model arms).

Thin-vs-eager quality judge (token-saving Phase 0) — INCONCLUSIVE (judge-limited)

Verdict: the live judge run was executed, and it does NOT trustworthily resolve whether the thin rule-projection holds output quality — the signal is non-significant, judge-inconsistent, and length-confounded. Recorded as an honest null; the quality-regression gate stays inert (by design — its report path is gitignored) pending a stronger, length-neutral judge.

The thin-vs-eager runner (src/scripts/bench_quality_run.ts) generates each labelled golden task’s answer under the THIN rule context (kernel bodies + non-kernel pointers, ~15k tok) and the EAGER context (all rule bodies, ~87k tok), then judges the pair in both orders (evaluatePair → reject-on-flip). Live run 2026-07-09, host + judge claude-haiku-4-5, 30 labelled tasks (evidence: internal/bench/reports/quality-run-2026-07-09-haiku-inconclusive.json):

metric	value	reading
decisive pairs	16 / 30	thin 5 · eager 11
thin win-rate	31%	below the 0.48 floor — but see the three confounds
Wilcoxon p	0.196	not significant — no reliable thin≠eager difference
judge inconsistency	33% (10/30 flip on order swap)	the haiku judge is unreliable here
length-confound	69% of decisive wins went to the LONGER answer	eager (87k ctx) answers longer → verbosity bias, exactly what the harness flags

Why it is not a regression verdict. The win-rate alone would trip the gate, but the three diagnostics disqualify it as evidence: the difference is not significant (p=0.20), a third of pairs are position-unstable, and two-thirds of decisive wins track answer length, not quality. A “thin regresses” claim off this run would be over-claiming a confounded, non-significant signal — the opposite of the honest-null discipline.

Re-open (fresh-spend follow-up). A trustworthy verdict needs (a) a stronger judge (sonnet-class) to cut the 33% inconsistency, and (b) a length-neutralised comparison (truncate/normalise answer length, or score against the anchors directly) to remove the 69% verbosity confound. Until then the thin lever is NOT adopted as default and the gate stays inert. The golden set also still covers only 14/89 rules (operator hand-labelling), a separate gap.

• Runner: src/scripts/bench_quality_run.ts (--dry-run free / live is API-gated); gate: src/scripts/check_quality_regression.ts (inert until a quality-run.json — gitignored — exists locally).
• Roadmap: agents/roadmaps/road-to-token-saving.md Phase 0.