01 — Correction-to-Competence Benchmark v2: de-saturating the eval

The eval can no longer tell two architectures apart; rebuild it so a behavior difference is the only thing it can register.

Status: TESTED-NULL (2026-07-11) | Thesis anchor: experiments.txt §14 (Correction-to-Competence Benchmark), §10 (mistake immune system), §9/§13 (compression / how to train), §1 (correction-gated cortex) | Goal anchor: GOALS.md Goal 1 (LM-side steering) + Goal 3 (organ tensor upgrades / metabolism loop) | Depends on / relates to: 02-kv-slot-fact-injection, 03-layer-l-hidden-extraction, 04-context-scoped-attractors, 05-metabolism-loop-closure, 06-bounded-growth-consolidation, 07-conversation-world-model-rl

Outcome (2026-07-11): TESTED-NULL. Campaign 0d48130 (colab, commit 537260c) ran the v2 scorecard: v2_behavior_delta_mock=0.0, v2_discrimination=0.0. The benchmark produced 5 de-saturation events (separated exact-vs-domain recall, signed interference forgetting both spread=1.0) but detected zero behavior delta — no architecture pair separated on the new scorecard. domain_recall=1.0 but exact_recall=0.0. The de-saturation itself is a result (the eval can now distinguish exact from domain), but the S/H behavioral null (H2) is the measured verdict. Evidence: ../experiments_logs/2026-07-11_campaign_0d48130.md.

Problem

The headline eval is saturated and cannot discriminate the architecture variants it exists to compare.

• Saturated headline. autoresearch.sh runs oczy.experiments.codebase_qa.benchmark; every recent run prints code_qa_accuracy=1.0 and cortex_agent_recall_accuracy=1.0 (src/oczy/experiments/logs/SUMMARY.md, runs #79–#105).
• Capped ratio metrics reward non-regression, not learning. In src/oczy/experiments/eval_suite.py three of the seven scorecard metrics are ratios clamped at 1.0: forgetting_score = min(1.0, post/pre) (line 374), identity_drift_score = min(1.0, post/pre) (line 375), consolidation_score = min(1.0, cons_overall/post_overall) (line 390). In SUMMARY.md both FastOnlyAgent and OrganismAgent score forget=consol=identity=1.0000; they differ only on transfer (0.1667 vs 0.25) and scope ties (0.1667 vs 0.1667). Any agent that does not regress on the old battery banks a free 1.0 on three of seven metrics.
• The memory metric is inverted and pickle-based. memory_bytes_per_delta = consolidated_size / max(1, successful_lessons) (eval_suite.py:396), surfaced as card key memory_bytes_per_behavior_delta (line 422). Lower is "better" — the literal inverse of the north star behavior_delta_per_byte_of_persistent_memory (rl_pipeline_design.md:342). Consequence in SUMMARY.md: the agent with the best transfer (OrganismAgent, 0.25) gets the worst Mem/Δ (68636.5 vs FastOnlyAgent 12.0). The composite punishes the best learner.
• The match function counts a domain shift as exact recall. _matches with sense_match returns True on ≥2 shared non-stopword tokens (eval_suite.py:179). A cvec that only shifts DOMAIN therefore scores as if it recalled the exact fact — even though briefs show cvec exact-token recall is provably 0 (2026-06-25_prefix_steering_poc.md; runs #136–#139).
• The "measures internal mechanics not behavior" trap. experiments_logs/2026-06-19_extended_learning_evaluation.md scored NeuralHippocampus aggregate 1.000, ranking it ABOVE the Oracle upper bound (0.844). That score is surprise-gating precision + replay accuracy + compression ratio — internal organ mechanics, not respond()-level behavior.
• Non-discrimination, concretely. S vs H consolidation are mechanically distinct — consolidation_strength 10.0 vs 36.0 and cold_drift 0.10 vs 0.64 — yet tie on EVERY behavioral metric: identical co_recall and domain_co_recall, and near-identical memory_bytes (29,211 vs 29,214 uncapped; 10,443 vs 10,444 capped) (runs #84, #85, #94, #95; experiments_logs/2026-06-27_memory_per_byte_sh.md and 2026-06-27_domain_recall_metric.md run #95). The eval cannot separate the two architectures it is meant to compare.

Hypothesis

• H1 (de-saturation). A behavior-only scorecard built from (a) signed interference-based forgetting, (b) separated exact-vs-domain recall, and (c) an un-inverted behavior_delta_per_byte over changed persistent bytes will produce non-overlapping 95% CIs between steering variants known to differ behaviorally (prefix / logit-bias exact_recall≈1 vs cvec-only exact_recall≈0) — pairs the current scorecard scores identically.
• H2 (null-capable S/H verdict). For S vs H consolidation the same scorecard will EITHER separate them (disjoint CIs on ≥1 behavioral metric) OR establish a measured null (CIs overlap, n≥5) — replacing today's saturated 1.0 tie with a quantified, falsifiable statement.

Why now / what unblocks it

• Every ingredient already exists; the blocker was metric design, not tooling. eval_suite.py owns the snapshot/score scaffold, baselines.py the matched comparators (ZeroMemoryAgent, ContextOnlyAgent, FastOnlyAgent, HippocampusOnlyAgent, IdentityOnlyAgent) plus OrganismAgent, multi_fact_stressor.py the S/H consolidation harness with mock + real LFM2.5 drivers and memory_bytes, and codebase_qa/benchmark.py the two-axis _score (semantic vs domain) and the known-ground-truth steering separation from runs #136–#139.
• The needed compression inputs are already captured but discarded: EvalResult stores raw_trace_size and consolidated_size (eval_suite.py:433–435) but final_card never surfaces a compression ratio.
• Recent logs explicitly ask for "a metric that moves" (memory_per_byte_sh open Q1; SUMMARY remaining-blocks: "further progress on behavior_delta_per_byte requires moving beyond exact-token recall probes").
• This project is upstream of all siblings (02–07): none of them can claim a win until the eval can register one.

Approach (tied to thesis §14)

• Behavior-only rule. Every metric is a pure function of agent.respond()/articulate() on held-out probes; no metric may read internal organ state (consolidation_strength, cold_drift, surprise-gating precision). This structurally kills the NeuralHippocampus-1.000 trap (§14, §10).
• Interference forgetting (§1, §3). Teach A, probe A; teach interfering B; re-probe A. forgetting_delta = acc_A_before − acc_A_after ∈ [−1,1], signed and uncapped. A stable no-op agent scores 0 here AND 0 on uptake — it cannot farm a free 1.0.
• Two-axis recall (§14; relates 02, 03). Keep exact-token and domain as separate scores (reuse benchmark.py _score) so domain-only steering can never count as exact recall.
• Un-invert the north star (§9, §13; relates 06). behavior_delta_per_byte = net_behavior_delta / max(1, Δpersistent_bytes), where Δpersistent_bytes is the serialized cold/persistent state AFTER − BEFORE learning (changed bytes, not absolute pickle). Surface compression_ratio = raw_trace_size / consolidated_size as its own sub-metric.
• Discrimination is the headline acceptance test. Across N seeds the benchmark must separate ≥1 known-different pair with disjoint 95% CIs. If it cannot even separate the steering pair (whose ground truth is known from runs #136–#139), the benchmark is rejected.

Success criteria (discriminating, with kill criteria)

Primary (discrimination power):

• Across ≥5 seeds, the v2 scorecard yields non-overlapping 95% CIs between prefix/logit-bias (exact_recall≈1) and cvec-only (exact_recall≈0) on the exact_recall axis. KILL: if those CIs overlap, the new exact axis is no better than the old token-overlap match — reject and revert.
• ≥3 of the 7 sub-metrics show a spread >0.2 across the baselines.py agent set + OrganismAgent. KILL: if ≥5 of 7 still saturate (all agents within 0.05), de-saturation failed.

Secondary (the S/H verdict):

• Report behavior_delta_per_byte for S and H with 95% CIs. SUCCESS = disjoint CIs (difference detected) OR a pre-registered null (CIs overlap, n≥5). Today's 1.0 tie is neither.

Hygiene:

• No scorecard metric reads internal organ state — verified by a unit test that mutates consolidation_strength/cold_drift and asserts the v2 card is byte-identical.
• behavior_delta_per_byte is monotone increasing in transfer at fixed bytes (a regression test the legacy inverted metric fails).

Risks & open questions

• S vs H may be a genuine behavioral null because gradients do not flow through the toy organs (rl_pipeline_design.md; SUMMARY). Then the honest deliverable is a measured null + handoff to 05-metabolism-loop-closure. That is still a de-saturation win (a moving, falsifiable number) but not an architecture win.
• Δpersistent_bytes may be ≈0 because cold_state is a fixed-shape array — its norm grows 0.34→3.6 over 20 cycles but its byte count is constant (falsification in 2026-06-25_svd_init_proj_c_persistence.md). Fallback: use ‖Δcold_state‖ (information delta) as the denominator; pre-registered.
• Real-driver CIs may be degenerate because greedy LFM2.5 decoding is deterministic per prompt; seed variance comes only from curriculum sampling + SVD-init. May need more probe items, not more seeds. The mock _MockDriver (hash embeddings) carries 0 semantics, so exact recall is structurally 0 there (runs #85) — report mock and real separately.
• Exact-token probes are at their sensitivity limit (commit 4ef82e4 "probe is at its sensitivity limit"); a single-target probe may not separate. Fall back to multi-fact co_recall (multi_fact_stressor.py) as the exact axis.
• The coherence heuristic also collapsed (commit 1a6f298: "coherence sensitivity=0 for both parallel and sequential modes, all coh=1"). With order-shuffle CIs degenerate and the new tiling-detector (2–6 char) fixing only the false-positive side, the heuristic joins the ranks of saturated metrics — direct evidence the eval went blind, not just one metric.

Prior evidence

• src/oczy/experiments/eval_suite.py: capped ratios (lines 374, 375, 390); inverted pickle memory metric (lines 396, 422); token-overlap sense match (line 179); raw_trace_size captured but absent from final_card (lines 433–435 vs 416–426).
• src/oczy/experiments/logs/SUMMARY.md scorecard: FastOnlyAgent/OrganismAgent tie at forget=consol=identity=1.0; Mem/Δ 12.0 vs 68636.5 inverts against transfer 0.1667 vs 0.25.
• experiments_logs/2026-06-19_extended_learning_evaluation.md: NeuralHippocampus 1.000 > Oracle 0.844 (internal-mechanics score).
• S vs H ties: runs #84, #85, #94, #95; experiments_logs/2026-06-27_memory_per_byte_sh.md (strength 10 vs 36, identical co_recall, near-identical memory_bytes 10,443 vs 10,444); 2026-06-27_domain_recall_metric.md run #95 (identical domain_co_recall 1/1, memory_bytes 29,211 vs 29,214).
• Steering ground truth to separate: prefix exact=1/domain=1 vs cvec exact=0/domain=1 (2026-06-25_prefix_steering_poc.md); logit-bias exact=1 (runs #136–#137); composition run #139.
• North star defined but never scored: behavior_delta_per_byte_of_persistent_memory (rl_pipeline_design.md:342); autoresearch.sh → codebase_qa/benchmark.py is the entry point.