Honest post-leakage-removal baseline (Sprint 0 / S0.2 DoD)

Date: 2026-07-01 Context: Sprint 0 (SPRINT.md) removed two classes of test-set leakage from the answer path (branch fix/s0-2-leakage, 11 sites across lanes/lane_04.py, scope_selectivity_stressor.py, run_curriculum.py):

1. _SCOPE_TEACHING dictionaries keyed to specific episode IDs (including entries authored for exactly the episodes that were failing), consumed by a pre-stage teaching pass.
2. prefix_targets=[probe.expected] — the probe's expected answer fed into the generation-time logit-bias path for scope probes.

This log records the curriculum scores after removal, per S0.2's definition of done. These are the numbers all future work must beat.

Configurations

• Frozen eval data: src/oczy/experiments/organism_curriculum/stages/ (44 episodes), semantic scoring, --split all (matches the historical runs the old claims came from; the dev/holdout split lands with S0.6 for future work).
• Vanilla column: VanillaAgent (no-op learn) over the same probes — new in S0.7, so historical runs have no equivalent.
• Reports: organism_curriculum/reports/honest_baseline_{mock,mock_seed0,real}_2026-07-01.json.

Results — real driver (LFM2.5-1.2B GGUF, --use-real-driver, 1 seed)

Results — mock CortexAgent driver (--use-cortex-agent-mock, 5 seeds)

Vanilla: 0.00 on every stage (verified answering, e.g. 'user profile' against corrected-sense probes — a genuine measured zero, not a harness gap).

Comparison against the pre-removal claims

The 2026-06-30/07-01 logs claimed (real driver, semantic): Stage 2 scope 1.00, Stage 5 scope 1.00 (after two-slot _SCOPE_TEACHING), Stage 5 retention 1.00. Post-removal, the same configuration measures Stage 2 0.69 and Stage 5 0.92 (aggregate per-stage accuracy; the old logs reported per-category scope/retention numbers, so the comparison is directional rather than exact — the aggregate mixes both categories).

Reading:

• Stage 2 lost ~0.3. A substantial share of the claimed scope-control ability was the harness teaching to the test and biasing probes toward their expected answers. 0.69 is the honest capability.
• Stage 5 held up well (0.92). The scope-slot reranker + residual wiring earn most of their cross-domain claim legitimately; the leakage was responsible for roughly the last tenth.
• Stages 0/1/4 are essentially unchanged (0.88/0.75/0.90) — consistent with the leakage being scope-specific.
• Vanilla = 0.00 everywhere confirms all measured accuracy is attributable to learning, not LM priors leaking through the probe format.

Methodology notes

1. Mock-path seed variance is exactly zero. Five seeds produce identical scores: the deterministic mock/shim answer ranking fully dominates; the seeded policy-head randomness never reaches the final answer. Multi-seed statistics only become informative on paths with real stochasticity (real-driver multi-seed runs are the follow-up — single real-driver seed here is a known limitation, flagged by the harness's own warning).
2. Mock and real driver are not comparable to each other (mock has zero semantics by design); each is a baseline for its own track.
3. Historical stage tables before 2026-06-30 are additionally invalidated by the scope-slot reranker bugs (see 2026-06-30_scope_slot_reranker_fix.md); this log supersedes them as the reference point.