16 — Standing external benchmark battery (Sprint 4 / S4.2)

Pre-registered 2026-07-02 (human-approved sprint setup). Agents MUST NOT edit this spec. Depends on research/11 (the organism under test is the minimal loop, plus whatever research/14–15 keep).

Problem

The 2026-07-01 external QA run showed the organism worse than vanilla cross-domain (0.388 vs 0.512) — the overfitting signature: home-curriculum gains that cost general capability. One-off external checks rot; the battery must be standing, uneditable by the optimizing loop, and include at least one benchmark this repo did not author.

Battery composition (fixed here)

1. External QA — the existing cross-domain QA set (repo-authored, out-of-curriculum). Frozen copy + SHA-256 manifest, same regime as eval/v2.
2. Pi tool-use benchmark — benchmarks/pi harness (repo-authored). Runs the version on main at battery time; its own manifest.
3. Non-authored benchmark — a fixed 200-item slice of an established public benchmark, feasible for a 0.5B CPU model, data vendored locally (offline, no network at run time). Selection criteria fixed now: multiple choice or short-answer, published before 2026, unmodified items, slice chosen by seeded hash of item ids (seed=42), scored by the benchmark's own published metric. Concrete choice (e.g. ARC-Easy slice) is recorded at implementation time in the battery manifest — items may not be selected by looking at model performance.

Protocol

• Runs weekly (standing job: scripts/weekly_battery.sh, cron-able; also runnable on demand) and appends — never edits — one row per run to the S4.4 dashboard in experiments_logs/.
• Conditions per benchmark: vanilla HFDriver and organism (minimal loop + kept components, trained on its home curriculum first), same decoding config, ≥3 seeds where the organism is stochastic.
• The battery script verifies all manifests before running and refuses to run on a dirty working tree for protected paths.

Primary metric & standing acceptance bar

external_delta(b) = organism − vanilla accuracy on benchmark b.

• Green: external_delta(b) ≥ −0.02 on ALL benchmarks (the organism may not buy home-curriculum gains with general capability).
• Red (overfitting signature): any benchmark with external_delta < −0.05 with 95% CI excluding −0.02. A red battery BLOCKS promotion of any concurrent home-eval claim to a headline until resolved.
• Between: YELLOW, reported, not blocking.

This is a standing gate, not a one-shot hypothesis: every weekly row gets a Green/Yellow/Red stamp by the fixed rule above.

Reporting

One dashboard row per run: date, git SHA, per-benchmark vanilla/organism accuracy ± CI, delta, stamp. First run logged to experiments_logs/<date>_s4_2_external_battery_baseline.md quoting this spec.