research spec
11 — Minimal metabolism loop on the HF substrate (Sprint 2 / S2.1)
- File
11-s2-minimal-metabolism-loop.md- Size
- 5.8 KB
- SHA-256
1c76a00ac127eed5…
11 — Minimal metabolism loop on the HF substrate (Sprint 2 / S2.1)
Pre-registered 2026-07-02 (human-approved sprint setup, before implementation). Agents running this experiment MUST NOT edit this spec; deviations are reported as deviations.
Problem
The audit's central architectural finding is inversion: in the full organism,
retrieval (scope-slot reranker, DSI, hippocampal lookup at answer time) does
the work the thesis attributes to changed dynamics. Every past "loop closed"
claim ran with five organs attached, so nothing could attribute the behavior
change to the fast-weight path. Sprint 0 froze the eval and re-measured honest
baselines; Sprint 1 delivered the HF substrate (Qwen2.5-0.5B-Instruct,
HFDriver). This experiment asks the thesis's minimal question with nothing
else in the room.
Hypothesis
H-LOOP: a minimal organism consisting of ONLY
HFDriver(Qwen/Qwen2.5-0.5B-Instruct, greedy, CPU float32),- a fast-weight cortex (warm/cold state, Hebbian
observe,consolidatemerging warm→cold), and NeuralHippocampusused strictly as a consolidation-time replay buffer (NEVER queried at answer time)
closes the loop: correction → fast-weight change → consolidation → measurably changed LM behavior on held-out frozen-eval probes, compounding with the number of corrections K.
Explicitly banned components: WorldModelCritic, IdentityHypernetwork,
SkillImmuneCortex, ExperienceAutoencoder, DifferentiableFactIndex, the
scope-slot reranker, logit bias, and ANY answer-time retrieval (no hippocampus
reinforce() during answer(), no per-probe lookup of stored corrections).
Content channel (S2.1 form)
At consolidate(), replayed corrections are compiled into a bounded
consolidated articulation state: a single articulation prefix of at most
48 tokens total (all facts share the budget; overflow = oldest-dropped and
reported), set once via HFDriver.set_articulation_prefix, plus optional cvec
posture. The prefix is built from consolidated summaries at consolidation
time — NOT re-derived per probe from raw traces. (S2.2 / research/12 replaces
this prefix with written KV entries; S2.5 / research/13 deletes the raw traces
and tests survival.)
Data & protocol
- Stage:
eval/v2/stage_0_grounding.json(frozen;verify_manifest()must pass before and after the run). - Split:
split_probes(stage, fraction=0.3, salt="v2")— development on dev, all primary numbers on holdout only. - Teaching: episodes taught cumulatively in a seed-shuffled order; each
teaching event feeds
correction_utterancethrough perceive/metabolize and stores the episode in the hippocampus; consolidation fires per the digestive gate (or forced at each checkpoint boundary — implementer's choice, fixed in code and reported). - K checkpoints: K ∈ {0, 1, 2, 4, N} where N = all stage-0 episodes.
After each checkpoint: consolidate, then score ALL holdout probes with the
frozen scorer (
oczy.eval_v2.scoring.probe_matches). - Seeds: ≥5 (vary teaching order + cortex init; LM is deterministic
greedy). Mean ± 95% CI via
oczy.common.stats. - Vanilla column: bare
HFDriveron the same holdout probes, mandatory.
Primary metrics & acceptance
loop_delta_holdout= mean over seeds of [holdout accuracy at K=N] − [vanilla holdout accuracy].loop_compounding_rho= Spearman ρ between K and mean-over-seeds holdout accuracy across the 5 checkpoints.
- Accept H-LOOP:
loop_delta_holdout > 0with 95% CI excluding 0, ANDloop_compounding_rho >= 0.6. - Refute: either fails. A refutation is a recorded result, not a failure.
Validity gate (not acceptance): vanilla holdout accuracy must be < 0.5 on stage 0 (otherwise there is nothing to learn and the run is INVALID, not a refutation).
Pre-registered secondary analyses (exploratory only — cannot flip acceptance)
- The S2.3 drift triple (Δ_target / Δ_control / Δ_target_clamped) at every
checkpoint, using the FIXED clamp-budget capture (the cross-instance
stochasticity artifact from
2026-07-01_s2_4_breakthrough_ablation.mdmust be repaired before this run). - Dev-split trajectory (for overfitting comparison dev vs holdout).
memory_bytesat each checkpoint and prefix-token count actually used.- Wall-clock per checkpoint.
Pre-registered fallbacks (fixed now, so they cannot be chosen post hoc)
- If wall-clock per seed exceeds 15 min: drop to 3 seeds, keep all checkpoints, and report the reduction as a deviation.
- If the digestive gate never fires: force consolidation at checkpoint boundaries and report it.
Reporting
Full per-seed, per-checkpoint table (holdout accuracy, drift triple,
memory_bytes); vanilla column; model id; exact commands; log to
experiments_logs/2026-07-02_s2_1_minimal_loop.md quoting this spec.
Amendment 2026-07-02 (before any primary verdict was drawn)
The pre-registered split call split_probes(stage, fraction=0.3, salt="v2")
was discovered to be degenerate on stage 0: an unlucky hash assigned all 8
probes to dev, 0 to holdout — a state validate_split itself defines as an
ERROR. The first S2.2 run executed against this empty holdout and is recorded
as INVALID (instrument failure); its 0/0 "REFUTE" is void and carries no
evidential weight for or against H-KVCONTENT.
Repair (instrument-level, not spec-level): split_probes now guarantees a
non-empty holdout for every stage by promoting the lowest-force-hash probes to
ceil(fraction × total) when thresholding yields none (stage 0: 3 holdout
probes). No previously non-empty split is altered (locked by regression test
test_split_guarantee_never_alters_nonempty_holdouts), so all previously
logged numbers remain comparable. The spec's split call, salt, and fraction
are unchanged. Amendment applies identically to research/11, 12, and 13.