# 03 — Real Hidden-State Extraction at Layer L
*Feed the cortex the residual where semantic intent actually forms, not the final-layer mean-pool it metabolizes today.*

Status: REFUTED (2026-07-01, S1.4) | Thesis anchor: experiments.txt §1 (correction-gated SSM cortex), §3 (neural hippocampus) | Goal anchor: GOALS.md Goal 2 (hidden-state extraction at layer L) | Depends on / relates to: 04-context-scoped-attractors (consumer), 05-metabolism-loop-closure (consumer), 02-kv-slot-fact-injection (sibling Goal 1, blocked), 01-correction-to-competence-benchmark (eval substrate)

> **Outcome (2026-07-01):** REFUTED. S1.4 HF layer-L probe (`../experiments_logs/2026-07-01_s1_4_hf_layer_probe.md`) measured warm_sep_silhouette on two architectures: Qwen2.5-0.5B-Instruct (gap −0.083, threshold +0.10) and LFM2.5-1.2B-Instruct (gap +0.058, threshold +0.10). Mid-layer hiddens do NOT cluster by concept better than the final layer on either model. This confirms lane_03's refutation on a substrate that can see every layer — the mid-layer assumption is a model property, not a llama.cpp keyhole.
>
> **Campaign note (2026-07-11):** Campaign 0d48130 attempted to re-run Exp03 on colab but was **infrastructure-blocked** — repeated HF snapshot transfers failed before execution; no metrics or ASI scores emitted. This is not a scientific null or refutation; the authoritative pre-campaign verdict remains S1.4 above. Evidence: `../experiments_logs/2026-07-11_campaign_0d48130.md`.

## Problem

The cortex's only window into the LM is a depth-final, position-averaged summary, and that is almost certainly the wrong signal.

- `CortexAgent.perceive()` feeds `cortex.observe()` the output of `driver.peek_embedding(utterance, last_token_only=False)` (`src/oczy/experiments/cortex_agent.py:354`), which is the **final-layer, MEAN-pooled** embedding produced by `llama_cpp` `create_embedding` (`src/oczy/lm/cvec_driver.py:514-572`, pooling type MEAN). The docstring on that call (cortex_agent.py:350-353) explicitly marks it "Goal 2 staging."
- `peek_layer` does **not** exist. `cvec_driver.py:519-520` states: "Layer-L intermediate extraction is not supported here yet (binding limitation tracked under Goal 2)." GOALS.md "Goal 2 — Hidden-state extraction at layer L" is unimplemented.
- The Goal-2 rationale in GOALS.md is the gap: "Real cortex metabolism needs the residual at a layer that actually carries semantic intent — empirically layers in the middle to upper third," not the embedding/final summary the cortex sees now.
- The cortex subsystem brief records this as an unanswered open question: *"Is the final-layer mean-pooled embedding rich enough for meaningful steering, or does Goal 2 (layer-L peek) materially change the proj_hidden → warm_state signal?"* No experiment has measured it.
- We cannot answer it with the current metrics. Every recent run reports `code_qa_accuracy=1.0` (runs #79, #80, #82, #84, #85, #95, #101), and wherever a recall path succeeds it pins straight to ceiling — `domain_co_recall` 1/1 (real driver, run #95) and prefix-driven exact `co_recall` 1/1 (run #101); mock-driver exact `co_recall` is 0/0 (run #85) only because the hash embeddings carry no semantics. None of these discriminate architecture variants. A new, non-saturating **representational** metric is required to tell whether layer-L input changes the cortex at all.

Grounded layer facts (these correct the stale `n_layers=28` in CLAUDE.md KEY FACTS — and the matching "28-layer transformer" phrasing in GOALS.md's own Goal-2 rationale): the actual checkpoint config at `~/.cache/huggingface/hub/models--LiquidAI--LFM2.5-1.2B-Instruct/snapshots/<rev>/config.json` is `model_type=lfm2`, `hidden_size=2048`, `num_hidden_layers=16`, `vocab_size=65536`, with `layer_types` placing `full_attention` blocks at config indices **[2, 5, 8, 10, 12, 14]** (the other 10 blocks are `conv`). The GGUF driver's `llama_n_layer` also reports **16** (brief), matching HF — so "layer L" ranges over 16 blocks, and the 6 attention blocks are the most plausible carriers of routed semantic intent.

## Hypothesis

- **H1 (depth/position matter).** A cortex fed the residual at a mid/upper attention block (HF `hidden_states` index ~9 or ~13) yields `warm_state` vectors that are **more semantically separable** — paraphrases cluster, distinct concepts separate — than the same cortex fed layer-0 embeddings or the current final-layer mean-pool, by a cosine-silhouette margin ≥ 0.10.
- **H2 (structure is semantic, not a Hebbian artifact).** `proj_hidden` trained with `KVCortex.train_step` (Hebbian, `plastic-cortex/src/plastic_cortex/kv_cortex.py:499-524`) on **real** layer-L hiddens aligns its row-subspace with the top-PCA subspace of those hiddens at ≥ 2× chance **and** ≥ 2× a column-shuffled-hidden control — i.e., the learned projector reflects the real hidden manifold, not self-amplified noise.

**Falsifier.** If layer-L silhouette ≤ final-mean-pool silhouette, or if `warm_state` trajectories for layer-L vs layer-0 stay cosine ≥ 0.98 over an identical utterance stream (the input source is irrelevant), then Goal 2 buys the cortex nothing measurable and the production swap is killed.

## Why now / what unblocks it

- `transformers==5.12.1` is installed and ships an `lfm2` module (`Lfm2ForCausalLM` is importable), and the **full-precision HF checkpoint is already cached locally** (`models--LiquidAI--LFM2.5-1.2B-Instruct`). So the brief's named "twin eval with activation capture" path is a one-call `from_pretrained(..., output_hidden_states=True)` forward — no binding fork, no network.
- The validation needs **no steering/articulation**, so it sidesteps the blocked KV-slot path (Goal 1 / sibling 02) and the saturated benchmark entirely: it only exercises `cortex.observe` input → `warm_state` and `proj_hidden`, both pure-numpy (`kv_cortex.py:173-225, 499-524`).
- A binding-side fallback exists if the twin is too heavy: `llama_context_default_params` exposes `cb_eval` / `cb_eval_user_data` (the ggml backend eval callback — verified present in `llama_cpp` 0.3.31), the brief's "binding hook" path, runnable on the already-loaded Q4 context.

## Approach (ties to §1, §3)

- Implement `peek_layer(prompt, layer_idx, pooling)` on an HF twin driver returning a `d_embd=2048` float32 vector: tokenize → forward with `output_hidden_states=True` → select `hidden_states[layer_idx]` → pool (last-token default; mean optional). This is the cortex's real input window (§1).
- Drive the existing `KVCortex.observe()`/`train_step()` **unchanged**; vary **only** the peek-source layer (matched-pair single-variable, the repo standard).
- Measure (a) representational separability of `warm_state` and (b) structural alignment of `proj_hidden` — §1 (does the SSM cortex metabolize real intent?) and §3 (do the resulting hidden traces have hippocampus-worthy structure?).
- Parity-check the twin against the GGUF final embedding so we know it is the same network (control).
- **Defer** the production wiring — swapping `cortex_agent.py:354` `peek_embedding` → `peek_layer(L)` — behind passing acceptance. That swap is what unblocks 04 (context-scoped attractors need rich, separable cortex state) and 05 (metabolism loop needs real intent in, not a position-mean).

## Success criteria (discriminating, non-saturating)

These deliberately avoid `code_qa_accuracy`/`co_recall`, which already sit at 1.0 wherever they apply.

1. **`warm_sep_silhouette` (headline).** Cosine-silhouette of `warm_state` vectors over a labeled paraphrase/distinct battery. Pass: `silhouette(L_mid) − silhouette(L0) ≥ 0.10` **and** `silhouette(L_mid) − silhouette(final-mean-pool) ≥ 0.10`. Cannot saturate: within-group paraphrase variance is non-zero, so silhouette < 1 by construction.
2. **`warm_traj_cos_L_vs_L0`.** Mean per-step cosine between layer-L and layer-0 `warm_state` over an identical 16-utterance stream. Pass: ≤ 0.80 (trajectories visibly diverge — operationalizes GOALS.md done-when #2, today only a qualitative claim).
3. **`projh_align_ratio`.** PCA-subspace alignment of Hebbian-trained `proj_hidden`: real-trained ÷ shuffle-trained. Pass: ≥ 2.0 **and** real alignment ≥ 2× chance (`k/d_embd`). Operationalizes GOALS.md done-when #3 ("non-trivial structure, not random") with a number.
4. **`twin_gguf_final_cos` (parity control).** Cosine between `peek_layer(HF, final)` and `peek_embedding(GGUF, final mean-pool)`. Pass ≥ 0.85, else the twin is not trusted as the same model and the other numbers are void.

**Kill criteria.** (a) `silhouette(L_mid) ≤ silhouette(final-mean-pool)` → depth/position no better than the current input; do not swap the production path. (b) `warm_traj_cos_L_vs_L0 ≥ 0.98` → input source irrelevant. (c) `projh_align_ratio < 1.2` → `proj_hidden` "structure" is a Hebbian artifact, satisfying done-when #3 only trivially.

## Risks & open questions

- **Memory.** Full-precision LFM2-1.2B in fp32 ≈ 4.7 GB; the host ran the Q4 GGUF at ~1.6 GB RSS. Mitigation: load the twin in bf16/fp16 (~2.4 GB) and run the validation with **only** the HF twin (no GGUF needed). If still tight, use the `cb_eval` Path-B on the Q4 context.
- **Quantization/tokenization mismatch.** HF fp twin vs GGUF Q4 may diverge; if `twin_gguf_final_cos < 0.85` (different chat template/special tokens or quant drift), prefer the `cb_eval` extraction so layer-L hiddens come from the exact model the cortex steers.
- **conv vs attention blocks.** Which of the 16 blocks is richest for the cortex is unknown — swept across L0, a mid attention block, an upper attention block, and final. The 6 attention indices [2,5,8,10,12,14] are the prior, not a certainty.
- **tanh saturation.** `observe` applies `tanh(proj_hidden @ h)`; if real layer-L hiddens have larger norm than the embedding, the projection may saturate and collapse separability. Monitored via `warm_norm`; if saturated, note and scale.
- **`d_cortex` sensitivity.** Default `d_cortex=128`; stressors/curriculum use `d_cortex=4`. Separability may depend on capacity — swept as a secondary axis.
- **Out of scope.** Whether better `warm_state` separability translates into better steering/recall is **not** tested here — that belongs to 04 and 05, which consume this driver.

## Prior evidence

- `cortex_agent.py:350-354` — perceive() uses `peek_embedding(last_token_only=False)` (final-layer mean-pool) and labels it "Goal 2 staging."
- `cvec_driver.py:514-572` — `peek_embedding` is final-layer only via `create_embedding` (MEAN pooling); `peek_layer` unimplemented (docstring 519-520).
- GOALS.md "Goal 2" — done-when: `peek_layer(prompt, layer_idx)` returns d_embd; cortex on real layer-L hiddens shows visibly different `warm_state` trajectories vs layer-0; pickled `proj_hidden` trained on real hiddens shows non-trivial structure. (The rationale text still says "28-layer transformer"; the checkpoint is 16 layers — see Problem.)
- `kv_cortex.py:173-225` — `observe`: `warm = (1−plasticity)·warm + plasticity·tanh(proj_hidden @ h)`; `kv_cortex.py:499-524` — `train_step` Hebbian on `proj_hidden`, lr default 0.001, per-row L2 renorm.
- HF config (`models--LiquidAI--LFM2.5-1.2B-Instruct/.../config.json`) — `lfm2`, hidden_size 2048, num_hidden_layers 16, full_attention at config indices [2,5,8,10,12,14], vocab 65536; corrects CLAUDE.md/GOALS.md `n_layers=28`.
- `llama_cpp` 0.3.31 — `llama_context_default_params` exposes `cb_eval`/`cb_eval_user_data` (binding-hook fallback); GGUF `llama_n_layer` = 16 (brief), matching HF.

### Tracked fix-up (out of scope of this proposal but discovered while grounding it)

`KVCortexConfig.n_layers` defaults to **28** at `plastic-cortex/src/plastic_cortex/kv_cortex.py:57`, and the docstring at `:50` claims "Defaults match LFM2.5-1.2B-Instruct: d_embd 2048, n_layers 28." That is wrong — LFM2.5-1.2B has `num_hidden_layers=16` (HF + GGUF, verified above). The mismatch is silent because `proj_c` is `(n_layers, d_embd, d_cortex)`; the extra 12 rows go unprojected under the current per-layer cvec emission path, but anyone wiring a real L into `peek_layer` would index past the model's depth. **Action:** before implementing project 03, patch `kv_cortex.py:57` (and the `:50` docstring) to `n_layers=16`, or pass `KVCortexConfig(n_layers=16)` explicitly at every construction site (`run_curriculum.py:38-43`, real-driver stressors) until the default is fixed.
- Saturated-metric context: runs #79/#80/#82/#84/#85/#95/#101 all report `code_qa_accuracy=1.0`, with `domain_co_recall` 1/1 (run #95) and prefix exact `co_recall` 1/1 (run #101) at ceiling — motivation for a new representational metric.