07 — Conversation World Model (RL Phase 0)

An agent should predict the correction before it arrives — then let that prediction error, not a keyword, decide what to learn.

Status: ACCEPTED-PARTIAL (2026-07-11) | Thesis anchor: experiments.txt §6 (world-model-first: predict OUTCOMES, correction = prediction error); supports §4 (learned plasticity) and §10 (mistake immune system) | Goal anchor: GOALS.md Goal 3 (upgrade organs from string features to tensor consumers so the metabolism loop closes); relates to Goal 2 (layer-L peek feeds richer hidden) | Depends on / relates to: 01-correction-to-competence-benchmark, 05-metabolism-loop-closure (cross-link), 03-layer-l-hidden-extraction (richer input signal)

Outcome (2026-07-11): ACCEPTED-PARTIAL. Campaign 0d48130 (colab, commit 537260c): POSITIVE for marker-free uptake — marker_free_uptake_gap=1.0 (≥ 0.40 threshold met), accept_pred_auc_string=1.0, accept_pred_auc_hidden=0.8125 (≥ 0.70 threshold met). NULL for critic AUC improvement — critic_auc_delta=0.0 (the hidden-feature critic does not beat the string-feature critic). H1 (predictive) accepted; H2 (behavioral) partially accepted: the world-model modulator recovers marker-free uptake but does not improve correction prediction over string features. Single-run, no cross-seed variance. Evidence: ../experiments_logs/2026-07-11_campaign_0d48130.md.

Problem

The cortex decides how much to learn from a turn via a single scalar correction_signal that gates plasticity in KVCortex.observe() (plasticity = alpha_warm*(1-c) + alpha_correction*c). Today that scalar is produced by a lexical stop-gap: perceive() sets correction_signal = 1.0 if _looks_like_correction(utterance) else 0.0 (src/oczy/experiments/cortex_agent.py:347-348), where _looks_like_correction (cortex_agent.py:68-71) is a substring match against a hand-coded _CORRECTION_MARKERS tuple ("no, ", "wrong, ", "correction:", "expected:", "i meant", "actually,", "rather than", …; cortex_agent.py:53-65).

This is a keyword detector wearing the mask of a world model. It fails in two documented ways:

• It is blind to marker-free corrections. A correction phrased without a marker ("project beta uses rook", a paraphrase, an implicit re-grounding) reads as correction_signal=0, so the cortex applies only the alpha_warm (0.02) baseline plasticity and the lesson is not absorbed. The salience-ablation run #82 already hit this edge case from the other side — "correction-marker salience embedded all chunks in the synthetic needle turn … because the synthetic needle contains no marker" (2026-06-26 salience log) — confirming marker presence and semantic correction are not the same event.
• The actual world model is built but never used as the modulator. CortexAgent constructs a WorldModelCritic with use_hidden=True, use_value_head=True, mlp_hidden_units=16, value_learning_rate=0.05 (cortex_agent.py:219-226). It has a real acceptance head (predict_acceptance → accepted_prob, correction_likelihood, key_uncertainty, reading a 16-unit MLP over [4 string features ; lm_hidden], critic.py:171-199, 425-440) and a TD(0) value head (reward=-1/+1, gamma=0.95, critic.py:319-327). But its prediction is consumed only as a digest scalar; it never replaces correction_signal, and per the RL/policy SUMMARY the value head "is trained with TD on every metabolize() … but [has] not been validated in a real correction/uptake loop". rl_pipeline_design.md lists Phase 0 (predictive foundation) and Phase 1 (critic acceptance AUC > 0.8) as design-only, and the RL brief's own open question states "No held-out perplexity / Phase 0 predictive-foundation experiment has been run."

So the predictive substrate thesis §6 demands ("Then correction becomes prediction error") exists in code but is wired as a passenger, while a keyword heuristic drives the steering wheel.

Hypothesis

1. (Predictive) A self-supervised acceptance predictor reading the LM hidden (WorldModelCritic with use_hidden=True) can predict, at answer time and before any correction text exists, whether a turn will be corrected, with held-out (leave-one-episode-out) AUC strictly above both chance (0.5) and a string-features-only critic (use_hidden=False). The lexical _looks_like_correction detector has AUC ≈ 0.5 on this pre-correction task by construction (the marker is not yet present).
2. (Behavioral) Replacing the lexical correction_signal with the world-model's realized prediction error as the cortex modulator recovers correction uptake on marker-free corrections that the lexical detector misses, with a marker-free uptake gap Δ ≥ 0.4 (world-model − lexical), without raising false plasticity on non-correction (retention/scope) turns.

Either claim is independently falsifiable.

Why now / what unblocks it

• The predictor already exists and is already fed the right tensor: metabolize() calls predict_acceptance(..., lm_hidden=hidden_for_critic) and record_outcome(..., lm_hidden=...) every turn (cortex_agent.py:463-478), so a trace corpus of (hidden, proposed_answer, actual_outcome) tuples is already flowing — it is simply discarded as a digest scalar instead of being trained and read back.
• perceive() already accepts an explicit correction_signal argument (cortex_agent.py:331), so swapping the modulator source is a clean single-variable change: pass the world-model prediction error instead of letting the lexical default fire. No cortex surgery.
• The labels are free: the organism curriculum (organism_curriculum/, 44 episodes across 6 stages: 8/8/8/4/10/6) plus its probe batteries already produce accept/correct outcomes via the curriculum's organism_curriculum/scoring.py (matches/probe_matches/categorize_results, imported in run_curriculum.py:26; probe categories transfer|scope|forgetting|retention). Initial episode answers are always corrected (label=1); post-learning probes are accepted-or-corrected (mixed labels). This is the "interaction traces" corpus Phase 0 asks for, minus the unimplemented recurrent-cortex perplexity objective.
• The discriminating stressor is built: multi_fact_stressor.py already has a --paraphrase mode that "omits the original keywords" and a marker-bearing Correction: fact, so marker-stripping is a config flip, not new infrastructure.

Approach

Tied to thesis §6 (model the interaction, not the fact) and §4 (the plasticity signal should be learned, not hand-coded):

• Build the trace corpus. Run the curriculum through CortexAgent and log per-turn (request, proposed_answer, lm_hidden, actual_outcome ∈ {accepted, corrected}). Outcome label comes from existing scoring.py (probe_matches) scoring, not from markers.
• Self-supervised pretrain (Phase 0 core). Train WorldModelCritic.predict_acceptance teacher-forced on the corpus to predict the binary outcome from the answer-time hidden, with leave-one-episode-out evaluation. This is the §6 "predict acceptance / correction probability before answering" objective, implemented with the MLP that already exists rather than the design-only recurrent SSM.
• Wire prediction error as the modulator. At metabolize time the realized error |predicted_accept − actual_outcome| (or correction_likelihood) becomes the correction_signal passed into cortex.observe(), replacing _looks_like_correction. This closes thesis §6's "correction = prediction error" loop and is the modulator-source single variable.
• (Secondary, data-permitting) Add a small correction-type head (sense-misgrounding vs scope-overgeneralization) keyed off probe category; report it descriptively, not as a gate, because the 44-episode corpus is thin for multiclass.

Success criteria

Discriminating, measurable, with kill thresholds. The saturated code_qa_accuracy=1.0 is explicitly not a success metric here; it is reported only to confirm we did not regress it.

Headline = marker_free_uptake_gap: it cannot saturate at 1.0 because it is a difference between two live arms, and it directly measures the capability the lexical stop-gap lacks.

Risks & open questions

• Data scarcity. 44 episodes + probes is a small corpus for an AUC with confidence; leave-one-episode-out plus reported CIs are mandatory, and the correction-type head may be undertrainable (kept descriptive).
• Final-layer-only hidden (Goal 2 not done). The critic reads peek_embedding (final-layer mean-pooled), not a mid-network layer-L residual. If acceptance signal lives mid-network, AUC may be capped — this is the explicit cross-link to project 03.
• Class imbalance. Initial requests are all corrected; mixed labels come only from probes. Pre-correction AUC must be computed on the probe-balanced subset, not the all-corrected initials, or the number is meaningless.
• Modulator amplitude vs plasticity clip. correction_signal feeds alpha_correction=5.0 then clips to [0,1]; a continuous predicted error must be scaled so it spans the useful plasticity band rather than saturating at full overwrite. Open: is a soft (continuous) modulator better than the current binary gate even before semantics?
• Does prediction error generalize past disambiguation? Stage 3+ (dialog/cross-domain) were never run through the policy loop; AUC may collapse off the disambiguation manifold.

Prior evidence

• WorldModelCritic already carries acceptance + TD value heads, fed lm_hidden every turn but unread as a modulator: cortex_agent.py:219-226, 463-478; world-model-critic/src/world_model_critic/critic.py:171-199, 305-329, 425-440.
• Lexical stop-gap that this replaces: cortex_agent.py:53-65 (_CORRECTION_MARKERS tuple) tested by _looks_like_correction (cortex_agent.py:68-71); cortex_agent.py:347-348 (drives correction_signal).
• Phase 0/1 are design-only; gate metrics specified but unrun: rl_pipeline_design.md (Phase 0 "predictive foundation", Phase 1 "critic acceptance AUC > 0.8"); RL/policy brief open question "No held-out perplexity / Phase 0 predictive-foundation experiment has been run."
• Value head "trained with TD on every metabolize() … not validated in a real correction/uptake loop" (src/oczy/experiments/logs/SUMMARY.md, runs #75–#79).
• Policy/value actor-critic loop runs end-to-end on the real LFM2.5 driver but Stage 2 scope uptake stayed 0/8 (runs #73–#77) — token-overlap terms dominate the small learned signal, motivating a predictive gate upstream of the policy rather than more policy capacity.
• Marker ≠ correction edge case: salience run #82 (2026-06-26 salience log) — a marker-free needle turn defeats correction-marker salience.
• Paraphrase recall already implemented and survives keyword omission: run #101 (2026-06-27 paraphrase log), multi_fact_stressor.py --paraphrase.
• Saturated headline that must not be the metric: code_qa_accuracy=1.0 across runs #79,#80,#82,#84,#85,#95,#101.

Cross-links: 01-correction-to-competence-benchmark (label source / curriculum), 05-metabolism-loop-closure (the modulator this feeds), 03-layer-l-hidden-extraction (richer critic input).