Experiment Log: Extended Learning Component Evaluation

Date: 2026-06-19 Experiment: Train all seven Oczy modules on a shared 30-item word-sense/trivia curriculum and evaluate each against task-specific metrics. Evaluator: NCode Command: uv run python eval_extended.py

Hypothesis / Goal

The Plastic World Model Agent architecture proposes that experience should flow through a metabolism:

experience -> fast change -> replay -> compression -> slow change -> forgetting raw trace

Most projects never test this end-to-end. This experiment asks: after an extended learning curriculum, which of the seven organs actually learns something, stays bounded in memory, and preserves the learned behavior?

Method

1. Curriculum: 30 word-sense ambiguity corrections (e.g. "profile" -> business vertical, "branch" -> git branch) plus 12 unrelated trivia facts as controls.
2. Training: Each module is exposed to the curriculum through a thin wrapper that maps answer() / correct() calls to the module's native API.
3. Metrics per module:
   • PlasticCortex: correction accuracy vs. normal-text-only control; plasticity ratio.
   • NeuralHippocampus: surprise-gating precision, replay accuracy before consolidation, compression ratio, slow-update rate.
   • WorldModelCritic: correction likelihood on repeated/similar queries vs. unrelated queries; discrimination and calibration.
   • IdentityHypernetwork: adapter retrieval accuracy, identity shift magnitude.
   • SkillImmuneCortex: detector precision, recall, F1, merge ratio.
   • ExperienceAutoencoder: compression ratio, reconstruction error, final identity size.
4. Baselines: Oracle agent (upper bound) and Always-Wrong agent (lower bound).

Results

Per-module details

• NeuralHippocampus: perfect surprise gating (high-surprise episodes stored, low-surprise trivia rejected), 100% raw-trace replay accuracy, strong trace compression, and every consolidated cluster became a slow update. This is the only module that exercises the full fast-write / replay / consolidate / decay loop.

• PlasticCortex: 96.7% correction accuracy with a normal-text exposure control near zero. The fast-weight correction gate clearly overrides slow priors. Still a word-association toy with no semantic generalization.

• SkillImmuneCortex: 0.98 F1 on trigger detection. Detectors fire on relevant queries and ignore unrelated trivia. Merging did not reduce detector count in this sparse curriculum, so compression reward was zero.

• WorldModelCritic: learns that corrected queries are risky, but the hard-coded ambiguity list and Jaccard sentence similarity fail to separate similar from unrelated queries. Discrimination is weak.

• ExperienceAutoencoder: total encoded identity is tiny (256 bytes), but the encoder is an untrained random projection. Reconstruction error is high.

• IdentityHypernetwork: fixed CONCEPT_VOCABULARY does not include the curriculum senses, so adapter retrieval fails completely. The latent shifts, but it cannot express the learned distinctions.

Conclusion

1. NeuralHippocampus is the most promising organ. It is the only module that validates the architecture's central claim: raw memory can be compressed and forgotten while the learned structure survives.

2. PlasticCortex proves the gate mechanism works, but the toy model needs real embeddings or adapters for transfer and scope control.

3. IdentityHypernetwork needs the most rework. Replace the hard-coded concept list with an open vocabulary/tokens before it can learn.

4. ExperienceAutoencoder should prioritize a trained encoder; the latent size is right but the signal is not preserved.

5. WorldModelCritic needs learned sentence similarity and structured outcome prediction, not a single scalar correction probability.

Next steps

• Wire NeuralHippocampus slow updates into PlasticCortex's inference path.
• Add a real embedding layer to IdentityHypernetwork and retest.
• Train the ExperienceAutoencoder encoder on synthetic episodes.
• Replace the WorldModelCritic's hard-coded features with a small learned similarity model.
• Re-run this evaluation after each organ upgrade.

Artifacts

• eval_extended.py — full evaluation script
• EVALUATION.md — summarized report
• correction-benchmark/ — canonical benchmark harness