INVALIDATION NOTICE — 2026-07-01
Classification: PARTIAL
Reason: The original A/B comparison body (lines 1–117) was run on code with three compounding bugs in the scope-slot reranker (all slots collapsed into one, retrieval threshold too high, topk degenerate). The conclusions drawn ("topk=3 is neutral," "scope=0.0") are INVALIDATED artifacts of the broken reranker. The "Update: bug fix changes conclusions (2026-06-30)" section (line ~118) correctly identifies and corrects these errors and is VALID.
See: 2026-06-30_scope_slot_reranker_fix.md for the full bug diagnosis; 2026-07-01_honest_post_leakage_baseline.md for the current reference point.

Scope-Slot Reranker A/B Comparison

Date: 2026-06-29 Branch: autoresearch/session-20260625 Driver: LFM2.5-1.2B-Instruct Q4 GGUF (real) Scoring: semantic neighbour fallback

Configurations Tested

All configs share the similarity-weighted boost (weight * scope_sim * overlap), which was introduced as the new default in this session. The previous flat boost (weight * overlap) is no longer tested.

Results

Stage-by-stage fixed counts

Stage 5 (Cross-domain disambiguation) detail

Analysis

Similarity-weighted boost is the critical improvement

The baseline config achieves 6/6 Stage 5 — up from 1/6 in the previous session (commit 0aba650). The only change is the scope_sim multiplier in the boost formula: weight * scope_sim * overlap instead of weight * overlap.

Mechanism: When a new request is dissimilar from a stored slot key (low cosine similarity), the boost is weakened proportionally. This prevents the reranker from boosting a wrong-sense label when the request's context doesn't match the slot's context. The previous flat boost applied the same boost regardless of request-slot similarity, causing cross-contamination between different senses of the same ambiguous word.

topk=3 is neutral

Topk=3 produces identical results to the baseline. When each request maps to exactly one slot (which is the case in this curriculum), only one slot passes the _ALLOC_THRESHOLD, so topk=3 degenerates to topk=1.

sense_split slightly hurts (5/6)

Excluding the ambiguous word from the overlap computation removes useful disambiguating signal in one Stage 5 episode. The ambiguous word itself can be part of the correct label (e.g., "file" in "disk file" is both the ambiguous word and part of the correct sense). Removing it makes the overlap computation noisier.

multi_label is catastrophic (1/6 across all stages)

Storing multiple labels per slot (joined by " | ") causes the reranker to match against all stored label parts, creating a "super-slot" that matches too broadly. The agent gets stuck on one answer ("the captain's journal" for everything in Stage 5) because the multi-label slot boosts the most-recently stored label regardless of context.

This degradation is not limited to Stage 5 — it affects ALL stages:

• Stage 0: 1/8 (vs 8/8 baseline)
• Stage 2: 0/8 (vs 8/8 baseline)
• Stage 4: 1/10 (vs 10/10 baseline)

The multi-label slot store fundamentally breaks the context-addressed retrieval that makes the reranker work.

combined inherits multi_label's toxicity

The combined config (topk3 + sense_split + multi_label) performs identically to multi_label on Stage 5 (1/6). The multi_label damage dominates any benefit from the other strategies. The slightly better Stage 0 (3/8 vs 1/8) is likely noise from the interaction of topk3's broader retrieval with multi_label's diluted labels.

Conclusion

The similarity-weighted boost (already the default) is the optimal configuration. It improves Stage 5 from 1/6 to 6/6 by preventing cross-sense contamination. The other strategies either don't help (topk3), slightly hurt (sense_split), or are catastrophic (multi_label).

Recommended config: defaults (scope_rerank_weight=2.0, scope_rerank_topk=1, scope_rerank_sense_split=False, scope_rerank_multi_label=False).

Remaining gap: scope=0.0

All configs show scope=0.0 on Stage 5 pre-accuracy, meaning no config achieves proactive cross-domain disambiguation (correctly answering before any correction). The 6/6 fixed count means the agent learns from corrections within Stage 5, but doesn't retain cross-domain knowledge from earlier stages. This is the next improvement target: enabling the scope-slot reranker to proactively select the correct sense based on context alone, without needing a correction first.

Update: bug fix changes conclusions (2026-06-30)

The A/B test above was run on code with three compounding bugs in the scope-slot reranker. The conclusions drawn from it are now known to be wrong in several respects. The historical results are preserved above as a session log; this section documents the corrected picture.

The three bugs

1. _scope_key used last_token_only=True (commit 43cfc9f): peek_embedding() with the default last_token_only=True embeds only the last token. Every curriculum request ends with ., so every request got an identical embedding (cosine sim = 1.0). All 44+ episodes collapsed into a single slot. Fix: last_token_only=False for mean-pooled whole-request embeddings.

2. _MAX_SLOTS=16 too small (commit 091046c): The slot store filled after Stage 1 (8 + 8 = 16), so Stage 2 corrections overwrote the Stage 0/1 labels. Fix: _MAX_SLOTS=64.

3. _ALLOC_THRESHOLD=0.85 used for label retrieval (commit 091046c): Mean-pooled embeddings of related-but-different requests have cosine sim ~0.3–0.65, well below 0.85. No labels were ever returned, so the reranker never fired. Fix: a separate _RETRIEVE_THRESHOLD=0.3 for label retrieval.

A fourth change (commit e316cb1) set scope_rerank_topk=3 as the new default, and commit 9e8eef4 updated the test defaults to match.

Why "topk=3 is neutral" was wrong

The original analysis observed that topk=3 produced identical results to the baseline (topk=1) and concluded topk=3 was neutral. This was an artifact of the bugs: with all slots collapsed into one (bug #1), topk=3 degenerated to topk=1 because there was only ever one slot to retrieve from. The original report even noted this degeneration ("only one slot passes the _ALLOC_THRESHOLD") but misattributed it to the curriculum having one slot per request rather than to the embedding collapse.

With the slots correctly separated (last_token_only=False, _MAX_SLOTS=64), topk=3 returns the correct technical-sense label that topk=1 misses. topk=1 only returns the single most-similar label, which is often the wrong sense. topk=3 gives the correct sense a chance to be retrieved and boosted. This is why scope_rerank_topk=3 is now the default.

Why "scope=0.0" was wrong

The scope=0.0 result across all configs was also a bug artifact: the reranker never fired because no labels passed the 0.85 retrieval threshold (bug #3). The "Remaining gap: scope=0.0" section above is therefore partially resolved — the gap was not fundamental, it was a retrieval threshold bug. With _RETRIEVE_THRESHOLD=0.3, the reranker fires and proactive cross-domain disambiguation works.

New recommended config

scope_rerank_weight=2.0
scope_rerank_topk=3
scope_rerank_sense_split=False
scope_rerank_multi_label=False

The only change from the old recommendation is scope_rerank_topk: 1 → 3. scope_rerank_weight=2.0, sense_split=False, and multi_label=False are unchanged.

New curriculum results (with the fix)

Real LFM2.5-1.2B Q4 driver, semantic scoring, topk=3, sense_split=False:

Stage 5 scope improved 0.00 → 0.50 (3/6 proactive correct), Stage 2 scope improved 0.12 → 1.00, and Stage 4 retention improved 0.10 → 1.00.

What still holds from the original report

The multi_label catastrophe conclusion still holds and was not caused by the three bugs. multi_label stores multiple labels per slot (joined by " | "), creating a "super-slot" that matches too broadly and boosts the most-recently stored label regardless of context. This breaks the context-addressed retrieval that makes the reranker work, and it degrades every stage, not just Stage 5. scope_rerank_multi_label=False remains correct.

The sense_split slightly hurts observation is also unaffected by the bug fixes in its reasoning (excluding the ambiguous word from overlap removes useful disambiguating signal), though its magnitude should be re-measured on the fixed code.

Reference

The detailed fix report, including the per-commit diagnosis and the full post-fix curriculum numbers, is in 2026-06-30_scope_slot_reranker_fix.md.