Real-Driver Needle Sweep — 2026-06-27

Question

Can the IngestionPipeline use a real foreign CPU sentence embedder to reduce wall-clock time versus same-LM embeddings on the LFM2.5 GGUF driver for the needle-recall stressor?

Method

Extended src/oczy/experiments/needle_sweep.py to optionally load the real LFM2.5 GGUF driver (--use-real-driver, --n-ctx). Added per-position and total wall-clock timing.

Real-driver runs at length 512, positions [0.0, 0.25, 0.5, 0.75, 1.0], 64-token chunks, lexical-novelty salience.

Also cached the MiniLM sentence-transformer model across embedder instances (after noticing the model was re-initializing for every position).

Results

Length 512, cached MiniLM

Both achieve perfect recall. Foreign-MiniLM is slower at this scale.

Interpretation

• lexical-novelty salience keeps only 1-2 chunks per position at length 512, so the total number of embeddings is small.
• LlamaCVecDriver.peek_embedding caches embeddings by prompt, and the filler chunks are mostly identical across positions; after the first chunk, same-LM embedding calls are essentially free.
• Foreign-MiniLM encodes each surviving chunk via torch on CPU. Even with a cached model, per-call overhead exceeds cached same-LM lookups in this regime.

This means the naive expectation ("foreign embedder is always cheaper") is wrong for short/medium turns with high lexical repetition. The cost advantage appears only when:

1. The turn contains many distinct chunks (cache misses dominate),
2. The foreign embedder is fast (e.g. ONNX/Core ML), or
3. Same-LM embedding is deliberately uncached or batched per-position.

Implication for architecture

The embedder fork decision depends strongly on input statistics. The current foreign-minilm integration is correct but not a universal win. To see savings, we must run length 4096 or higher where same-LM does many distinct forward passes.

Open questions

1. Does foreign-minilm beat same-lm at length 4096 or 8192?
2. Would pass-through salience (embed all chunks) change the ratio?
3. Is there a smaller/faster sentence model (ONNX) that reduces per-chunk overhead enough to win at length 512?

Artifacts

• src/oczy/experiments/needle_sweep.py
• src/oczy/experiments/tests/test_ingestion_needle.py
• src/oczy/experiments/ingestion.py (MiniLM cache)

Commits

• fb954fe — Make needle_sweep runnable against real LlamaCVecDriver; cache MiniLM model.
• dc408b5 — Update SUMMARY with run #89.

Run

Run #89: benchmark code_qa_accuracy=1.0, fast suite 300 passed.