kinoTou

experiment log

Real-Driver Needle Sweep at Length 4096 — 2026-06-27

File
2026-06-27_length_4096_needle_sweep.md
Size
2.7 KB
SHA-256
ac6dd3d757679b8c…
Download raw file ↓Back to research →
Primary source. This is the verbatim Oczy document. The analytical field notes on the research page interpret and summarize these sources.

Real-Driver Needle Sweep at Length 4096 — 2026-06-27

Question

At length 4096, does the foreign-MiniLM embedder become cheaper than same-LM because many chunks must be embedded and same-LM embedding cache hits drop?

Method

Reused the real-driver needle_sweep.py with --length 4096, --n-ctx 8192, 256-token chunks, and pass-through salience so every chunk is embedded (17 chunks per position). Ran 2 positions (0.0 and 0.5) for each embedder to keep wall time bounded.

Results

embedder recall@0.0 recall@0.5 mean_recall traces total wall seconds
same-lm 1 0 0.50 17 20.1
foreign-minilm 1 0 0.50 17 30.2

Interpretation

  • Same-lm is still faster than foreign-minilm even when 17 chunks are embedded. The gap is smaller than at length 512 (20.1s vs 30.2s for 2 positions, vs 29.6s vs 42.8s at length 512 for 5 positions), but same-lm remains ahead.
  • The 0.5-position needle was missed because with 256-token chunks the needle tokens do not always land in a chunk that survives as a distinct trace (pass-through keeps all chunks, but hippocampal replay matches on query text, and the exact needle text may not be prominent in the retrieved chunk). This is a chunk-boundary artifact, not an architecture finding.
  • LlamaCVecDriver.peek_embedding appears to be efficiently cached and llama.cpp's embedding-only forward path is fast enough that the CPU torch overhead of MiniLM does not win.

Implication for architecture

The embedder fork, under the tested local CPU environment and tested sentence model, favors same-lm. Foreign-MiniLM is not a universal cost win. Before pursuing foreign embeddings further, the better leverage would be:

  1. Use a much faster foreign backend (ONNX Runtime / Core ML / quantized), or GPU-offloaded torch.
  2. Increase context to 32k+ where same-lm embedding batching/caching breaks down.
  3. Measure embedding-call counts directly on the real driver to confirm cache behavior.

Given that same-lm is fast and accurate enough, and foreign-minilm adds a heavy optional dependency without a clear win, the priority should return to the original north star: behavior_delta_per_byte_of_persistent_memory.

Open questions

  1. Would an ONNX foreign embedder change the cost ratio?
  2. Is there a length/context where same-lm consistently loses?
  3. Does disabling same-lm embedding cache (if possible) expose the true cost?

Artifacts

  • No code changes in this iteration; reused needle_sweep.py.
  • Data recorded in autoresearch run #90.

Commits

  • 205ae57 — Update SUMMARY.md with run #90 findings.

Run

Run #90: benchmark code_qa_accuracy=1.0, fast suite 300 passed (no code changes, so same suite as run #89).