Campaign 959e114 — Research/20 INT8 meta_cortex/v2 DEV Training

Date: 2026-07-15 – 2026-07-16 Campaign ID: 959e114e44ffbd0183c94e9aa11f60ad4e94156e

Goal

Launch a clean INT8 meta_cortex/v2 developmental-training campaign on Kaggle CPU, replacing the obsolete FP32 meta_cortex/v1 queue. Train five checkpoint models from five canonical developmental seeds, then dispatch 90 DEV calibration shard jobs. Meta-test execution remains out of scope and blocked.

Immutable source commits and providers

Executed-on-remote commit: 959e114e44ffbd0183c94e9aa11f60ad4e94156e

Source dataset: abdellahkadem/oczy-source-959e114e44ff Source archive SHA-256: d56249037f2757a546684f6206634aacd724af3825f7f6dbf075db79ad5a3c18

Contract: All jobs ran under CPU-only contract (cuda_available=false, torch 2.10.0+cpu). Runtime manifest SHA-256: a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e.

Lawful scope

This campaign freezes the instrument at meta_cortex/v2 and the runtime at oczy/runtime-manifest/v2. No metrics, thresholds, baselines, episodes, scoring, acceptance criteria, kill criteria, or manifests were modified. Meta-test remains blocked — no sealed seed was inspected or published, no meta-test command appears in any job or config.

Instrument and runtime identity

• Instrument: meta_cortex/v2 (INT8 organ cutover from FP32 v1)
• Definition SHA-256: f62f18ef3dd4eb7cf62d82e24b7c0fea5011516dbaf16d3ea50cc7890c9db14f
• Public instrument dataset: abdellahkadem/oczy-r20-int8-calibration-f62f18ef3dd4
• Archive SHA-256: 46216f6248226b6ead260efe48e33fe31e16fa40169c6ff3ed9a39313b0b37e1
• Organ identity: Qwen/Qwen2.5-0.5B-Instruct@torchao-int8-weight-only-w8a32-per-row-v2
• Kaggle organ hash: a342431c0fdb02bf1bbed95255795ad52df3e799c821318c6206021a46a3f9ea
• Quantization: TorchAO 0.17.0, Int8WeightOnlyConfig v2, per-row, FP32 activations
• Dependencies: Python 3.12.13, torch 2.10.0+cpu, transformers 5.0.0, tokenizers 0.22.2, safetensors 0.7.0, torchao 0.17.0 (pinned offline wheel 87a418ce…)

Concurrency and batch sequence

Kaggle jobs ran as private, internet-off CPU kernels with 60-second inter-account submission pacing. Pool inventory limit: 2. Concurrency: 5.

Per-run results — First wave (d0–d4, 30 train tasks, 2 outer steps)

Observations: 2 outer steps × 30 train tasks × 3 families exceeds Kaggle's 12h CPU hard limit for 3 of 5 seeds. d2 and d4 completed; d0/d1/d3 were killed.

Per-run results — Second wave (d0h/d1h/d3h, 15 train tasks, 2 outer steps)

Replacement jobs used 15 training tasks to fit the 12h window. d0h and d3h completed; d1h still exceeded the hard limit.

Per-run results — Third wave (d1q8, 8 train tasks, 2 outer steps)

d1q8 is a uniquely named retry of the remaining seed. It changes only train_tasks_per_family from 15 to 8; seed, outer steps, validation tasks, optimizer, model, runtime, source, and instrument bindings remain unchanged.

Infrastructure fixes

1. Bootstrap routing (959e114): Kaggle bootstrap was passing --observed-manifest-json directly to oczy.experiments.meta_cortex, which does not own that CLI. Fixed by routing through infrastructure.kaggle.run_experiment_module with experiment arguments forwarded via repeated --arg entries. Added kernel_id to job spec for execution provenance. 401 campaign regression tests passed.

2. Orchestrator crash-loop (c8daf85): The staged orchestrator raised OrchestratorError on any non-zero scheduler exit, triggering an on-failure restart loop (65+ restarts observed). Fixed with _raise_if_jobs_not_terminal() guard: when all jobs are terminal, the orchestrator lets validation produce a single clear error instead of crash-looping.

3. Stale lock cleanup (ebb7b6e): Added _clean_stale_lock() before both scheduler stages. Detects orphaned state.json.owner.lock files whose owner PID is dead and removes them, preventing lock-contention cascades when the orchestrator restarts.

4. Lock contention under crash pressure: The orchestrator's in-process scheduler + hub on-failure restart creates lock-contention cascades when the scheduler process survives the orchestrator's exit. Workaround: run scheduler standalone via systemd user service.

5. Retry-aware orchestration (920c3ef): Training validation now preserves failed attempts while accepting uniquely named retries. It requires exact canonical seed coverage and at least one runtime-manifest-verified success per seed; missing/unexpected seeds and unverified successes fail closed.

Durable scheduling

Training scheduler deployed as systemd user service:

• Unit: oczy-r20-training.service
• Path: ~/.config/systemd/user/oczy-r20-training.service
• Restart: on-failure with 60s backoff; exit status 1 is terminal and does not restart (RestartPreventExitStatus=1)
• Enabled: yes (survives logout and reboots)
• Poll interval: 30s
• Job timeout: 46,800s locally, allowing provider completion/collection lag beyond Kaggle's 12h execution cap
• Shared lease: ~/.local/state/oczy/runner-pool-leases.json
• Automatic continuation: training unit OnSuccess starts oczy-r20-dev-orchestrator.service. The staged orchestrator revalidates one verified success per canonical seed, publishes the checkpoint archive, then materializes and dispatches calibration.
• Calibration fan-out: 90 DEV tasks / 5-task ranges = 18 ranges; 18 ranges × 5 checkpoint seeds = 90 calibration jobs.

Seed distributions

Not yet available — pending completion of all five training jobs. The calibration stage will produce cross-seed variance data across 90 shards.

Nulls and refutations

• Infrastructure NULL (d0/d1/d3 first wave): Three seeds failed to complete within Kaggle's 12h CPU limit. This is an infrastructure constraint, not a scientific verdict. The 15-task replacement wave is in progress.
• Infrastructure NULL (d4 scheduler timeout): d4 completed on Kaggle but the scheduler's 12h job timeout fired before output collection. Manual recovery confirmed exit 0, runtime manifest verified, checkpoint intact. This is a scheduler timeout calibration issue, not a scientific verdict.

Non-runnable inventory

• All 90 DEV calibration shard jobs are gated on successful completion of all five training checkpoints and their archive/publication.
• Meta-test remains blocked pending DEV calibration completion, power analysis, candidate finalization, and explicit human signoff.

Artifact provenance paths

Campaign root: ~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v2/

Initial next steps (superseded)

These launch-time steps are retained as history. The dated recovery sections below record the completed training, failed original calibration, and active bounded requeue.

1. Wait for d1q8 to complete (systemd service oczy-r20-training).
2. Automatic continuation validates all five checkpoints: identity, organ hash, seeds, source commit, and runtime manifest.
3. It builds the deterministic checkpoint archive and publishes the private Kaggle dataset.
4. It generates and dispatches 90 DEV calibration shard jobs.
5. After collection: local merge → calibrate-dev → POWER_ANALYSIS.json → candidate finalization.
6. Meta-test remains blocked pending explicit human signoff on exact manifest hash, margin, and task count.

2026-07-19 transport recovery and bounded calibration requeue

The original v3 calibration execution is infrastructure-invalid, not a scientific result. All 90 jobs failed before experiment code because Kaggle auto-extracted the registered instrument-public.tar.gz; the bootstrap required the whole archive as a regular file for SHA-256 verification.

Commit 949871b3d2d6cbb8a68af41396639cfb3f79ee5e completed the transport cutover:

• generic offline archives now require opaque .tar.gz.bin transport names while retaining format=tar.gz, whole-archive SHA-256 verification, and explicit r:gz extraction;
• checkpoint publication writes checkpoints.tar.gz.bin;
• the publication resume record is bound to dataset slug, published filename, and archive SHA, so a legacy boolean cannot skip the corrected publication;
• focused verification completed with 200 passing tests.

The unchanged registered bytes were republished under new immutable private dataset identities:

The first corrected five-task v4 canary successfully extracted both opaque archives and observed the exact runtime manifest a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e, but Kaggle killed it at the 12-hour CPU limit before it wrote a shard. This is a second infrastructure null: five calibration tasks are too wide for the provider window.

The v5 execution changes only the control-plane partition:

• one-task canary r20-i8v5-cal-949871b-d0-t00-01 succeeded in 2h50m;
• source, CPU hardware, runtime manifest, archive hashes, organ hash, shard header hashes, shard self-hash, and output lists passed the promotion gate;
• the complete partition contains 116 jobs including the canary, covers every one of the 5 × 90 registered (developmental seed, task) cells exactly once, and limits remaining jobs to at most four tasks;
• after canary promotion, the scheduler started five Kaggle CPU jobs in parallel. At the initial fanout check: 1 succeeded, 5 running, 110 pending, 0 failed.

Durable paths:

• campaign: ~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v5/;
• batch: calibration-949871b/batch.json;
• state: calibration-949871b/state.json;
• dispatch plan: calibration-949871b/dispatch-plan.json;
• service: oczy-r20-v5-cal-fanout.service;
• shared lease: ~/.local/state/oczy/runner-pool-leases.json.

Meta-test remains blocked and was not accessed. DEV aggregation and scientific interpretation remain pending completion and collection of the bounded fanout.

Live bounded-fanout update — 2026-07-19 20:02 UTC

The first five four-task shards reached terminal state:

The live scheduler state is now 5 succeeded (including the one-task canary), 1 failed, 5 running, and 105 locally pending. The fanout service remains active and keeps all five Kaggle CPU slots occupied.

This establishes that four-task shards do not have sufficient deterministic margin under the provider's 12-hour limit: four completed, but one crossed the limit. The failed attempt remains preserved as required by queue policy; it was not rewritten to pending and has not been counted as a scientific null. A future replacement must use a new immutable job identity and a narrower task range. The currently running and pending jobs remain execution-in-progress and provide no scientific evidence yet.

Source commit, model, runtime manifest, instrument, checkpoints, registered seeds, tasks, conditions, metrics, and scoring remain unchanged. Meta-test remains blocked and was not accessed.

Failed-shard diagnosis and immutable replacements — 2026-07-20 16:34 UTC

A second four-task shard, cal-d0-t21-25, terminated after 12.013h with the same provider error as cal-d0-t09-13 at 12.010h. Both attempts reached Kaggle's 12-hour CPU execution limit. They remain preserved as failed attempts; neither state entry was reset or overwritten.

The control plane was paused while Kaggle OAuth was renewed. The five already submitted kernels continued remotely, and authenticated status checks confirmed they were still running. No experiment process or artifact was altered.

The two failed ranges were split into four new immutable two-task jobs:

Only the execution partition changed. Source commit 949871b3d2d6cbb8a68af41396639cfb3f79ee5e, model, runtime manifest a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e, organ hash, instrument, checkpoints, registered DEV cells, metrics, thresholds, baselines, and scoring remain unchanged. All replacement kernels are private, internet-off, CPU-only jobs.

The canonical batch now contains 120 jobs. The regenerated dispatch plan reported all_assigned=true, ready_for_dispatch=true, and no errors. The fanout service resumed active/running with 13 succeeded, 2 preserved failed, 5 running, and 100 pending; all four replacements are pending with zero attempts and will enter the provider when capacity opens. Meta-test remains blocked and was not accessed.

Partition-dependent C6 invalidation and corrected width-3 restart — 2026-07-22

Runtime evidence reached 29 successful and 6 provider-timeout four-task attempts. Successful provider execution ranged from 8.616h to 11.764h; the four-task width therefore has no deterministic margin under Kaggle's 12-hour CPU limit.

A subsequent partition audit invalidated the stronger claim above that shard width changes only execution packaging. In source commit 949871b…, collect_calibration_shard selected the C6 state_swapped donor from the next task inside the current shard. Changing shard boundaries therefore changed the donor state and the scored state_addressing_delta. One-task shards omitted C6 entirely, while endpoint aggregation requires all six C1–C6 conditions. Consequently, every v5 shard remains valid infrastructure/runtime evidence but the collection is invalid for scientific aggregation. No v5 metric or scientific verdict may be derived from it.

After this was disclosed, the human decision was: version fix, allow the five already-active legacy kernels to finish, then perform a full width-3 DEV rerun. Commit a8c98d638209a8425b14a0f853e9fc46ae7da581 makes donor selection partition-independent by using the canonical next task within each full frozen validation family. Width-1 collection now includes C6, and the same target cell is invariant across shard selections. Focused verification: 89 passed; the partition-invariance subset independently passed 4 passed.

Corrected immutable source:

• dataset: abdellahkadem/oczy-source-a8c98d638209;
• archive SHA-256: 40708cb9e195cba45302d64d11a9110cc4f91b74ce0325b5a4856908b1e941ca;
• source worktree dirty entries: 0.

The v5 scheduler now drains only the 31 succeeded, 6 failed, and 5 running attempts present at cutover; its original 120-job batch remains preserved as batch-pre-width3.json and batch-full.json. No never-submitted width-4 job will be launched.

The corrected v6 campaign contains exactly 150 three-task jobs and covers all 5 × 90 registered DEV cells exactly once. Every generated kernel is private, internet-off, CPU-only, and bound to the unchanged runtime manifest, model, organ, public instrument archive, and checkpoint archive. The width-3 canary cal3-d0-t00-03 is pending behind the five legacy kernels. Its durable systemd chain will verify corrected source provenance, runtime identity, CPU-only hardware, shard self-hash, exact record counts, and C1–C6 presence before atomically promoting the remaining 149 jobs.

Durable corrected paths:

• campaign: ~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v6/;
• queue: calibration-a8c98d6/;
• canary service: oczy-r20-v6-cal-width3-canary.service;
• promotion service: oczy-r20-v6-cal-width3-promote.service;
• fanout service: oczy-r20-v6-cal-width3.service.

Meta-test remains blocked and was not accessed. No thresholds, metrics, registered tasks, seeds, checkpoints, model, or scoring formulas changed.