# Campaign 959e114 — Research/20 INT8 meta_cortex/v2 DEV Training

**Date:** 2026-07-15 – 2026-07-16
**Campaign ID:** `959e114e44ffbd0183c94e9aa11f60ad4e94156e`

## Goal

Launch a clean INT8 `meta_cortex/v2` developmental-training campaign on Kaggle
CPU, replacing the obsolete FP32 `meta_cortex/v1` queue. Train five checkpoint
models from five canonical developmental seeds, then dispatch 90 DEV calibration
shard jobs. Meta-test execution remains out of scope and blocked.

## Immutable source commits and providers

| Commit | Provider(s) | Jobs |
|--------|-------------|------|
| `959e114e44ffbd0183c94e9aa11f60ad4e94156e` | kaggle (CPU-only) | Training d0–d4 (first wave, 30 tasks); d0h/d1h/d3h (second wave, 15 tasks); d1q8 (third wave, 8 tasks) |

**Executed-on-remote commit:** `959e114e44ffbd0183c94e9aa11f60ad4e94156e`

**Source dataset:** `abdellahkadem/oczy-source-959e114e44ff`
**Source archive SHA-256:** `d56249037f2757a546684f6206634aacd724af3825f7f6dbf075db79ad5a3c18`

**Contract:** All jobs ran under CPU-only contract (`cuda_available=false`,
`torch 2.10.0+cpu`). Runtime manifest SHA-256:
`a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e`.

## Lawful scope

This campaign freezes the instrument at `meta_cortex/v2` and the runtime at
`oczy/runtime-manifest/v2`. No metrics, thresholds, baselines, episodes,
scoring, acceptance criteria, kill criteria, or manifests were modified.
Meta-test remains blocked — no sealed seed was inspected or published, no
meta-test command appears in any job or config.

## Instrument and runtime identity

- **Instrument:** `meta_cortex/v2` (INT8 organ cutover from FP32 v1)
- **Definition SHA-256:** `f62f18ef3dd4eb7cf62d82e24b7c0fea5011516dbaf16d3ea50cc7890c9db14f`
- **Public instrument dataset:** `abdellahkadem/oczy-r20-int8-calibration-f62f18ef3dd4`
- **Archive SHA-256:** `46216f6248226b6ead260efe48e33fe31e16fa40169c6ff3ed9a39313b0b37e1`
- **Organ identity:** `Qwen/Qwen2.5-0.5B-Instruct@torchao-int8-weight-only-w8a32-per-row-v2`
- **Kaggle organ hash:** `a342431c0fdb02bf1bbed95255795ad52df3e799c821318c6206021a46a3f9ea`
- **Quantization:** TorchAO 0.17.0, Int8WeightOnlyConfig v2, per-row, FP32 activations
- **Dependencies:** Python 3.12.13, torch 2.10.0+cpu, transformers 5.0.0, tokenizers 0.22.2, safetensors 0.7.0, torchao 0.17.0 (pinned offline wheel `87a418ce…`)

## Concurrency and batch sequence

| Wave | Batch SHA-256 | Jobs submitted | Seeds | Train tasks/family | Result |
|------|--------------|----------------|-------|--------------------|--------|
| 1 | `abd22fd6…` | d0–d4 | 5 canonical | 30 | 1 COMPLETE (d2), 1 recovered (d4), 3 BLOCKED (12h CPU limit) |
| 2 | `5dffa1bc…` | d0h/d1h/d3h | 3 failed seeds | 15 | d0h COMPLETE; d3h recovered after scheduler timeout; d1h BLOCKED by 12h CPU limit |
| 3 | `360a75f3…` | d1q8 | remaining seed | 8 | RUNNING under systemd `oczy-r20-training.service` |

Kaggle jobs ran as private, internet-off CPU kernels with 60-second inter-account
submission pacing. Pool inventory limit: 2. Concurrency: 5.

## Per-run results — First wave (d0–d4, 30 train tasks, 2 outer steps)

| Job | Seed | Kaggle status | Scheduler | Checkpoint | Notes |
|-----|------|--------------|-----------|------------|-------|
| d0 (`7057…`) | `7057119071884125452` | `CANCEL_ACKNOWLEDGED` | failed (timeout) | none | Killed by Kaggle 12h CPU limit |
| d1 (`5996…`) | `5996565330439180020` | `CANCEL_ACKNOWLEDGED` | failed (timeout) | none | Killed by Kaggle 12h CPU limit |
| d2 (`8811…`) | `8811378008969065672` | `COMPLETE` | **succeeded** | yes | Runtime manifest verified; theta.npz + checkpoint.json collected |
| d3 (`8429…`) | `8429388655155408982` | `CANCEL_ACKNOWLEDGED` | failed (timeout) | none | Killed by Kaggle 12h CPU limit |
| d4 (`8461…`) | `8461256526005535010` | `COMPLETE` | failed (timeout) | yes | Completed on Kaggle; scheduler timed out before collection. Manually recovered, runtime manifest verified. |

**Observations:** 2 outer steps × 30 train tasks × 3 families exceeds Kaggle's
12h CPU hard limit for 3 of 5 seeds. d2 and d4 completed; d0/d1/d3 were killed.

## Per-run results — Second wave (d0h/d1h/d3h, 15 train tasks, 2 outer steps)

| Job | Seed | Status |
|-----|------|--------|
| d0h | `7057119071884125452` | **Succeeded**, runtime manifest verified |
| d1h | `5996565330439180020` | `CANCEL_ACKNOWLEDGED`; no checkpoint |
| d3h | `8429388655155408982` | **Completed** on Kaggle; scheduler timed out before collection; output recovered, exit 0, runtime verified |

Replacement jobs used 15 training tasks to fit the 12h window. d0h and d3h
completed; d1h still exceeded the hard limit.

## Per-run results — Third wave (d1q8, 8 train tasks, 2 outer steps)

| Job | Seed | Status |
|-----|------|--------|
| d1q8 | `5996565330439180020` | **Running** |

d1q8 is a uniquely named retry of the remaining seed. It changes only
`train_tasks_per_family` from 15 to 8; seed, outer steps, validation tasks,
optimizer, model, runtime, source, and instrument bindings remain unchanged.

## Infrastructure fixes

1. **Bootstrap routing (959e114):** Kaggle bootstrap was passing
   `--observed-manifest-json` directly to `oczy.experiments.meta_cortex`, which
   does not own that CLI. Fixed by routing through
   `infrastructure.kaggle.run_experiment_module` with experiment arguments
   forwarded via repeated `--arg` entries. Added `kernel_id` to job spec for
   execution provenance. 401 campaign regression tests passed.

2. **Orchestrator crash-loop (c8daf85):** The staged orchestrator raised
   `OrchestratorError` on any non-zero scheduler exit, triggering an
   `on-failure` restart loop (65+ restarts observed). Fixed with
   `_raise_if_jobs_not_terminal()` guard: when all jobs are terminal, the
   orchestrator lets validation produce a single clear error instead of
   crash-looping.

3. **Stale lock cleanup (ebb7b6e):** Added `_clean_stale_lock()` before both
   scheduler stages. Detects orphaned `state.json.owner.lock` files whose owner
   PID is dead and removes them, preventing lock-contention cascades when the
   orchestrator restarts.

4. **Lock contention under crash pressure:** The orchestrator's in-process
   scheduler + hub `on-failure` restart creates lock-contention cascades when
   the scheduler process survives the orchestrator's exit. Workaround: run
   scheduler standalone via systemd user service.

5. **Retry-aware orchestration (920c3ef):** Training validation now preserves
   failed attempts while accepting uniquely named retries. It requires exact
   canonical seed coverage and at least one runtime-manifest-verified success
   per seed; missing/unexpected seeds and unverified successes fail closed.

## Durable scheduling

Training scheduler deployed as systemd user service:

- **Unit:** `oczy-r20-training.service`
- **Path:** `~/.config/systemd/user/oczy-r20-training.service`
- **Restart:** `on-failure` with 60s backoff; exit status 1 is terminal and
  does not restart (`RestartPreventExitStatus=1`)
- **Enabled:** yes (survives logout and reboots)
- **Poll interval:** 30s
- **Job timeout:** 46,800s locally, allowing provider completion/collection
  lag beyond Kaggle's 12h execution cap
- **Shared lease:** `~/.local/state/oczy/runner-pool-leases.json`
- **Automatic continuation:** training unit `OnSuccess` starts
  `oczy-r20-dev-orchestrator.service`. The staged orchestrator revalidates one
  verified success per canonical seed, publishes the checkpoint archive, then
  materializes and dispatches calibration.
- **Calibration fan-out:** 90 DEV tasks / 5-task ranges = 18 ranges;
  18 ranges × 5 checkpoint seeds = **90 calibration jobs**.

## Seed distributions

Not yet available — pending completion of all five training jobs. The
calibration stage will produce cross-seed variance data across 90 shards.

## Nulls and refutations

- **Infrastructure NULL (d0/d1/d3 first wave):** Three seeds failed to complete
  within Kaggle's 12h CPU limit. This is an infrastructure constraint, not a
  scientific verdict. The 15-task replacement wave is in progress.
- **Infrastructure NULL (d4 scheduler timeout):** d4 completed on Kaggle but
  the scheduler's 12h job timeout fired before output collection. Manual
  recovery confirmed exit 0, runtime manifest verified, checkpoint intact.
  This is a scheduler timeout calibration issue, not a scientific verdict.

## Non-runnable inventory

- All 90 DEV calibration shard jobs are gated on successful completion of all
  five training checkpoints and their archive/publication.
- Meta-test remains blocked pending DEV calibration completion, power analysis,
  candidate finalization, and explicit human signoff.

## Artifact provenance paths

Campaign root:
`~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v2/`

| Artifact | Path |
|----------|------|
| Training batch (retry-aware) | `training-959e114/batch.json` |
| Training state | `training-959e114/state.json` |
| Dispatch plan | `training-959e114/dispatch-plan.json` |
| Training kernels | `training-959e114/jobs/d0h/`, `d1h/`, `d1q8/`, `d3h/` |
| Training results | `training-959e114/results/d0h/`, `d2/`, `d3h/`, `d4/` |
| Instrument definition | `instrument-v2-959e114/DEFINITION.json` |
| Orchestrator config | `orchestrator-config-959e114.json` |
| Orchestrator state | `orchestrator-state-959e114.json` |
| Runtime manifest | `runtime/runtime_manifest.json` |
| Pinned TorchAO wheel | `dependencies/torchao-0.17.0-…` |
| Sealed meta-test seed | `secrets/meta_test_seed.bin` (mode 0600, never inspected) |

## Initial next steps (superseded)

These launch-time steps are retained as history. The dated recovery sections
below record the completed training, failed original calibration, and active
bounded requeue.

1. Wait for d1q8 to complete (systemd service `oczy-r20-training`).
2. Automatic continuation validates all five checkpoints: identity, organ
   hash, seeds, source commit, and runtime manifest.
3. It builds the deterministic checkpoint archive and publishes the private
   Kaggle dataset.
4. It generates and dispatches 90 DEV calibration shard jobs.
5. After collection: local merge → `calibrate-dev` → `POWER_ANALYSIS.json` →
   candidate finalization.
6. Meta-test remains blocked pending explicit human signoff on exact manifest
   hash, margin, and task count.

## 2026-07-19 transport recovery and bounded calibration requeue

The original v3 calibration execution is **infrastructure-invalid**, not a
scientific result. All 90 jobs failed before experiment code because Kaggle
auto-extracted the registered `instrument-public.tar.gz`; the bootstrap
required the whole archive as a regular file for SHA-256 verification.

Commit `949871b3d2d6cbb8a68af41396639cfb3f79ee5e` completed the transport
cutover:

- generic offline archives now require opaque `.tar.gz.bin` transport names
  while retaining `format=tar.gz`, whole-archive SHA-256 verification, and
  explicit `r:gz` extraction;
- checkpoint publication writes `checkpoints.tar.gz.bin`;
- the publication resume record is bound to dataset slug, published filename,
  and archive SHA, so a legacy boolean cannot skip the corrected publication;
- focused verification completed with 200 passing tests.

The unchanged registered bytes were republished under new immutable private
dataset identities:

| Input | Dataset | SHA-256 |
|-------|---------|---------|
| Source | `abdellahkadem/oczy-source-949871b3d2d6` | `4396aa7a0ad3731c343055a66aa98e8e23a878e6e4bbe654876a73cafb991790` |
| Public instrument | `abdellahkadem/oczy-r20-int8-calibration-46216f624822-opaque-v1` | `46216f6248226b6ead260efe48e33fe31e16fa40169c6ff3ed9a39313b0b37e1` |
| Five checkpoints | `abdellahkadem/oczy-r20-int8-checkpoints-03dfd23873e8-opaque-v1` | `03dfd23873e83431494237ed3ac5b99db99e5df17379e81caddde8363c366eb9` |

The first corrected five-task v4 canary successfully extracted both opaque
archives and observed the exact runtime manifest
`a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e`,
but Kaggle killed it at the 12-hour CPU limit before it wrote a shard. This is
a second infrastructure null: five calibration tasks are too wide for the
provider window.

The v5 execution changes only the control-plane partition:

- one-task canary `r20-i8v5-cal-949871b-d0-t00-01` succeeded in 2h50m;
- source, CPU hardware, runtime manifest, archive hashes, organ hash, shard
  header hashes, shard self-hash, and output lists passed the promotion gate;
- the complete partition contains 116 jobs including the canary, covers every
  one of the 5 × 90 registered `(developmental seed, task)` cells exactly once,
  and limits remaining jobs to at most four tasks;
- after canary promotion, the scheduler started five Kaggle CPU jobs in
  parallel. At the initial fanout check: 1 succeeded, 5 running, 110 pending,
  0 failed.

Durable paths:

- campaign: `~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v5/`;
- batch: `calibration-949871b/batch.json`;
- state: `calibration-949871b/state.json`;
- dispatch plan: `calibration-949871b/dispatch-plan.json`;
- service: `oczy-r20-v5-cal-fanout.service`;
- shared lease: `~/.local/state/oczy/runner-pool-leases.json`.

Meta-test remains blocked and was not accessed. DEV aggregation and scientific
interpretation remain pending completion and collection of the bounded fanout.

### Live bounded-fanout update — 2026-07-19 20:02 UTC

The first five four-task shards reached terminal state:

| Job | Duration | Outcome |
|-----|----------|---------|
| `cal-d0-t01-05` | 10.821h | succeeded |
| `cal-d0-t05-09` | 11.778h | succeeded |
| `cal-d0-t09-13` | 12.010h | **infrastructure failure** — Kaggle execution limit |
| `cal-d0-t13-17` | 10.666h | succeeded |
| `cal-d0-t17-21` | 11.188h | succeeded |

The live scheduler state is now 5 succeeded (including the one-task canary),
1 failed, 5 running, and 105 locally pending. The fanout service remains active
and keeps all five Kaggle CPU slots occupied.

This establishes that four-task shards do not have sufficient deterministic
margin under the provider's 12-hour limit: four completed, but one crossed the
limit. The failed attempt remains preserved as required by queue policy; it was
not rewritten to pending and has not been counted as a scientific null. A
future replacement must use a new immutable job identity and a narrower task
range. The currently running and pending jobs remain execution-in-progress and
provide no scientific evidence yet.

Source commit, model, runtime manifest, instrument, checkpoints, registered
seeds, tasks, conditions, metrics, and scoring remain unchanged. Meta-test
remains blocked and was not accessed.

### Failed-shard diagnosis and immutable replacements — 2026-07-20 16:34 UTC

A second four-task shard, `cal-d0-t21-25`, terminated after 12.013h with the
same provider error as `cal-d0-t09-13` at 12.010h. Both attempts reached
Kaggle's 12-hour CPU execution limit. They remain preserved as failed attempts;
neither state entry was reset or overwritten.

The control plane was paused while Kaggle OAuth was renewed. The five already
submitted kernels continued remotely, and authenticated status checks confirmed
they were still running. No experiment process or artifact was altered.

The two failed ranges were split into four new immutable two-task jobs:

| Failed attempt | Replacement jobs |
|----------------|------------------|
| `cal-d0-t09-13` | `retry-d0-t09-11`, `retry-d0-t11-13` |
| `cal-d0-t21-25` | `retry-d0-t21-23`, `retry-d0-t23-25` |

Only the execution partition changed. Source commit `949871b3d2d6cbb8a68af41396639cfb3f79ee5e`,
model, runtime manifest `a6214355c1c6b9192d435e62f3add6bef5db8c3a6c1cf3a55cb2a9dbfc91182e`,
organ hash, instrument, checkpoints, registered DEV cells, metrics, thresholds,
baselines, and scoring remain unchanged. All replacement kernels are private,
internet-off, CPU-only jobs.

The canonical batch now contains 120 jobs. The regenerated dispatch plan
reported `all_assigned=true`, `ready_for_dispatch=true`, and no errors. The
fanout service resumed active/running with 13 succeeded, 2 preserved failed,
5 running, and 100 pending; all four replacements are pending with zero
attempts and will enter the provider when capacity opens. Meta-test remains
blocked and was not accessed.

### Partition-dependent C6 invalidation and corrected width-3 restart — 2026-07-22

Runtime evidence reached 29 successful and 6 provider-timeout four-task
attempts. Successful provider execution ranged from 8.616h to 11.764h; the
four-task width therefore has no deterministic margin under Kaggle's 12-hour
CPU limit.

A subsequent partition audit invalidated the stronger claim above that shard
width changes only execution packaging. In source commit `949871b…`,
`collect_calibration_shard` selected the C6 `state_swapped` donor from the next
task inside the current shard. Changing shard boundaries therefore changed the
donor state and the scored `state_addressing_delta`. One-task shards omitted C6
entirely, while endpoint aggregation requires all six C1–C6 conditions.
Consequently, every v5 shard remains valid infrastructure/runtime evidence but
the collection is **invalid for scientific aggregation**. No v5 metric or
scientific verdict may be derived from it.

After this was disclosed, the human decision was: **version fix, allow the five
already-active legacy kernels to finish, then perform a full width-3 DEV
rerun**. Commit `a8c98d638209a8425b14a0f853e9fc46ae7da581` makes donor
selection partition-independent by using the canonical next task within each
full frozen validation family. Width-1 collection now includes C6, and the same
target cell is invariant across shard selections. Focused verification:
`89 passed`; the partition-invariance subset independently passed `4 passed`.

Corrected immutable source:

- dataset: `abdellahkadem/oczy-source-a8c98d638209`;
- archive SHA-256: `40708cb9e195cba45302d64d11a9110cc4f91b74ce0325b5a4856908b1e941ca`;
- source worktree dirty entries: 0.

The v5 scheduler now drains only the 31 succeeded, 6 failed, and 5 running
attempts present at cutover; its original 120-job batch remains preserved as
`batch-pre-width3.json` and `batch-full.json`. No never-submitted width-4 job
will be launched.

The corrected v6 campaign contains exactly 150 three-task jobs and covers all
5 × 90 registered DEV cells exactly once. Every generated kernel is private,
internet-off, CPU-only, and bound to the unchanged runtime manifest, model,
organ, public instrument archive, and checkpoint archive. The width-3 canary
`cal3-d0-t00-03` is pending behind the five legacy kernels. Its durable
systemd chain will verify corrected source provenance, runtime identity,
CPU-only hardware, shard self-hash, exact record counts, and C1–C6 presence
before atomically promoting the remaining 149 jobs.

Durable corrected paths:

- campaign: `~/.local/state/oczy/remote-queue/campaigns/r20-int8-dev-calibration-v6/`;
- queue: `calibration-a8c98d6/`;
- canary service: `oczy-r20-v6-cal-width3-canary.service`;
- promotion service: `oczy-r20-v6-cal-width3-promote.service`;
- fanout service: `oczy-r20-v6-cal-width3.service`.

Meta-test remains blocked and was not accessed. No thresholds, metrics,
registered tasks, seeds, checkpoints, model, or scoring formulas changed.