Research campaign report · 2026-08-08/09
Recursive self-improvement on karpathy/autoresearch
One rented H100, 5-minute training budget per experiment, only train.py editable. Claude (Fable) selected every experiment from a structured log with noise-floor gating, literature mining, self-critique, and calibration tracking.
0.9718 ± 0.0008
final val_bpb, mean of 3 seeds (from 0.99555 baseline)
0.9773 → beaten
karpathy's README curve, passed at experiment 7 of ours vs 83 of his
28 runs · ~$25
13 keeps, 12 discards, 1 crash, 2 replicates · ~5.5 h H100
0.9697 + 0.0021
community frontier (#43, 126 exps) — not reached
Progress vs the reference curves
This campaign (running best)
Karpathy README (83 exps)
#43 frontier 0.9697
×discards
The final recipe (13 kept changes)
| # | Change | val_bpb | Δ | Source |
|---|
| 0 | Baseline (stock train.py, H100 SXM5) | 0.99555 | — | — |
| 1 | Total batch 524K → 262K, warmdown 0.5 → 0.7 | 0.98591 | −0.0096 | README replay |
| 2 | Depth 9 (dim 512) · SSSSL windows · short = ⅛ ctx | 0.98183 | −0.0041 | README replay |
| 3 | x0 init 0.05 · unembedding LR 0.008 · embedding LR 0.8 | 0.98083 | −0.0010 | README replay |
| 4 | RoPE base 200 000 (later proven load-bearing: +0.0013 if reverted) | 0.98069 | −0.0001 | README replay |
| 5 | AdamW weight decay — lm_head 0.01 · wte 0.001 · VE 0.002 | 0.97813 | −0.0026 | Literature (#43) |
| 6 | Matrix init scale 0.7× | 0.97702 | −0.0011 | Literature (#43) |
| 7 | Attention softmax scale 0.12 | 0.97602 | −0.0010 | Literature (modded) |
| 8 | Delete 5% warmup (simplicity win — a replayed keep was harmful) | 0.97576 | −0.0003 | Log-analyst critique |
| 9 | Muon momentum cooldown 0.95 → 0.90 in warmdown | 0.97512 | −0.0006 | Near-miss retry, decomposed |
| 10 | Warmdown 0.7 → 0.8 | 0.97475 | −0.0004 | Hot-category push |
| 11 | Width dim 512 → 640 (81M params, MFU 36 → 39%) | 0.97338 | −0.0014 | VRAM-headroom probe |
| 12 | Compile max-autotune (untimed) · CUDA sync every 8 steps | 0.97183 | −0.0015 | Throughput analysis |
What the machinery contributed
- Literature mining found the two community sessions on this exact benchmark and delivered three keeps (weight decay, init scale, softmax scale) plus a known-negative list that saved ≥6 wasted slots.
- Noise-floor measurement (seed replicates, σ ≈ 0.0001 early) made 0.0003-level deltas resolvable — several keeps were below what session #43 could have trusted. Re-measured at the end: σ grew ~10× on the final config, so the campaign correctly switched to a 3-seed mean for the reported number.
- Log-analyst self-critique (a fresh-context agent reading the experiment log) caught a harmful replayed keep (warmup), stale LR tunings, and the never-touched-knob list; its re-ranked queue produced 3 keeps.
- Near-miss retry with decomposition turned a failed bundle (momentum schedule, +0.0005) into a win (cooldown alone, −0.0006).
- Calibration tracking measured literature results transferring at 0.2–0.4× claimed magnitude and my own predictions running ~0.001 optimistic — both fed back into queue ranking.
- Pre-flight guardrails (grown from crash post-mortems) kept the GPU-slot waste to one 35-second crash across 28 runs.
Honest accounting
- Karpathy's curve was beaten on the same hardware class, same budget, same rules — but with the enormous advantage of hindsight: his 15 kept changes and two community sessions were public. The genuinely novel contributions of this campaign are the warmup deletion, the momentum-cooldown decomposition, the width-640 optimum, the untimed-compile arbitrage, and the sync-reduction — roughly −0.006 of the −0.024 total.
- The #43 frontier (0.9697) was not reached. The remaining gap (0.0021 ≈ 2.6σ of the final config) would need several more real wins; the queue's best remaining candidates are Muon
ns_steps 3, fp8 lm_head, dataloader overlap, and the softcap × softmax joint probe.
- Cross-platform comparability caveat: our baseline ran 0.0024 below karpathy's reported baseline on identical code, so absolute cross-machine comparisons carry that offset; all our deltas are internally consistent on one instance.