Codeseys/composer-replication-framework

1# Vision Validation: Does the Framework Encapsulate the Original Brief?2 3> **## Status as of 2026-06 (current through ADR-014)**4> The framework is past-skeleton: 8 subpackages (`composer_replication/*`), 266 passing (canonical count + env-variance note in docs/V1_V8_COVERAGE.md)5> tests + 1 skip-marked (see [`docs/V1_V8_COVERAGE.md`](V1_V8_COVERAGE.md) for the6> canonical count), and operational end-to-end examples (`gsm8k_grpo`,7> `sdpo_with_real_traces_production`). The 3-channel loss, layered hint-generation,8> trace-ingestion, and DiLoCo have all shipped and been cross-family reviewed. The base9> RL objective is now a **selectable menu** (default Dr.GRPO; ADR-014) rather than10> hardcoded. **Channel 3 (trace-replay-DPO) is the framework's own additive research11> channel — not part of Cursor's Composer recipe** (Composer = channels 1 Dr.GRPO + 212> SDPO only; ADR-014).13> 14> **Two remaining honest gaps (NOT closed):** 15> 1. Docker/TorchForge substrate E2E is hardware-blocked — the test exists and skips16>    cleanly, but lacking a local multi-GPU rig the orchestrator layer is unrun.17> 2. The 8B LMA channel-ladder is **not fully runnable today**: only **A1 (GRPO-only)**18>    has a real Modal runner — [ADR-014](adrs/ADR-014-policy-optimization-objective-menu.md)19>    records that "the A1 run used `dr_grpo`" and that threading the `objective=` menu20>    through the rest of the ladder runners is an open follow-up. **A2 (SDPO) / A321>    (replay-DPO) / A4 (combined)** are scaffold + plan-builder only; running them on a22>    real 8B checkpoint additionally needs a real error-trace SDPO dataset, a replay-DPO23>    preference corpus, and an A100 entrypoint that don't exist yet. The real 8B run is24>    *additionally* user-budget-gated (the sole remaining acceptance-gate box in25>    [ADR-013](adrs/ADR-013-lma-integration-channel-ladder.md)).26 27> **Status:** Self-audit, 2026-05-25 (Wave 6).28> **Question:** Does what we've built reflect what was originally asked for, or did we drift?29> **Method:** Recover original brief verbatim → atomic-clause decomposition → traceability matrix → adversarial self-review → user-journey simulation → concrete pass/fail scorecard with gap-closing actions.30 31This document is **uncomfortable on purpose.** Unit tests answer "does the code work"; this answers "is the code doing what was asked." Those are different questions, and skipping the second is a common failure mode in research projects that drift between brief and ship.32 33## 1. The original vision, recovered verbatim34 35From the originating message in session `20260525_005800_723eccb8` (timestamp `1779696689.2033243`):36 37> *"can you dive into Composer 2.5 and understand what makes it so much better? I want to see if I can take that and combine it with **diloco (decoupled, open, any variant of diloco)** and monarch/torchforge/openenv/VeRL/TRL and make a framework that we can use to further RL training of models to take them to the next level. One of the ideas that I had that might be a parallel to this is to **use traces from an llm-application usage** then **replay the traces with different models** to see at each llm-step what the llm would do. by doing this we get distillation data from any number of models that could be used to train the target model further. can we reserach all of this and see how we could try to set this up as a framework **to take any model from huggingface and be able to further RL train it to get results to Composer 2.5 which is post-trained kimi-k2.5**"*38 39Atomic-clause decomposition (the unit each deliverable maps onto):40 41| Clause | Vision element | Verbatim phrasing |42|---|---|---|43| **V1** | Understand Composer 2.5 internals | *"dive into Composer 2.5 and understand what makes it so much better"* |44| **V2** | Integrate **DiLoCo** (any variant: decoupled, open, etc.) | *"combine it with **diloco (decoupled, open, any variant of diloco)**"* |45| **V3** | Integrate **Monarch / TorchForge / OpenEnv / VeRL / TRL** | *"and monarch/torchforge/openenv/VeRL/TRL"* |46| **V4** | Build it as a **framework** (not a one-off recipe) | *"make a framework that we can use to further RL training of models"* |47| **V5** | **Trace-replay from real llm-application usage** as the novel idea | *"use traces from an llm-application usage then replay the traces with different models to see at each llm-step"* |48| **V6** | N-teacher distillation from those traces | *"distillation data from any number of models that could be used to train the target model further"* |49| **V7** | Research the whole space rigorously | *"can we reserach all of this"* |50| **V8** | **Generalize to any HF model**, target Composer-2.5-quality outcomes | *"to take **any model from huggingface** and be able to further RL train it to get results to Composer 2.5"* |51 52## 2. Traceability matrix — what's where in the repo53 54Map each vision clause to its concrete deliverable. Citations are file paths in this repository.55 56| Vision | Status | Deliverable evidence | Honest assessment |57|---|---|---|---|58| **V1** Composer 2.5 internals | 🟢 Strong | `research/01-composer-2.5.md` (parallel-research dispatch), `docs/COMPOSER_RECIPE_MAPPING.md` (primary-source audit, every claim tagged `[BLOG-VERIFIED]` / `[INFERRED]` / `[EXTRAPOLATED]`) | Caught and patched the SDPO/OPSD discovery that the initial dispatch missed. Audit notice on the original research note. **Solid.** |59| **V2** DiLoCo integration | 🟡 **Deferred** | `research/02-diloco-family.md` (covered conceptually); `framework/composer-replication-framework.md` § "Distributed sync" (decision: defer until multi-cluster); `docs/INTEGRATION_ARCHITECTURE.md` (mentions DiLoCo as v0.2) | We decided DiLoCo is v0.2 work. **The decision is documented but it is a deviation from the original brief.** The brief said "combine it with diloco," not "consider diloco." See § 4.1. |60| **V3** Monarch / TorchForge / OpenEnv / VeRL / TRL | 🟢 Strong | `research/03-monarch-torchforge-openenv.md`, `research/04-verl-trl.md`, `docs/INTEGRATION_ARCHITECTURE.md` (extension-point matrix), `spikes/005-integrated-trainer-skeleton/` (TRL + VeRL working code) | TRL and VeRL paths are coded; OpenEnv is the env substrate; Monarch + TorchForge are correctly assessed (Forge is "development paused"). **Five out of five components addressed; two have working code, three are correctly characterized as patterns/reference.** |61| **V4** Framework, not one-off recipe | 🟡 **Skeleton, not framework** | `spikes/005-integrated-trainer-skeleton/` has component-modular code (`opsd_loss.py`, `teacher_replay.py`, `data_collator.py`, two trainer paths); `docs/INTEGRATION_ARCHITECTURE.md` documents the composition contract | What we have is a **trainer skeleton with verified composition**, not yet a productized framework with installable package, CLI, examples directory. See § 4.2. |62| **V5** Real llm-application traces | 🔴 **Substituted with synthetic** | `spikes/001-teacher-replay-cost/synthesize_trace.py` builds 50 hand-crafted SWE-bench-lite-shaped states; `spikes/002a-trace-collection-trl/README.md` plans the real-trace path but unrun | **The brief explicitly says "traces from an llm-application usage."** We validated the *replay mechanism* on synthetic states. Real traces from a real agentic application are not yet ingested. See § 4.3. |63| **V6** N-teacher distillation | 🟢 Strong | `spikes/001-teacher-replay-cost/` ($0.98/trace verified, 150 real OpenRouter calls, 0 errors); `spikes/005-integrated-trainer-skeleton/teacher_replay.py` (DPO-pair extractor, 7 unit tests); economic feasibility is the strongest empirical result so far | Verified. The novel claim's economic floor is established. **Strongest part of the work.** |64| **V7** Rigorous research | 🟢 Strong | 5 deep-dives by 5 LLM families (`research/01..05`), 16KB methodology paper (`publications/PAPER_v0.md`), recipe-mapping audit, integration architecture, 38/38 unit tests, full citation graph | Process discipline visible: blog audit caught primary-source omissions, DeepWiki audits verify framework extension surfaces, every claim is sourced. **Solid.** |65| **V8** Any HF model → Composer-quality | 🔴 **Architecturally yes, empirically untested** | `spikes/005-integrated-trainer-skeleton/` plans `Qwen3-7B` (v0.0) → `Qwen3-32B` (v0.1); but the only smoke test is on a 10K-parameter custom `TinyLM`. No real HF model has been touched yet. | **Massive gap between architecture and evidence.** The framework targets `AutoModel.from_pretrained(...)` but has never loaded one. See § 4.4. |66 67## 3. Honest scorecard68 69Ten concrete pass/fail tests covering both "do we encapsulate the vision" and "is what we have actually correct":70 71| # | Test | Pass/Fail | Evidence | Gap-closer (if fail) |72|---|---|---|---|---|73| 1 | Original brief is recoverable verbatim and clause-decomposed | ✅ | § 1 of this doc | — |74| 2 | Each of {Composer, DiLoCo, Monarch, Forge, OpenEnv, VeRL, TRL, trace-replay, HF-base} has a documented deliverable | ✅ | § 2 traceability | — |75| 3 | The Composer 2.5 mechanism (SDPO/OPSD link) is correctly identified | ✅ | `docs/COMPOSER_RECIPE_MAPPING.md` § 2.1; primary-source audited | — |76| 4 | The novel TR-DPO channel is empirically feasible (not just paper) | ✅ | spike 001: 150 real calls, $0.98/trace, 0 errors | — |77| 5 | All three reward channels compose and don't fight each other | ✅ | spike 005 `test_loss_composition_smoke.py`: 5-step train decreases loss | — |78| 6 | DiLoCo is integrated *somewhere* in the runnable stack | ❌ | Conceptually documented in `research/02`; **no code** | Spike 008 (proposed § 6) — Streaming DiLoCo outer loop on a stub trainer |79| 7 | A real HuggingFace model can load + run a single forward pass through `ComposerReplicationTrainer` | ❌ | TinyLM only; never `AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.5B")` | Spike 006 (proposed § 6) — real-HF-model smoke test |80| 8 | At least one trace from real LLM-application usage is ingested end-to-end | ❌ | Synthetic 50-state fixture only | Spike 007 (proposed § 6) — real-trace ingestion from Cline / OpenHands / Claude Code session export |81| 9 | The framework is *installable* (a user can `pip install` and have working entrypoints) | ❌ | No `pyproject.toml`, no installable package | Wave 7 — packaging |82| 10 | A non-author can complete the "I have Qwen3-7B, I want a Composer-style variant" journey by reading docs | ⚠️ partial | Possible to read your way through `INTEGRATION_ARCHITECTURE.md` + `composer_trainer.py`, but no end-to-end runnable example | Spike 006 + a `examples/qwen3_7b_quickstart.md` |83 84**Score: 5/10 pass, 4/10 fail, 1/10 partial.** The framework's design is solid; the gap is between design and runnable artifact.85 86> **Update 2026-05-26 — Wave 7+8+9+10 closeout (deep work loop) + cross-model audit**87>88> Initial self-claim was 5/10 → 9/10. A cross-model adversarial review (Phase 11 of the deep work loop, doc at `docs/research/WAVE_7_10_FINAL_REVIEW.md`) found three of those ✅s were letter-of-the-law rather than spirit. **Honest re-scoring: 5/10 → 7/10 ✅, 1/10 ⚠️, 2/10 ❌-spirit.**89>90> | # | Test | Status | New evidence + honest caveat |91> |---|---|---|---|92> | 6 | DiLoCo integrated in runnable stack | ⚠️ partial | Spike 008 has `composer_replication.diloco.make_diloco_outer_loop` wrapping `torchft.local_sgd.DiLoCo`, with a 5/5 test suite that pins the sign convention. **But**: the BACKLOG required a *2-replica convergence* smoke; what shipped is a 1-replica machinery test with a passthrough no-op `allreduce`. The recon doc's "ready-to-paste 2-replica pattern" hits a single-process post-hook sequencing bug we couldn't fix without rewriting torchft. The DiLoCo wrapper is also **not yet integrated with `ComposerReplicationTrainer`** — it's an independent context manager. Calling V2 ✅ overstates: real DiLoCo training is GPU-multi-process which we haven't touched. |93> | 7 | Real HF model loads + runs through `compose_loss` | ✅ (with caveat below) | Spike 006 — Qwen2.5-0.5B-Instruct on CPU, 5 backward steps, loss 0.7390 → 0.0031, 9/9 tests. **Caveat**: SDPO channel is `0.0` throughout (silently disabled by ctx_student vs ctx_teacher shape mismatch — correct fallback, but means SDPO is not exercised end-to-end on a real model anywhere in the repo yet). DPO uses dummy reference logprobs. The 5-step loss decrease on a fixed batch is closer to "memorization works" than "the 3-channel composition is correct." Still: the framework now demonstrably loads real HF models, which it didn't before. V8 is closed in the literal sense. |94> | 8 | Real LLM-application trace ingested end-to-end | ❌ spirit | Spike 007 — `ClaudeCodeIngester` ingests real Claude Code session JSONL → `TraceState` records, 15/15 tests including a real-session smoke. **But**: BACKLOG acceptance criterion #3 said "end-to-end smoke: real trace → ingester → collator → 1-step `compose_loss`." That last hop is **not tested**. The spike stops at "ingester emits TraceStates correctly." Closing V5 in spirit needs a 50-LOC test that pipes ingested records all the way through the loss. Open. |95> | 9 | Framework is *installable* with working entrypoints | ✅ | Wave 10 — `pyproject.toml` ships `composer_replication` package, `pip install -e .` works, `examples/qwen_05b_quickstart/run.py` runs end-to-end via the package API. (Caveat acknowledged: `compose_loss` is documented as a verification harness, not production. The production loss is `ComposerReplicationTrainer._compute_loss`.) |96> | 10 | Non-author can complete the "I have X, I want a Composer variant" journey | ❌ spirit | Quickstart works for "verify the loss composition runs" but not for "train a real model" — that requires real GRPO rollouts, real teacher calls, and GPU. The brief's intended user wants the latter. We have not closed that path. |97>98> **The remaining 1/10 + 2/10 spirit gaps + the unverified 9/10 ⚠️** are the post-replication GPU phase: Spike 002a/b (real trace collection on GPU), Spike 003 (DPO-pair signal density), Spike 004 (A/B SWE-bench-lite), and a real-multi-process DiLoCo test. Those are GPU-budget-gated and out of scope for the deep work loop's CPU-only constraint.99>100> **Time spent on Wave 7-10**: ~1 session. **No GPU spend.** Modal evaluated but rejected for the smoke phase (ADR-001 — local 5090 wins on iteration cycle 10× over Modal L4 for 0.5B verification work). **The local 5090 was also not used** — Spike 002a-mini (the planned local-GPU smoke) was not run. The framework as of this commit has zero GPU evidence of any kind. That is honest about where this work lands: **a tested, installable methodology repo with real CPU smokes and primary-source-validated research, not a trained model.**101>102> **Update 2026-05-26 (later) — Wave 12 closeout, post-cross-model-review fixes**103>104> Cross-model review's priority items 3, 4, 5, 9 addressed; V1-V8 brief now105> tracks at **6/8 closed, 2/8 partial**. Coverage matrix:106> [`docs/V1_V8_COVERAGE.md`](V1_V8_COVERAGE.md), substrate-by-substrate107> coverage: [`docs/V3_SUBSTRATE_COVERAGE.md`](V3_SUBSTRATE_COVERAGE.md).108>109> | Item | Closed by |110> |---|---|111> | #3 SDPO never exercised on real model + tautology critique | **Spike 006-strict** (`spikes/006/tests/test_strict.py`) — 3 tests on real Qwen2.5-0.5B-Instruct: alternating-batch loss decrease, SDPO channel actually fires (sdpo_jsd > 0), SDPO off-vs-on total differs. **All 3 pass on CPU.** This was the single largest evidence gap from the review — **closed in spirit**, not just letter. |112> | #4 Zero GPU evidence | **Spike 002a-mini-gpu-smoke** (`spikes/002a-mini-gpu-smoke/run_gpu_smoke.py`) — 50 steps on RTX 5090 sm_120 in bf16. Loss 0.7354 → 0.00034 (99.95% reduction). Peak VRAM 5.31 GB. Median 480 ms/step. ADR-001's "use local 5090" claim now empirically verified. |113> | #5 run.log vs verdict.md numerical inconsistency | `torch.manual_seed(42)` + `random.seed(42)` pinned in both `spikes/006/run_smoke.py` and `examples/qwen_05b_quickstart/run.py`. Loss curves now reproducible. |114> | #9 V5 ingester→loss e2e test missing | **Spike 007 e2e** (`spikes/007/tests/test_e2e_with_loss.py`) — 2 tests pipe ingested `TraceState` records all the way through to `compose_loss` + backward. Synthetic fixture (3 states) + real Claude Code session (3 sampled states from a 628-line trace). **Both pass.** Closes V5 in spirit. |115>116> **Honest re-scoring after Wave 12**: 5/10 → **8/10 ✅** + 1/10 ⚠️ (Spike 008 multi-replica) + 1/10 ❌ (test 10 "non-author can complete journey for any HF model — only verified on 0.5B; the 7B+ path is GPU-budget gated"). Better than the 7/10 post-Wave-11 honest re-rating, by 1 point because tests 7, 8, and the SDPO-firing aspect of test 7 all materially improved.117>118> **Total tests passing**: 77 (38 Spike 005 + 9 Spike 006 + 3 Spike 006-strict + 15 Spike 007 + 2 Spike 007 e2e + 5 Spike 008 + 5 quickstart-via-package). **Plus** 1 GPU smoke on real hardware.119>120> **Items deferred to GPU/post-replication phase**: cross-model review items 6 (Claude Code circularity in code), 7 (compose_loss naming — addressed via package docstring rather than rename to keep API stable), 8 (dual sources of truth — same reason: spike copies are verification harnesses by design), 10 (sign-convention docstring — already addressed in Wave 11).121 122## 4. The four real gaps, each examined123 124### 4.1 V2: DiLoCo deferral — is this a drift?125 126**The drift:** the brief says *"combine it with diloco."* The framework documents say DiLoCo is v0.2 work, deferred until training spans multiple clusters.127 128**The defense:** Streaming DiLoCo's outer loop is only useful when training cannot fit on one cluster. For a Qwen3-7B (v0.0) or Qwen3-32B (v0.1) run on a single 8×H100 node, FSDP2 is sufficient — adding a DiLoCo outer loop would be bolt-on infrastructure with no measurable benefit. PRIME-RL (which we recommend as the substrate) has DiLoCo-shape sync between geographically distributed inference workers, but the trainer itself is single-cluster FSDP2. INTELLECT-2 (Prime Intellect's 32B QwQ run) is the only production-scale precedent for trainer-side DiLoCo, and even there the headline contribution was the orchestrator/trainer/inference split, not the gradient sync.129 130**The honest read:** the deferral is technically defensible, but it is **a deviation from the brief** and we should not pretend otherwise. The user explicitly said "any variant of diloco" — meaning the brief permits weak forms (e.g., outer-loop sync between geographically distributed inference workers, even with a single trainer cluster). We could add that *now*, on the existing v0.0 architecture, without waiting for v0.2.131 132**Concrete gap-closer:** **Spike 008** — implement Streaming DiLoCo outer-loop sync between two simulated "clusters" (could literally be two FSDP groups on the same machine for the smoke test). Validates that the outer-loop integrates with PRIME-RL's vLLM↔FSDP2 weight broadcast (SHARDCAST) without breaking the GRPO + SDPO + TR-DPO loss composition. Estimated effort: ~2 days, no new GPU budget if we use a tiny model. Closes V2.133 134### 4.2 V4: framework vs. skeleton135 136**The drift:** the brief says *"make a framework."* We have a "trainer skeleton" with component-modular code and 38 unit tests but no installable package, no CLI, no examples directory, no quickstart that resolves to a working training run.137 138**The honest read:** spike 005 is genuinely modular (`opsd_loss.py`, `teacher_replay.py`, `data_collator.py`, `composer_trainer.py`, `composer_adv.py` are independent components composing through clean interfaces). But "framework" carries connotations of installability, examples, documentation site, versioned releases. We have the *components* of a framework; we have not assembled them into the *artifact* a third party would call a framework.139 140**Concrete gap-closer:** **Wave 7 — packaging.** Add a top-level `pyproject.toml` with `composer-replication-framework` as a package; expose `from composer_replication_framework import ComposerReplicationTrainer, ComposerDataCollator, generalized_jsd_loss`; ship `examples/qwen3_7b_swe_bench_lite/` with a runnable `train.py`. Estimated effort: ~half a day once spike 006 (real-model smoke) lands. Closes V4 properly.141 142### 4.3 V5: synthetic states vs. real llm-application traces143 144**The drift:** the brief is unambiguous: *"use traces from an llm-application usage."* Spike 001 used **50 hand-crafted SWE-bench-lite-shaped states**, not real traces from a real agentic coding application.145 146**The defense:** the goal of spike 001 was to measure the *economic floor* of N-teacher replay. Synthetic states with realistic shape and token-count distributions are sufficient for that purpose — we get unbiased latency and cost numbers. The shape of real traces (multi-turn, ~250-500 tokens of context per state, tool-call decision points) was matched.147 148**The honest read:** spike 001's economic verdict generalizes to real traces *if* their shape is similar. But the brief's intent is bigger than "measure cost" — the brief envisions ingesting real traces (e.g., from Cursor session exports, OpenHands traces, Claude Code transcripts, Cline rollouts) and harvesting them for training data. **We have the *replay mechanism* but no *ingestion pipeline*.** Real traces have warts our synthetic ones don't: malformed tool calls, mid-rollout context truncation, vendor-specific schema, PII to scrub.149 150**Concrete gap-closer:** **Spike 007 — real-trace ingestion.** Pick one real source (proposal: Claude Code session JSONL exports, since I have access to my own and they're well-structured), write an adapter that converts to the `TraceExample` schema the data collator expects, run it through spike 005's pipeline. Validates the real → synthetic → trainer path works without contortion. Estimated effort: ~1 day, no GPU. Closes V5 substantively.151 152### 4.4 V8: HF model generalization — architecture vs. evidence153 154**The drift:** the brief targets *"any model from huggingface"* with Composer-2.5-quality outcomes. The architecture is designed for `AutoModelForCausalLM.from_pretrained(...)`, but the only smoke test is on a 10K-parameter custom `TinyLM`.155 156**The defense:** the integration claim ("all three channels compose, ablate, and train without divergence") is *generic*. It holds for any sufficiently-differentiable model. The TinyLM is a stand-in for any HF model. Real-HF-model testing is GPU-bound work that's properly in the spike 002+ tier.157 158**The honest read:** "the architecture is generic" is theoretically true and practically dangerous. Real HF models have:159- Tokenizer chat templates that the data collator must respect (`StubTokenizer` in spike 005 fakes `apply_chat_template`).160- Real vocab sizes (Qwen3 = 152K vs TinyLM's 64) where top-k restrictions in the SDPO loss matter.161- FlashAttention-2 attention paths the OPSD reference relies on.162- vLLM rollout integration for the outer GRPO loop.163 164Any of these can break the "tested in the small" claim when scaled up. **We should run a single forward pass and a single loss computation on a real (small, but real) HF model before claiming the framework generalizes.**165 166**Concrete gap-closer:** **Spike 006 — real-HF-model smoke.** Load `Qwen/Qwen3-0.5B` (or `Qwen/Qwen2.5-0.5B-Instruct` if 3 isn't out yet), wire up a real `AutoTokenizer` (not `StubTokenizer`) to the data collator, run a single forward pass + a single backward pass through `composer_total_loss` with all three channels active. CPU-only is fine; the test is wiring correctness, not training. Estimated effort: ~half a day, no GPU rental. **Closes V8's evidence gap without requiring spike 002–004's GPU budget.**167 168## 5. User-journey walkthrough — find the breaks169 170Simulate the brief's intended user: *"I have Qwen3-7B. I want a Composer-style variant. Walk me through it."*171 172The journey, as it would actually run today:173 174| Step | What the user does | What happens | Break? |175|---|---|---|---|176| 1 | Lands on the HF repo | Reads the README | ✅ Clear status, links to publications |177| 2 | Reads `publications/PAPER_v0.md` | Understands the architecture | ✅ Comprehensive |178| 3 | Reads `docs/INTEGRATION_ARCHITECTURE.md` | Picks TRL path | ✅ Clear extension-point matrix |179| 4 | Clones the repo, navigates to `spikes/005-integrated-trainer-skeleton/` | Reads the skeleton README | ✅ |180| 5 | Tries to install dependencies | **No `pyproject.toml`. No `requirements.txt` at the spike level.** | ❌ Break — has to figure out deps from imports |181| 6 | Tries to run an example | **No `examples/` directory.** Skeleton tests pass but there's no end-to-end "load Qwen3-7B + train one step" script | ❌ Break — has to assemble it themselves from components |182| 7 | Tries `from trl_path.composer_trainer import ComposerReplicationTrainer` | Works iff TRL is installed | ⚠️ Latent: needs TRL ≥ some version, undocumented |183| 8 | Wires up an `AutoModel` and `AutoTokenizer` | **Untested — `data_collator.py` falls back to a stub tokenizer code path; real chat templates may not be exercised** | ⚠️ Latent risk |184| 9 | Tries to source training data | **No instructions for trace collection (spike 002 unrun); synthetic stub fixture is in spike 001 but not labeled as a starter dataset** | ❌ Break |185| 10 | Realizes they need teacher API credentials | `OPENROUTER_API_KEY` envvar — documented in `teacher_replay.py` docstring but not in a top-level setup guide | ⚠️ Findable but suboptimal |186| 11 | Wants to run the spike-004 A/B comparison | **No script. No config template. Spike 004 README is planning notes, not runnable code** | ❌ Break — they'd have to write the experiment harness themselves |187 188**Verdict:** the architecture is reachable from docs, but a third-party can't currently complete the journey end-to-end without significant assembly. The framework needs **packaging + examples + a quickstart** to credibly claim "any HF model" generalization.189 190## 6. Proposed gap-closing spikes (no GPU budget required)191 192These three sub-projects close the four real gaps identified in § 4 and don't need GPU rental — they're CPU-only or use $5 of API. They can run *before* spike 002–004 to make the framework actually deliver on the brief.193 194### Spike 006 — Real-HF-model smoke (closes V8)195 196- **Goal:** load `Qwen/Qwen2.5-0.5B-Instruct` via `AutoModelForCausalLM` + `AutoTokenizer`, wire to `ComposerDataCollator` with real `apply_chat_template`, run one forward + one backward through `composer_total_loss(α=0.1, β=0.05)`, verify finite gradient on every parameter.197- **Hardware:** CPU sufficient (model fits in ~1GB RAM).198- **Effort:** ~half a day.199- **Pass criterion:** test in `tests/test_real_hf_model_smoke.py` passes; loss is finite and decreases over 5 steps on a fixed batch.200 201### Spike 007 — Real-trace ingestion (closes V5)202 203- **Goal:** Write `adapters/claude_code.py` (or `cline.py` or `openhands.py` — pick one) that converts a real session export into a list of `TraceExample` dicts. Run spike 001's `replay_trace` on 5 real states. Run spike 005's pipeline end-to-end on the resulting batch.204- **Hardware:** CPU.205- **Effort:** ~1 day.206- **Pass criterion:** `python adapters/claude_code.py < session.jsonl > traces.jsonl` produces collator-compatible output; `composer_total_loss` runs on it without error; one DPO pair successfully extracted from teacher disagreement.207 208### Spike 008 — Streaming DiLoCo smoke (closes V2)209 210- **Goal:** Bolt a Streaming DiLoCo outer loop onto the `composer_total_loss` smoke test. Use two FSDP process groups on the same node as a stand-in for two clusters. Verify pseudo-gradient sync every H steps doesn't break loss composition.211- **Hardware:** CPU sufficient (TinyLM scale).212- **Effort:** ~2 days (DiLoCo's PyTorch reference is a ~200 LOC outer loop).213- **Pass criterion:** 5-step training run with α=0.1, β=0.05, DiLoCo H=2 still decreases loss; pseudo-gradient sync produces no NaN.214 215After 006 + 007 + 008, the scorecard goes from 5/10 pass to 8/10 pass. Wave 7 (packaging) closes #9 and #10. **Then the framework genuinely delivers on the brief, before any of the GPU-bound spikes 002–004 run.**216 217## 7. Adversarial self-review218 219The five strongest objections to "this framework encapsulates the vision," steelmanned and answered:220 221### Objection 1: "You spent more time on publication materials than on closing real gaps."222 223**Steelman:** Wave 5 produced 1,200 lines of publication materials. Wave 6 (this doc) is more publication. Meanwhile spike 006/007/008 — actual integration work — is unwritten. Optimizing for paper-readiness over framework-readiness is a known failure mode.224 225**Answer:** Conceded with caveat. Wave 5 was at the user's explicit request. This wave (vision validation) was the first introspective check; gap-closers are now scoped and ready to execute. The next wave should be 006 (real-model smoke) before any further publication work. **Reordering accepted.**226 227### Objection 2: "Spike 001's $0.98/trace doesn't generalize to real traces. You measured what's cheap (50 short hand-crafted states), not what's real (10K-token rollouts with embedded code blobs)."228 229**Steelman:** Real Cursor / Cline / OpenHands rollouts can have 10–100K tokens of context. At Opus pricing ($15/Mtok input), a 50-step replay over 50K-token rollouts costs $37.50 in input tokens alone *per teacher per trace*. With 3 teachers that's ~$112 per trace, dwarfing the synthetic-state estimate.230 231**Answer:** Largely valid. Spike 001's verdict is *valid* but its *generalization* to real traces is unproven. Spike 007 (real-trace ingestion) is the right next experiment. Once it lands, we can rerun spike 001's analysis on real-shaped traces and report an updated cost number. The framework's economic claim should be updated to include both the synthetic-floor result *and* a real-trace measurement when available.232 233### Objection 3: "VeRL is recommended for v0.2 but the only smoke test is in TRL. The VeRL `composer_adv.py` has zero unit tests."234 235**Steelman:** `verl_path/composer_adv.py` ships as untested code. The DeepWiki audit confirmed extension surfaces but didn't validate that the resulting `compute_grpo_composer_advantage` function correctly composes with VeRL's `compute_advantage` dispatcher. Anyone choosing the VeRL path is running unverified code.236 237**Answer:** Correct. The VeRL path is a *design verified by primary-source audit*, not *a tested implementation*. Closing this requires installing VeRL + Ray + a real model — non-trivial. Reasonable interim mitigation: explicitly mark `verl_path/` as `STATUS: design-only` in its README and warn users the TRL path is the only tested one. Long-term gap-closer: spike 002b's PRIME-RL/VeRL run is the natural place to validate.238 239### Objection 4: "The 'integration with Monarch and TorchForge' claim is paper-thin. Forge is paused. Monarch K8s is documented but not used. Where's the runnable Monarch ActorMesh?"240 241**Steelman:** The integration matrix in `INTEGRATION_ARCHITECTURE.md` mentions Monarch ActorMesh patterns for SDPO and TR-DPO, but no code. TorchForge is "paused" so we route around it. The actual integration with Meta's stack is documentation, not code. This is V3 partial, not V3 strong.242 243**Answer:** Half right. Forge being paused upstream is genuinely orthogonal to our work — we can't depend on a paused project. But the Monarch integration *is* paper-only. The honest framing: we *integrate with the design philosophy* of Monarch (single-controller actor-mesh orchestration) by ensuring the components are *placeable* on a Monarch mesh, but we have no runnable Monarch code. Should soften the V3 claim in the README from "integrate with all five" to "integrate with TRL + VeRL + OpenEnv (coded); align with Monarch + Forge philosophy (documented)."244 245### Objection 5: "You claim 'any HF model from huggingface' but the architecture is implicitly designed around a chat-template-having causal LM. What about base models without chat templates? Encoder-decoder models? VLMs?"246 247**Steelman:** `data_collator.py::_tokenize_messages` calls `apply_chat_template` and falls back to plain text concat. For a base model without a chat template (e.g. `gpt2`, `Qwen3-0.5B-Base`), the fallback path may not produce coherent input. For encoder-decoder models the whole "single forward, gather logits" assumption breaks. VLMs add another dimension. The "any HF model" claim is overstated.248 249**Answer:** Accurate. The framework is designed for **causal LM with chat templates**, which is the standard target for agentic-coding RL post-training. The brief's "any model from huggingface" should be re-scoped to **"any HuggingFace causal LM with a chat template (Qwen, Llama, Mistral, DeepSeek, Phi, Gemma families)."** README + paper should say this explicitly. Encoder-decoder, base-no-chat-template, and VLM support is out of scope for v0.0/v0.1. Adding it would be a separate research direction.250 251## 8. What this validation actually says252 253The framework **partially encapsulates** the vision. Strengths:254 255- Composer 2.5 mechanism is correctly identified and grounded in published prior art (V1 ✅)256- Five-component agentic-RL stack is mapped with primary-source-audited extension points (V3 ✅)257- N-teacher distillation channel is empirically feasible (V6 ✅)258- Research process is rigorous, sourced, and self-correcting (V7 ✅)259- Composition smoke testing on a tiny model is a real empirical claim (parts of V4, V8 ✅)260 261Real gaps the brief asked for that we punted on:262 263- DiLoCo deferral is documented but is a deviation (V2 🟡)264- "Framework" is a skeleton; no installable package or examples (V4 🟡)265- Real llm-application traces are not ingested anywhere (V5 🔴)266- "Any HF model" is architecturally generic but never tested on a real HF model (V8 🔴)267 268These gaps are closable without GPU rental via three CPU-only spikes (006, 007, 008) totaling ~3.5 days of effort. After they land + a packaging wave (Wave 7), the scorecard goes from 5/10 to 9/10 and the framework genuinely delivers on the original brief.269 270The gap that remains unclosed at 9/10 is #4 ("the framework actually trains better models than baseline GRPO") which requires the GPU spikes 002–004. That's correctly out-of-scope for vision encapsulation — it's *empirical validation of the methodology*, not encapsulation of the brief.271 272## 9. Recommended next moves, ordered273 274In recommended-do-next order:275 2761. **Spike 006 (real-HF-model smoke)** — half a day, CPU-only. Closes V8's biggest credibility gap and surfaces any tokenizer / chat-template / vocab-size issues hiding in the skeleton. *Highest value per hour.*2772. **Spike 007 (real-trace ingestion)** — 1 day, CPU + ~$5 OpenRouter. Closes V5's real-vs-synthetic gap. Picks one source (Claude Code session JSONL? Cline transcripts?) and writes the adapter.2783. **Soften over-claims in README and paper** — half-hour. README: "TRL coded, VeRL design-only." Paper: "any causal LM with a chat template." `verl_path/README.md`: add `STATUS: design-only — validate via spike 002b before production use`.2794. **Wave 7 (packaging)** — half a day, after 006 + 007 land. `pyproject.toml` + `examples/qwen_05b_quickstart/` directory + entry-point exposure.2805. **Spike 008 (Streaming DiLoCo smoke)** — 2 days, CPU-only. Closes V2's biggest deviation from the brief. Lowest priority because the deferral is technically defensible, but worth doing for completeness.281 282Items 3 + 4 are documentation/packaging chores. Items 1 + 2 + 5 are real engineering. None require GPU budget. Total: ~5 days of sequential effort to take the framework from 5/10 to 9/10 vision encapsulation.283 284After that, GPU-bound spikes 002–004 are what move 9/10 → empirical validation of the methodology itself, which is a separate project phase (the "v0.1 follow-up paper" framing in the publication wave).285 286## 10. How to keep validating going forward287 288This document captures one snapshot. Vision encapsulation drifts as projects evolve. Three mechanisms to keep checking:289 290- **Re-audit at every release wave.** Each commit message currently includes a wave number. Wave 7 (packaging), wave 8 (post-spike-006), etc. should each end with a one-paragraph "vision-check delta" — what gaps closed, what new ones opened, whether the README's claims still match the code.291- **External review via the HF Discussions tab.** The pre-experimental release posts (drafted in `publications/`) explicitly ask for critical reads of the integration architecture and adjacent-work pointers. Specifically pin one Discussion thread for "vision encapsulation feedback" — invite people to point out where the framework deviates from its stated goals.292- **Calibration check post-spike-004.** When the GPU spikes finally run and produce a result (positive or negative), revisit this document and update the scorecard. If TR-DPO doesn't beat plain GRPO, V6 needs to be downgraded. If it does, V8 should incorporate the empirical evidence. The validation is a living artifact, not a one-shot audit.293 294---295 296*Self-audit complete. Honest read: 5/10 today, 9/10 reachable in ~5 days of CPU-only work, last 1/10 is the empirical question that gates the v0.1 paper.*297