Codeseys/composer-replication-framework
0
1# Integration Architecture: 3-Channel Reward Composition Across the Agentic-RL Stack2 3> **Status:** Architecture spec — verified against framework source code via DeepWiki on 2026-05-25.4> **Companion doc:** [`docs/COMPOSER_RECIPE_MAPPING.md`](COMPOSER_RECIPE_MAPPING.md) defines the three reward channels (RLVR / Composer-SDPO / N-Teacher-Replay). This document specifies *where each one hooks into each framework* — the actual function names, decorator surfaces, and DataProto fields you'd touch. Working code skeleton at [`spikes/005-integrated-trainer-skeleton/`](../spikes/005-integrated-trainer-skeleton/).5 6## TL;DR — the unified loss7 8For any framework choice, the v0.1 trainer computes:9 10```11total_loss = grpo_loss12 + α * sdpo_kl_loss (Composer hint-distill, channel 2)13 + β * trace_replay_loss (N-teacher novel channel, channel 3)14```15 16Where:17- **`grpo_loss`** = standard GRPO+DAPO over RLVR scalar rewards (channel 1, the substrate).18- **`sdpo_kl_loss`** = `generalized_jsd_loss(student_logits, teacher_logits, labels=…, beta=0.5, …)` — single-model self-distillation, where `teacher_logits` come from a forward pass on the student model with a hint inserted into the context. **Lifted verbatim from `siyan-zhao/OPSD::generalized_jsd_loss`** (verified self-contained static method, MIT licensed).19- **`trace_replay_loss`** = DPO-style preference loss (or PRM-style score regression) over `(chosen, rejected)` pairs derived from N external teacher disagreements at each step.20 21The novel architectural claim is that **all three channels can run simultaneously** in a single trainer step, with the cost split as: (1) one extra forward pass per error site for SDPO, (2) N teacher API calls per replayed step for trace-replay. Spike 001 verified the API economics (✅ $0.98/trace, 5× headroom).22 23## Stack-by-stack integration matrix24 25| Component | TRL | VeRL | TorchForge | Monarch | OpenEnv |26|---|---|---|---|---|---|27| **Channel 1 (RLVR/GRPO)** | `GRPOTrainer._compute_loss(model, inputs)` — base class behavior, no change | `core_algos.compute_grpo_outcome_advantage` (registered via `@register_adv_est("grpo")`) | `forge.controller.GRPO` recipe (paused; pattern reference only) | Orchestrates rollout/trainer/rewarder ActorMeshes | Env exposes RLVR-shaped reward via `step()` |28| **Channel 2 (SDPO hint-distill)** | **Subclass override** of `_compute_loss`; lift `generalized_jsd_loss` from OPSD | **New advantage estimator** registered as `@register_adv_est("grpo_sdpo")`; reads `data.batch["sdpo_teacher_logprobs"]`; OR keep adv_estimator=grpo and add SDPO term in critic worker's compute_loss | Add a new ActorMesh `SDPOTeacherActor` that re-runs forward with hint-conditioned context; wire into trainer's loss | No-op at orchestration layer (just routes hint pairs) | Env emits "error site" markers in tool response so trainer knows where to insert hints |29| **Channel 3 (N-teacher trace-replay)** | **Subclass override** of `_compute_loss`; add DPO-pair term using teacher logprobs in `inputs["teacher_action_distributions"]` | **Custom adv_estimator**; teacher distributions stashed in `data.non_tensor_batch["teacher_actions"]`; precedent: distillation already attaches `teacher_log_probs` to rollout DataProto | Add a new `TeacherReplayActor` ActorMesh that holds OpenRouter client; called on a delayed-reward channel (RFC-004) | Routes teacher queries via `service.spawn(TeacherReplayActor, n=K)` for K parallel teacher pools | Env's `state()` API exposes step-level state needed for teacher replay |30| **Multi-turn rollout async** | ❌ **Blocking** — tool-call stalls GPU | ✅ `AsyncServer` + `AgentLoop` async; tool-call doesn't block GPU | ✅ Generator ActorMesh async via vLLM; tool-call waits don't block trainer | ✅ ActorMesh + supervision tree; native async | Env supports async via WebSocket multiplexed sessions |31| **Weight sync (vLLM ↔ FSDP)** | Co-located vLLM (no resharding) | ✅ **3D-HybridEngine** (resharding between FSDP↔TP) — most efficient | TorchStore RDMA weight broadcast | Monarch RDMA data plane | N/A (env-side) |32| **Scale ceiling** | ~32 GPUs / 70B FSDP | ✅ 671B+ proven, Megatron-LM | Reference patterns only (paused) | Thousands of GPUs (mesh) | 10K+ concurrent env sessions |33 34**Reading the matrix:** rows are "what each reward channel touches in each framework." Columns are framework choices. The matrix shows the v0.1 framework choice is non-trivial:35- **TRL** = simplest extension story (one subclass override) but doesn't async-decouple tool calls and caps at ~70B.36- **VeRL** = most flexible at scale (custom `adv_estimator` + DataProto extension is well-trodden) and has async agent loop, but Ray-heavy and steeper curve.37- **TorchForge + Monarch** = cleanest abstraction but Forge is "development paused" — use as reference, not foundation.38- **OpenEnv** = orthogonal substrate — works with all of the above; not a choice, a default.39 40## Architecture diagrams (mechanism-level, all three channels)41 42### 1. Composer SDPO hint-distill flow (single model, hint-conditioned self-teacher)43 44```45 ┌─────────────────────┐46 │ Hint Generator │47 │ - templates v0.1 │48 │ - LLM-driven v0.2 │49 └──────────┬──────────┘50 │ generates hint text51 ▼ at error sites52 Trace, mid-rollout: ┌────────────────┐53 …turn_4 (OK) │ Build paired │54 turn_5 (ERROR: tool not found) ────│ contexts: │55 …turn_6 (OK) │ ctx_student │56 │ ctx_teacher │57 │ (= ctx_student│58 │ + hint at │59 │ turn_5) │60 └───────┬────────┘61 │62 ┌─────────────────┴──────────────────┐63 │ │64 ▼ ▼65 ┌──────────────────┐ ┌────────────────────┐66 │ Student forward │ │ Teacher forward │67 │ on ctx_student │ │ (SAME MODEL on │68 │ → student_logits│ │ ctx_teacher) │69 │ │ │ → teacher_logits │70 └──────────┬───────┘ └────────┬───────────┘71 │ │72 └──────────┬────────────────────┘73 │ feed both into74 ▼75 ┌─────────────────────────────────────────┐76 │ generalized_jsd_loss( │77 │ student_logits=…, │78 │ teacher_logits=…, │79 │ labels=… (mask non-error turns), │80 │ beta=0.5, # JSD │81 │ temperature=1.0, │82 │ token_clip=…) │83 │ │84 │ → sdpo_kl_loss (a scalar) │85 └──────────────┬──────────────────────────┘86 │87 ▼88 add to total_loss with α weight89```90 91**Key implementation note:** Per the DeepWiki audit, OPSD's `SelfDistillationDataCollator` builds two prompts per example:92- `ctx_student` = problem only (or problem + rollout up to error turn).93- `ctx_teacher` = problem + privileged info (in OPSD's case, the verified solution; in our case, the hint).94 95For Composer-style hint-distill, we adapt this: `ctx_teacher = ctx_student + injected_hint` at the specific turn boundary, with `labels` masked to keep loss only at the post-hint tokens of that turn.96 97### 2. N-Teacher trace-replay flow (N external teachers, novel)98 99```100 Trace, frozen post-rollout:101 turn_1 (state_1, action_1_student, reward=…)102 turn_2 (state_2, action_2_student, reward=…)103 …104 turn_50 (state_50, action_50_student, reward=…)105 │106 │ for each turn t in trace:107 ▼108 ┌───────────────────────────┐109 │ teacher pool (frozen) │110 │ ┌──────────────────────┐ │111 │ │ Opus 4.7 (anthro) │ │112 │ │ GPT-5 (openai) │ │113 │ │ DeepSeek V4 Pro │ │114 │ └──────────────────────┘ │115 │ parallel API calls │116 └───────────┬───────────────┘117 │ teacher_t = [a_t^Opus, a_t^GPT, a_t^DS]118 ▼119 ┌───────────────────────────────────┐120 │ disagreement scorer: │121 │ if 2+ teachers agree on X │122 │ and student picked Y ≠ X: │123 │ chosen=X, rejected=Y │124 │ (DPO pair) │125 │ else if all 3 disagree: │126 │ skip (no signal) │127 │ else if all agree with student: │128 │ skip (no signal) │129 └──────────────┬────────────────────┘130 │ DPO pairs[]131 ▼132 ┌───────────────────────────────────┐133 │ DPO loss term: │134 │ L = -log σ(β·(logπ(chosen|s) │135 │ − logπ_ref(chosen|s) │136 │ − logπ(rejected|s) │137 │ + logπ_ref(rejected|s)))│138 │ │139 │ → trace_replay_loss (a scalar) │140 └──────────────┬────────────────────┘141 │142 ▼143 add to total_loss with β weight144```145 146**Key implementation note:** unlike SDPO, this happens **post-rollout**, not during. The trace is frozen, teacher calls are batched, DPO pairs are extracted offline, and the loss is computed in a follow-up training step. This decouples teacher-API-call latency from the trainer's GPU loop entirely. Spike 001 verified ~20s p95 step latency for parallel 3-teacher calls — acceptable at offline-batch cadence.147 148### 3. The combined trainer step (all three channels)149 150```151 ┌──────────────────────────────────────────────────────────┐152 │ ROLLOUT PHASE (per episode) │153 │ Generator (vLLM) → Env (OpenEnv) → trace JSONL │154 │ → emits (state_t, action_t, reward_t, error_marker_t) │155 └────────────────────────┬─────────────────────────────────┘156 │157 ┌──────────────────┼──────────────────────────┐158 │ │ │159 ┌─────────▼─────────┐ ┌──────▼─────────┐ ┌───────────▼─────────┐160 │ RLVR scoring │ │ Hint detection │ │ Teacher replay │161 │ (test pass etc.) │ │ at error_marker│ │ (post-rollout, async│162 │ │ │ → hint_text │ │ via OpenRouter API)│163 │ → reward_outcome │ │ → ctx_teacher │ │ → teacher_actions[] │164 └─────────┬─────────┘ └──────┬─────────┘ └───────────┬─────────┘165 │ │ │166 │ │ ┌─────────────┘167 │ │ │ disagreement→DPO pairs168 │ │ │169 └──────────────────┼────────────┘170 ▼171 ┌──────────────────────────────────────────────────────────┐172 │ TRAINING PHASE (per gradient step) │173 │ │174 │ forward(student, ctx_rollout) → student_logits │175 │ forward(student, ctx_teacher) → teacher_logits ← SDPO │176 │ │177 │ grpo_loss = compute_grpo_loss(reward_outcome) │178 │ sdpo_kl_loss = generalized_jsd_loss(s_logits, │179 │ t_logits, labels=error_mask) │180 │ trace_replay_loss= dpo_loss(student_logprobs, │181 │ ref_logprobs, dpo_pairs) │182 │ │183 │ total_loss = grpo_loss + α*sdpo_kl_loss + β*replay_loss │184 │ │185 │ total_loss.backward() │186 │ optimizer.step() │187 └──────────────────────────────────────────────────────────┘188```189 190**Cost composition per training step (v0.0/v0.1 estimate):**191 192| Operation | Cost |193|---|---|194| Rollout forward (vLLM, async) | k tokens × inference TFLOPs |195| Teacher forward (training-mode FSDP, hint-conditioned) | ~1 extra FW pass per error site (sparse — maybe 5% of tokens) |196| RLVR reward eval | ~test execution overhead, env-bound, async |197| Teacher API replay (post-rollout, batched) | ~$0.02/step × parallel 3-teacher = ~$1/trace at 50 steps (verified by spike 001) |198| GRPO + SDPO + DPO loss compute | Negligible vs forward passes |199| Backward + optimizer step | Standard FSDP step |200 201The SDPO channel is **forward-pass-bound** (one extra FW per error site). The trace-replay channel is **API-call-bound** (offline, post-rollout, ~$0.30/trace with VOI gating in v0.1). They don't compete for the same resource.202 203## Per-framework integration recipes204 205### Recipe A: TRL `GRPOTrainer` subclass (recommended for v0.0/v0.1)206 207**Why this is the right v0.1 choice:** simplest extension; OPSD code lifts cleanly; Qwen3-7B fits comfortably in TRL's scale ceiling; first-class OpenEnv integration via `environment_factory`.208 209```python210from trl import GRPOTrainer211from opsd_trainer import generalized_jsd_loss # lifted from siyan-zhao/OPSD212 213 214class ComposerReplicationTrainer(GRPOTrainer):215 """v0.1 trainer: GRPO + SDPO hint-distill + N-teacher trace-replay-DPO."""216 217 def __init__(self, *args, alpha_sdpo=0.1, beta_replay=0.05, **kwargs):218 super().__init__(*args, **kwargs)219 self.alpha_sdpo = alpha_sdpo220 self.beta_replay = beta_replay221 222 def _compute_loss(self, model, inputs):223 # Channel 1: standard GRPO loss224 grpo_loss = super()._compute_loss(model, inputs)225 226 # Channel 2: SDPO hint-distill at error sites227 sdpo_kl = self._compute_sdpo_loss(model, inputs)228 229 # Channel 3: trace-replay DPO from teacher disagreement230 replay_dpo = self._compute_trace_replay_loss(model, inputs)231 232 # Compose233 total_loss = grpo_loss + self.alpha_sdpo * sdpo_kl + self.beta_replay * replay_dpo234 235 # Log all three components for ablation236 if self.state.global_step % self.args.logging_steps == 0:237 self.log({238 "loss/grpo": grpo_loss.detach().item(),239 "loss/sdpo_kl": sdpo_kl.detach().item(),240 "loss/trace_replay_dpo": replay_dpo.detach().item(),241 "loss/total": total_loss.detach().item(),242 })243 244 return total_loss245 246 def _compute_sdpo_loss(self, model, inputs):247 if "ctx_teacher_input_ids" not in inputs or inputs["ctx_teacher_input_ids"].numel() == 0:248 # No error sites in this batch — SDPO is a no-op.249 return torch.tensor(0.0, device=model.device)250 251 student_logits = model(input_ids=inputs["input_ids"]).logits252 with torch.no_grad():253 # Teacher = same model, hint-injected context. NO grad.254 teacher_logits = model(input_ids=inputs["ctx_teacher_input_ids"]).logits255 256 return generalized_jsd_loss(257 student_logits=student_logits,258 teacher_logits=teacher_logits,259 labels=inputs["sdpo_loss_mask"], # only error-turn tokens260 beta=0.5,261 temperature=1.0,262 token_clip=10.0,263 )264 265 def _compute_trace_replay_loss(self, model, inputs):266 if "dpo_chosen_input_ids" not in inputs:267 return torch.tensor(0.0, device=model.device)268 269 # Standard DPO loss using teacher-disagreement-derived pairs270 chosen_logprobs = self._get_logprobs(model, inputs["dpo_chosen_input_ids"])271 rejected_logprobs = self._get_logprobs(model, inputs["dpo_rejected_input_ids"])272 ref_chosen_logprobs = inputs["dpo_chosen_ref_logprobs"] # precomputed273 ref_rejected_logprobs = inputs["dpo_rejected_ref_logprobs"]274 275 beta_dpo = 0.1276 logits = beta_dpo * (chosen_logprobs - ref_chosen_logprobs277 - rejected_logprobs + ref_rejected_logprobs)278 return -F.logsigmoid(logits).mean()279```280 281The data collator (a sibling to OPSD's `SelfDistillationDataCollator`) is responsible for assembling the extra fields:282- `ctx_teacher_input_ids` — the hint-augmented context, when error markers fire283- `sdpo_loss_mask` — which token positions are post-hint and should contribute to KL284- `dpo_chosen_input_ids` / `dpo_rejected_input_ids` — pairs from spike-003-style extraction285- `dpo_*_ref_logprobs` — precomputed under the reference (student-init) policy286 287**OpenEnv plumbing** stays untouched — the `environment_factory=…` kwarg of `GRPOTrainer` already handles the SWE-bench-lite env.288 289### Recipe B: VeRL custom `adv_estimator` + DataProto extension (recommended for v0.2 scale)290 291**Why this is the right v0.2 choice:** VeRL has the only proven 70B+/671B RL story; HybridFlow's 3D-HybridEngine is the production reference for FSDP↔vLLM resharding; VeRL has precedent for exactly this pattern (`teacher_log_probs` already used for distillation per the DeepWiki audit).292 293```python294# verl_extensions/composer_adv.py295from verl.trainer.ppo import core_algos296from verl.trainer.ppo.core_algos import register_adv_est297 298 299@register_adv_est("grpo_composer")300def compute_grpo_composer_advantage(token_level_rewards, eos_mask, index, **kwargs):301 """GRPO advantage with SDPO + N-teacher trace-replay shaping.302 303 Reads from kwargs (passed via DataProto.batch / non_tensor_batch):304 - sdpo_teacher_logprobs: per-token logprobs from hint-conditioned forward305 - teacher_actions: list of N teacher action distributions per step306 - alpha_sdpo, beta_replay: weights307 """308 # Standard GRPO advantage (same as built-in)309 base_adv = core_algos.compute_grpo_outcome_advantage(310 token_level_rewards, eos_mask, index311 )312 313 # SDPO shaping: at error-site tokens, add an extra advantage term314 # proportional to (teacher_logprob - student_logprob) — this nudges315 # the policy gradient toward the hint-conditioned distribution.316 sdpo_teacher_lp = kwargs.get("sdpo_teacher_logprobs")317 if sdpo_teacher_lp is not None:318 student_lp = kwargs["old_log_prob"]319 sdpo_term = kwargs["alpha_sdpo"] * (sdpo_teacher_lp - student_lp)320 # Only apply at error-mask positions321 sdpo_term = sdpo_term * kwargs["sdpo_error_mask"]322 base_adv = base_adv + sdpo_term323 324 # Trace-replay shaping: per-step PRM signal from teacher consensus325 teacher_actions = kwargs.get("teacher_actions")326 if teacher_actions is not None:327 prm_signal = compute_teacher_consensus_prm(teacher_actions, kwargs["student_actions"])328 base_adv = base_adv + kwargs["beta_replay"] * prm_signal329 330 return base_adv331```332 333In the run config:334 335```yaml336# ppo_trainer.yaml337algorithm:338 adv_estimator: grpo_composer339 alpha_sdpo: 0.1340 beta_replay: 0.05341```342 343In the rollout worker, attach the extra fields to `DataProto`:344 345```python346# verl_extensions/composer_rollout.py347def attach_composer_fields(data: DataProto, sdpo_teacher_lp, teacher_actions):348 data.batch["sdpo_teacher_logprobs"] = sdpo_teacher_lp349 data.batch["sdpo_error_mask"] = build_error_mask(...)350 data.non_tensor_batch["teacher_actions"] = teacher_actions351 return data352```353 354This pattern is **identical to how VeRL already handles distillation rollouts** (per the DeepWiki audit: *"teacher log-probabilities are stashed on the rollout output and later concatenated into the per-batch DataProto for the student training step"*).355 356### Recipe C: TorchForge + Monarch (reference patterns only, not a production target)357 358Forge is "development paused per the upstream banner; lift patterns, don't depend on it. The relevant patterns are:359 360- **`SDPOTeacherActor` ActorMesh** — runs the hint-conditioned forward pass on a separate compute group, returns logits via TorchStore RDMA back to the trainer. Useful when SDPO forward is expensive enough to warrant offload.361- **`TeacherReplayActor` ActorMesh** — pool of K parallel actors, each holding an OpenRouter HTTP client. Trainer calls `service.spawn(TeacherReplayActor).query(state, n=3)` and gets back N teacher distributions.362- **Delayed-reward channel (OpenEnv RFC-004)** — for teacher replay where the signal arrives post-rollout, not at `step()`. Map to a separate reward stream that the trainer subscribes to.363 364If/when Monarch's K8s story matures and we move to v0.2 multi-cluster decentralized scale, lift these patterns into the VeRL stack rather than building on Forge directly.365 366### Recipe D: OpenEnv (substrate, not a choice)367 368OpenEnv is **orthogonal** — it works with TRL, VeRL, TorchForge, and any custom trainer. The contract:369 370- Env exposes `reset(...)`, `step(action)`, `state()`, `close()`.371- Env optionally exposes tools via MCP (RFC-003).372- Env optionally emits delayed rewards (RFC-004).373- Container deploys via Docker; trainer connects via WebSocket multiplexed sessions.374 375For our framework, the env contract needs **two lightweight extensions** (both backward-compatible):376 3771. **Error-site markers in tool responses.** When a tool call fails (404, type error, runtime exception), the env's `step()` response includes `meta["error_kind"]` and `meta["hint_template_key"]` — pre-defined keys the trainer's hint generator dispatches on. This lets the trainer decide *where* in the trace to insert hints without re-running the env.3782. **State-replay endpoint.** For trace-replay, the env supports `state(t)` returning the exact same observation the agent saw at step `t` — needed so external teachers see identical context. This is purely additive; existing OpenEnv envs without this can fall back to "feed teacher the conversation history" mode.379 380We'll publish both extensions as proposed RFCs against `meta-pytorch/OpenEnv` once the v0.0 spike validates the full framework.381 382## Why all three channels can run simultaneously (the architectural argument)383 384These three channels do **not** compete for any shared resource:385 386| Resource | Channel 1 (RLVR) | Channel 2 (SDPO) | Channel 3 (replay) |387|---|---|---|---|388| GPU forward pass | rollout (vLLM, async) | extra FW per error (training, FSDP) | none — uses precomputed logprobs |389| GPU backward pass | yes | yes (added to total_loss) | yes (added to total_loss) |390| External API budget | none | none | $0.30–1/trace (verified, spike 001) |391| Latency-critical path | yes — gates next rollout | minor — extra FW <5% of tokens | no — async, post-rollout |392| Storage | rollout JSONL | extra ctx + mask in collator | DPO pairs JSONL (separate dataset repo) |393 394Furthermore the **gradients are additive** by design — the three loss terms each have their own α/β weights, so we can ablate any subset by setting the weight to 0. The v0.1 ablation matrix:395 396| Run | α (SDPO) | β (replay) | Tests |397|---|---|---|---|398| Baseline | 0 | 0 | pure GRPO+RLVR |399| +SDPO only | 0.1 | 0 | Composer recipe replication |400| +Replay only | 0 | 0.05 | the v0.0 novel claim, scaled to 32B |401| Full | 0.1 | 0.05 | combined channel test (v0.1 winner candidate) |402 403This 4-arm A/B at 32B is the v0.1 terminal experiment. Total cost ~$1200 (4 runs × 3 seeds × ~$100 each). Roadmap.404 405## Open questions / followups (for v0.1 design phase, not v0.0)406 4071. **Hint generator architecture (open since the recipe-mapping doc).** Templates first; LLM-driven generator if templates plateau on style/communication errors.4082. **SDPO weight `α` schedule.** OPSD paper used constant; SDPO paper uses constant; Cursor never says. Likely warmup-from-0 then constant; ablate.4093. **DPO pair extraction threshold.** Spike 003 will determine: do we want only "2-of-3 teachers agree" pairs (high signal, fewer pairs), or also "1-of-3 differs from student" (more pairs, noisier)?4104. **Teacher pool composition.** Spike 001 used Opus 4.7 + GPT-5 + DeepSeek V4 Pro. Question for v0.1: should we add a fourth teacher (Qwen3-Max-MoE? Kimi K2.5?) as a same-family voice to balance Anthropic/OpenAI? Cost adds linearly.4115. **Reward hacking monitoring.** Cursor mentioned (without specifics) "agentic monitoring tools." Our v0.1 environment needs sandbox hardening: disable `find`, `unzip`, bytecode tools, and Python type-cache reads, so the model can't reverse-engineer deleted features the way Composer 2.5's model did.412 413## Citations414 415Primary sources verified for this document:416 417- **TRL `GRPOTrainer._compute_loss`** — verified via DeepWiki query against `huggingface/trl` repo on 2026-05-25. `environment_factory` kwarg confirmed for OpenEnv plumbing.418- **VeRL `@register_adv_est` + `DataProto`** — verified via DeepWiki query against `volcengine/verl` repo on 2026-05-25. Distillation precedent (`teacher_log_probs` already attached to rollout DataProto) confirms the pattern.419- **OPSD `generalized_jsd_loss`** — verified via DeepWiki query against `siyan-zhao/OPSD` repo on 2026-05-25. Static method, self-contained, MIT licensed, FlashAttention-2 compatible. Function signature reproduced verbatim above.420- **Cursor blog** — [Introducing Composer 2.5](https://cursor.com/blog/composer-2-5), read directly via `tavily_extract` advanced mode. Footnote 1 cites the three self-distillation papers.421- **SDPO paper** — Hübotter et al., [arXiv:2601.20802](https://arxiv.org/abs/2601.20802), ICLR 2026 Scaling Post-training Workshop.422- **OPSD paper** — Zhao et al., [arXiv:2601.18734](https://arxiv.org/abs/2601.18734), code at [github.com/siyan-zhao/OPSD](https://github.com/siyan-zhao/OPSD) (MIT).423- **Existing research notes** — `research/03-monarch-torchforge-openenv.md` (Monarch/Forge/OpenEnv) and `research/04-verl-trl.md` (VeRL/TRL) for framework-level context. Audit notes on those files apply: trust extension-point claims here over framework-level claims there when in conflict.424 425This document is the bridge between the **conceptual** 3-channel composition (in `COMPOSER_RECIPE_MAPPING.md`) and the **executable** trainer skeleton (in `spikes/005-integrated-trainer-skeleton/`). Anyone implementing v0.1 starts here, then opens the skeleton.426