Codeseys/composer-replication-framework

1# ADR-006 — RL framework strategy: TRL + VeRL + PRIME-RL2 3**Status**: Accepted4**Date**: 2026-05-265**Wave**: 136**Amended-by**: ADR-008 — the implication that any of the three recipes can host the full 3-channel loss is amended: the **SDPO channel requires full vocabulary logits and is TRL-hosted only**. PRIME-RL hosts channels 1+3 (PG + trace-replay-DPO); its `LossInputs` exposes log-probs, not logits, so `recipes/prime_rl/composer_loss.py` raises `NotImplementedError` for `alpha_sdpo>0` until upstream PRIME-RL exposes logits. The framework selection, Monarch decision, and three-recipe matrix below are preserved.7 8## Context9 10The brief's V3 clause names six substrates: **monarch, torchforge,11openenv, VeRL, TRL** (plus DiLoCo). Cross-model review (Wave 11) flagged12that V3 was thin on the RL-framework side: TRL has working code, VeRL has13a config skeleton, and Monarch/TorchForge/OpenEnv are research-only.14 15User's 2026-05-26 expansion: *"see if there are other frameworks that are16more popular that we could try to use. meta's pytorch agentic stack17components are something that I'd like to explore."*18 19`docs/research/RL_FRAMEWORKS_LANDSCAPE.md` audited:20- 6 RL frameworks: OpenRLHF, PRIME-RL, NeMo-Aligner, Unsloth, LLaMA-Factory,21  DeepSpeed-Chat22- 4 Meta PyTorch stack components: Monarch, TorchTitan, TorchForge, torchchat23 24## Options considered25 26| Framework | License | GRPO/DAPO? | Custom-loss extension | Verdict |27|---|---|---|---|---|28| OpenRLHF | Apache-2 | ✅ DAPO | Fork `openrlhf/models/loss.py` + Trainer subclass (~400-600 LOC) | Strong but heavyweight |29| **PRIME-RL** | **Apache-2** | **✅ GRPO + DAPO** | **First-class `CustomLossConfig` with `LossInputs` struct (~200-300 LOC)** | **Chosen** |30| NeMo-Aligner | Apache-2 | ❌ no GRPO/DAPO | n/a | Reject |31| Unsloth | Apache-2 | TRL patcher | Closed `unsloth_zoo` loss kernels — unhookable | Reject |32| LLaMA-Factory | Apache-2 | ❌ delegates to EasyR1 | n/a | Reject |33| DeepSpeed-Chat | Apache-2 | ❌ PPO+DPO only | feature-stale since 2023 | Reject |34 35| Meta stack | License | Active? | Role |36|---|---|---|---|37| **Monarch** | **BSD-3** | **✅ v0.4.1 stable, v0.5 dev** | **Actor mesh — coordination layer for any SPMD trainer** |38| TorchTitan | BSD-3 | ✅ active | Distributed-training stack (already a transitive dep of PRIME-RL) |39| TorchForge | BSD-3 | ❌ paused | Patterns only, per repo banner |40| torchchat | BSD-3 | active | Inference only — out of scope |41 42## Decision43 44**Add PRIME-RL as the third RL framework after TRL+VeRL, and Monarch as the45agentic-stack coordination layer.**46 47### Why PRIME-RL48 49PRIME-RL ships a **first-class `CustomLossConfig` with an `import_path`**50that lets us drop in a Python function returning a tensor. The config51exposes a `LossInputs` struct with exactly the tensors we need:52`trainer_logprobs`, `inference_logprobs`, `teacher_logprobs`,53`advantages`, `loss_mask`. This is **the cleanest possible extension54point for a 3-channel loss** — no fork, no Trainer subclass, no monkey-55patching.56 57It also uses the `verifiers` env protocol (OpenEnv-compatible by design),58so it slots into the framework's existing data path without translation.59 60PRIME-RL was used to train INTELLECT-1 (10B base, 30 nodes) and INTELLECT-261(32B QwQ); production-tested on real distributed runs.62 63### Why Monarch (not TorchForge or TorchTitan as a top-level)64 65- **Monarch is what's actually shipping** from Meta's agentic stack. v0.4.166  is stable, v0.5 dev daily. BSD-3.67- **TorchForge is paused** per its own repo banner. We document it68  (research/03) but don't depend on it.69- **TorchTitan is a transitive dep** of PRIME-RL already, so we get its70  benefits without needing to build a direct integration. If we wanted a71  TorchTitan-only path, it would be redundant with PRIME-RL.72- **torchchat is inference-only** and doesn't fit the training-framework73  conversation.74 75Monarch's role in our stack: **the actor mesh that hosts trainer/generator/76rewarder/judge actors**. PRIME-RL's three-actor split (trainer, generator,77rewarder) maps naturally onto Monarch primitives.78 79## Consequences80 81### Accepted82 83- `composer_replication/recipes/prime_rl/` directory:84  - `prime_rl_recipe.md` — integration recipe (parallel to TRL Recipe A,85    VeRL Recipe B)86  - `composer_loss.py` — the 3-channel loss adapted to PRIME-RL's87    `LossInputs` struct (~200-300 LOC)88  - `prime_rl_config.yaml` — example PRIME-RL config wiring our loss in89- `composer_replication/recipes/monarch/` directory:90  - `monarch_actor_layout.md` — design doc for the actor mesh91  - `actors.py` — placeholder Monarch actor definitions (skeleton only;92    full integration is post-replication)93- New optional dependencies in `pyproject.toml`:94  - `[prime-rl]` extra: `prime-rl>=0.5`95  - `[monarch]` extra: `monarch>=0.4.1`96- `docs/V3_SUBSTRATE_COVERAGE.md` updated to reflect the new additions.97 98### Three-recipe production matrix99 100| User scenario | Recommended recipe |101|---|---|102| Quick start, single-cluster, ≤7B | TRL Recipe A |103| Production multi-node, ≤32B | VeRL Recipe B |104| Decentralized / DiLoCo-shape, any size | PRIME-RL recipe (NEW) — channels 1+3 only; **SDPO channel TRL-hosted, see ADR-008** |105| Coordination-heavy multi-actor RL | Monarch + any of the above |106 107### Trade-offs explicitly accepted108 109- **Three RL frameworks is a maintenance burden.** We accept this because110  no single one covers all the user scenarios above. The framework's111  contribution is the 3-channel loss + the trace-replay channel, expressed112  in three different framework idioms. Each recipe is ~200-300 LOC; total113  triplication tax ~700 LOC vs. picking one framework.114- **Monarch is BSD-3 not MIT.** The framework is MIT; users opting in to115  Monarch take on its license. Documented in pyproject.toml's optional116  extras.117- **PRIME-RL's API may evolve.** The `LossInputs` struct is currently the118  contract; if PRIME-RL stabilizes a different shape we'd need to bump.119  Pin to v0.5.x in our optional extras.120 121## Source122 123`docs/research/RL_FRAMEWORKS_LANDSCAPE.md` (2026-05-26 subagent recon,124primary-sourced from DeepWiki audits + GitHub repo READMEs + PyPI release125metadata).126