CooperBench/cooperdata-v3-midtrain-blend

CooperData v3 — Midtraining Blend (Qwen3.5-9B cooperative SWE agents)

All-token midtraining mixture that bridges Qwen/Qwen3.5-9B (instruct) toward the cooperative multi-agent SWE-coding SFT distribution. One document per row (text, tagged by source) — NOT packed — so trl.SFTTrainer(packing=False) tokenizes per-doc and the Gated-DeltaNet recurrence stays per-document. ~390M tokens.

Composition

General ~74% / bridge ~26% (coop-dominant). Agentic/social sources are rendered with the model's real chat template (so coop tool-calls become native <tool_call>/<tool_response>, matching the qwen3_coder serving parser).

Methodology

Grounded in "Midtraining Bridges Pretraining and Posttraining Distributions" (2510.14865): specialized data is introduced late (a finished instruct model), so the conservative-weight finding (high specialized weight late -> catastrophic forgetting) dictates a general-dominant mix. The bridge is coop-dominant — near-domain agentic-coding on a coding model (low forgetting risk) and the exact SFT target. Heavy agentic specialization is left to SFT.

Usage (all-token, packing=False)

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
ds = load_dataset("CooperBench/cooperdata-v3-midtrain-blend", split="train")
SFTTrainer(model="Qwen/Qwen3.5-9B", train_dataset=ds,
           args=SFTConfig(dataset_text_field="text", packing=False, max_length=8192))

Build

coopertrain/train/modal/datamix/ (sources + mixture + cleaning). Inherits the licenses of its constituent datasets (Dolmino, mini-coder-trajs, OdysSim) + our coop trajectories.