Codeseys/composer-replication-framework

1# Real-trace SDPO alignment validation2 3Runs the full **ingestion → adapter → collator → SDPO** data path against your4own local Claude Code session logs (`~/.claude/projects/**/*.jsonl`) and reports5the live SDPO mask alignment ratio. This is the population-level proof that6Wave 21's `_build_chat_aligned_mask` fix holds on real-world data, not just the7synthetic fixture.8 9## Run10 11```bash12python examples/validate_real_trace_alignment/run.py13# options:14#   --projects-dir ~/.claude/projects   where to discover sessions15#   --max-sessions 8                    how many error-bearing sessions to sample16#   --model Qwen/Qwen2.5-0.5B-Instruct  a real chat-template tokenizer17#   --pass-threshold 0.95               min alignment ratio to PASS18#   --strip-thinking                    (default OFF — see below)19```20 21Exit code: `0` PASS (alignment ≥ threshold, no crashes), `1` FAIL, `2` no22error-bearing sessions found / no chat template.23 24## What it measures25 26- **ingestion yield** — states emitted, error sites detected27- **structural vs string-only flagging** — the Wave 21 `is_error` fix. The28  ingester sets a structural `tool_error: True` boolean; `string-tag-only`29  should be ~0 (the brittle `[TOOL_RESULT (ERROR)]` grep is fallback-only).30- **empty-recovery rate** — see below.31- **SDPO alignment** — fraction of in-loss `sdpo_loss_mask` positions where32  student token id == teacher token id. ~100% means the mask lands exactly on33  content tokens; <95% means chat-template drift has regressed.34 35## The `--strip-thinking` gotcha (important for SDPO)36 37`ClaudeCodeIngester(strip_thinking=...)` controls whether `[THINKING]` blocks38survive. For most ingestion you strip them. **For SDPO hint-distillation you39must NOT** — on real Claude Code traces the error-*recovery* turn is very often40**pure thinking** (the model reasons about the failure, then silently retries a41tool). Strip it and that turn's content goes empty, so ~67% of error sites carry42no recovery content to distill against and produce a zero-signal SDPO row.43 44This script therefore defaults to `strip_thinking=False`. The collator also45guards against the empty case (an empty-recovery error turn is treated as a46non-error site rather than firing an all-`ignore_index` mask), but the *signal*47only exists if you keep the thinking. Pass `--strip-thinking` to see the48empty-recovery warning fire.49 50## Representative result (Codeseys' machine, 2026-05-28)51 52```53sessions processed:       10/1054total error sites:        14155structural-flagged users: 17056string-tag-only users:    057empty-recovery sites:     0/141 (0%)     # strip_thinking=False58SDPO alignment (REAL):    832/832 = 100.0%59RESULT: PASS ✅60```61 62With `--strip-thinking` the same sessions report ~67% empty-recovery and the63measurable in-loss positions collapse accordingly — the lever is visible.64