Xnhyacinth/codegen

HarnessMix Codegen RL Data

This repository contains the current HarnessMix no-Docker code-generation training and held-out evaluation data prepared from a local Harness-RL workspace.

The rows are Slime/Polar JSONL examples. Each row has a chat-style prompt, an empty label, and metadata for benchmark provenance, source/test assets, evaluator command, and contamination controls.

Files

• —data/train.jsonl: 3992 training rows.
• —data/train_manifest.jsonl: audit manifest for the training rows.
• —data/eval.jsonl: 559 held-out evaluation rows, combining EvalPlus, OJBench, and the current LiveCodeBench smoke row.
• —data/eval_manifest.jsonl: audit manifest for all evaluation rows.
• —data/eval_evalplus_ojbench.jsonl: 558 EvalPlus + OJBench held-out rows.
• —data/eval_livecodebench_smoke.jsonl: 1 LiveCodeBench smoke row.
• —assets/: source and test files referenced by the JSONL metadata.
• —raw/ojbench_testdata/: selected OJBench raw prompt/testdata files needed by the current OJBench subset.
• —metadata/: generation and filtering reports.

Training Split

All training rows have metadata.split = "train", metadata.scenario = "code_generation", and metadata.contamination.heldout = false.

This training mix was produced by the HarnessMix quality/diversity filter. It excludes EvalPlus, LiveCodeBench, OJBench, raw APPS/TACO/deepmind CodeContests, and other explicitly held-out evaluation sources.

Held-Out Evaluation Split

All evaluation rows have metadata.split = "eval", metadata.contamination.heldout = true, and metadata.contamination.exclude_from_training = true.

Schema

Each JSONL row has this shape:

{
  "prompt": [{"role": "user", "content": "..."}],
  "label": "",
  "metadata": {
    "benchmark": "...",
    "split": "train|eval",
    "scenario": "code_generation",
    "example_id": "...",
    "dataset_id": "...",
    "source_path": "assets/.../solution.py",
    "test_path": "assets/.../test_solution.py",
    "prepare_actions": [{"type": "upload_file", "source": "assets/...", "target": "/polar/session/workspace/..."}],
    "evaluator_command": "python3 test_solution.py",
    "expected_output_json": {"...": "PASSED"},
    "contamination": {
      "heldout": false,
      "exclude_from_training": false,
      "do_not_mix_with_eval_benchmarks": ["EvalPlus", "LiveCodeBench", "OJBench"]
    }
  }
}

Paths in this repository are relative to the dataset repository root. If you run the rows from a different working directory, either run from the downloaded dataset root or rewrite metadata.source_path, metadata.test_path, and metadata.prepare_actions[].source accordingly.

Evaluation Runtime Notes

EvalPlus and LiveCodeBench rows use local Python test files under assets/ and python3 test_solution.py as the evaluator command.

OJBench rows call the official ojbench/DMOJ judge from their generated test_solution.py. To execute OJBench rows, install the OJBench package and its runtime dependencies (DMOJ, pypy3, and a C++17 compiler such as g++). The selected testdata is included under raw/ojbench_testdata/.

Training-Time Eval Example

In Harness-RL, the no-Docker launcher can pass held-out data to Slime eval with name/path pairs:

PROMPT_DATA=data/train.jsonl PROMPT_MANIFEST=data/train_manifest.jsonl EVAL_PROMPT_DATA="heldout data/eval.jsonl" EVAL_INTERVAL=5 N_SAMPLES_PER_EVAL_PROMPT=1 EVAL_MAX_RESPONSE_LEN=4096 bash examples/harnessmix/no_docker_smoke/run_train.sh

ProRL-Agent-Server / Polar Slime Usage

The ProRL codegen branch should track code, configs, and documentation only. It should not duplicate this dataset in Git. From a ProRL-Agent-Server checkout, download the dataset into the codegen example directory before running smoke tests or GRPO training:

export HF_TOKEN=<your-huggingface-token>  # only needed if this dataset is private
python - <<'PY_INNER'
import os
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Xnhyacinth/codegen",
    repo_type="dataset",
    local_dir="examples/codegen_slime_grpo",
    token=os.environ.get("HF_TOKEN"),
    allow_patterns=["data/**", "assets/**", "metadata/**", "raw/**"],
)
PY_INNER

After download, the ProRL codegen example expects these local paths:

• —examples/codegen_slime_grpo/data/train.jsonl
• —examples/codegen_slime_grpo/data/eval.jsonl
• —examples/codegen_slime_grpo/assets/
• —examples/codegen_slime_grpo/metadata/
• —examples/codegen_slime_grpo/raw/ojbench_testdata/ for the current OJBench subset

Minimal ProRL smoke command:

RUN_ID=codegen-smoke PROMPT_DATA=examples/codegen_slime_grpo/data/train_smoke.jsonl REF_LOAD=/path/to/Qwen3.5-4B_torch_dist WANDB_MODE=offline CUDA_VISIBLE_DEVICES=0,1,2,3 bash examples/codegen_slime_grpo/run.sh

For full training, use PROMPT_DATA=data/train.jsonl and EVAL_PROMPT_DATA=data/eval.jsonl through the ProRL codegen launcher. Keep W&B logs, checkpoints, Ray temp files, and downloaded dataset files outside Git.

Provenance and Licensing

This dataset is a processed mix of multiple upstream public datasets and benchmarks. Users are responsible for complying with each upstream dataset's license and usage terms. This repository does not claim a single unified license over upstream content.

The included data is intended for code-generation RL training/evaluation research and contamination-controlled held-out measurement.