Datasetpublic

rootxhacker/patchbench

PatchBench A multi-language benchmark for evaluating whether an LLM can fix a bug correctly and without introducing new bugs. Each row presents a real issue report plus a large code context (~500–1500 lines); the model under test produces a patch; a per-row test suite then checks that (a) previously failing tests now pass and (b) the rest of the suite stays green. Status: v0 spec. Rows are being built. Row schema Field Type In tasks config Description… See the full description on the dataset page: https://huggingface.co/datasets/rootxhacker/patchbench.

sourceHugging Faceupdated 27d agoView on Hugging Face

0likes283downloads

Dataset Card

PatchBench

A multi-language benchmark for evaluating whether an LLM can fix a bug correctly and without introducing new bugs. Each row presents a real issue report plus a large code context (~500–1500 lines); the model under test produces a patch; a per-row test suite then checks that (a) previously failing tests now pass and (b) the rest of the suite stays green.

Status: v0 spec. Rows are being built.

Row schema

Field	Type	In `tasks` config	Description
`id`	string	✅	`patchbench-{lang}-{source}-{nnnn}`
`language`	string	✅	`python` \	`javascript` \	`typescript` \	`go` \	`java` \	`rust`
`source`	string	✅	`real` (mined bug-fix PR) \	`synthetic` (injected bug)
`origin`	string	✅	Source dataset, e.g. `swe-bench-live-multilang`, `swesmith-go`
`repo`	string	✅	GitHub repo the code came from
`base_commit`	string	✅	Commit the buggy state is taken at
`issue_text`	string	✅	Issue/PR report the model must diagnose from
`context_files`	list of `{path, content}`	✅	~500–1500 lines total incl. the buggy file(s), callers, and related modules
`buggy_files`	list of string	✅	Paths the fix is expected to touch
`test_files`	list of `{path, content}`	✅	Test patch materialized as files
`test_cmds`	list of string	✅	Commands that run the tests for this row (language-native)
`env_setup_cmds`	list of string	❌ (harness only)	Install/build commands needed before `test_cmds`
`fail_to_pass`	list of string	❌	Tests that MUST fail pre-fix and pass post-fix
`pass_to_pass`	list of string	❌	Tests that MUST stay passing post-fix (the regression check)
`gold_patch`	string	❌	Reference fix, for harness/self-eval only

Two configs prevent leakage:

—`tasks` (public eval surface): everything an agent sees — issue, context, tests.
—`gold` (harness-only): fail_to_pass, pass_to_pass, gold_patch, env_setup_cmds.

Verification contract (every released row)

1.Pre-fix: apply test_files to the repo at base_commit → every fail_to_pass test fails; every pass_to_pass test passes.
2.Post-fix: apply gold_patch → every fail_to_pass and pass_to_pass test passes.
3.Determinism: each check is run twice; any row with a flaky test is dropped.
4.No-leak check: issue text must not name the exact function/line to change (auto-screen + spot manual review).

A model's fix is scored: resolved iff all fail_to_pass pass AND all pass_to_pass still pass. Rows also record per-test results so partial behavior (fix works but broke something) is visible — that is the "introduced a new bug" signal.

Sourcing plan (target ~1,500–2,000 rows)

Language	Real source	Synthetic top-up	Target
Python	SWE-Gym + SWE-bench Verified/train	SWE-smith-py	~550
Rust	SWE-bench-Live rust + rustbench	SWE-smith-rs	~300
Java	SWE-bench-Live java + Multi-SWE-bench java	SWE-smith-java	~300
Go	SWE-bench-Live go + Multi-SWE-bench go	SWE-smith-go	~250
JS/TS	SWE-bench-Live js+ts + Multi-SWE-bench	SWE-smith-js/ts	~400

Context expansion is PatchBench's own contribution: no upstream dataset ships sliced 500–1500-line windows. We materialize the repo at base_commit, compute the failing-test dependency closure, and slice the modules inside it, filtering out rows whose closure exceeds the window.

Harness

Language-native runner (no Docker required, adapted from the SWE-bench grading loop + microsoft/RepoLaunch per-language build/test recipes): clone at base_commit → env_setup_cmds → apply test_files (+ candidate patch) → run test_cmds → parse per-test results → grade against F2P/P2P. Per-repo env caching makes verification tractable.

License audit

Upstream dataset licenses: SWE-bench-Live (MIT), SWE-Gym (MIT), SWE-smith (MIT). Multi-SWE-bench / PrimeIntellect rows are CC0/“other” (ByteDance community terms) — those rows are re-verified from original permissively-licensed repos before inclusion, or excluded.