rootxhacker/patchbench
PatchBench A multi-language benchmark for evaluating whether an LLM can fix a bug correctly and without introducing new bugs. Each row presents a real issue report plus a large code context (~500–1500 lines); the model under test produces a patch; a per-row test suite then checks that (a) previously failing tests now pass and (b) the rest of the suite stays green. Status: v0 spec. Rows are being built. Row schema Field Type In tasks config Description… See the full description on the dataset page: https://huggingface.co/datasets/rootxhacker/patchbench.
pilot rust: incremental 8 rows
pilot rust: incremental 7 rows
pilot rust: incremental 6 rows
pilot rust: incremental 5 rows
pilot rust: verified 4
pilot rust: incremental 4 rows
pilot rust: incremental 3 rows
pilot rust: incremental 2 rows
pilot rust: incremental 1 rows
harness v1.3: create repo/ symlink inside run_tests (after clean_state)
pilot rust: incremental 8 rows
pilot rust: incremental 7 rows
pilot rust: incremental 6 rows
pilot rust: incremental 5 rows
pilot rust: verified 4
pilot rust: incremental 4 rows
pilot rust: incremental 3 rows
pilot rust: incremental 2 rows
pilot rust: incremental 1 rows
harness v1.2: repo/ nested-layout symlink for rows whose test_cmds expect it; context-expansion skips missing test files instead of crashing (eza row 0070)
pilot rust: incremental 8 rows
pilot rust: incremental 7 rows
pilot rust: incremental 6 rows
pilot rust: incremental 5 rows
pilot rust: incremental 4 rows
pilot rust: incremental 3 rows
pilot rust: incremental 2 rows
pilot rust: incremental 1 rows
pilot rust: incremental 5 rows
pilot rust: incremental 4 rows
pilot rust: incremental 3 rows
pilot rust: incremental 2 rows
pilot rust: incremental 1 rows
pilot rust: incremental 2 rows
pilot rust: incremental 1 rows
harness v1.1: cargo debuginfo stripped + target-dir-scoped watchdog
pilot java: incremental 6 rows
pilot java: incremental 5 rows
pilot java: incremental 4 rows
pilot java: verified 3
pilot java: incremental 3 rows
pilot java: incremental 2 rows
pilot java: incremental 1 rows
harness v1.0: fix blind du guard (timeout+defensive parse), 32GiB limit, 20s watchdog poll w/ reclaim, incremental per-row push
pilot java: results
pilot rust: results
harness v0.9: java report collection (surefire/gradle XML + gradle HTML), du-based container disk guard, rust du watchdog
pilot java: results
pilot js: verified
pilot js: results