n-alignment/USACO-Judge
USACO-Judge A benchmark for judging competitive-programming solutions: given a problem and a candidate solution, decide Accept / Reject, and on Reject, produce a concrete input that breaks the code. Existing hacking benchmarks are built entirely from known-wrong candidates, so they only test the breaking half of verification. USACO-Judge is balanced 1:2 AC:non-AC, so it also tests whether a verifier correctly accepts solutions that are actually correct. Built from 39 official… See the full description on the dataset page: https://huggingface.co/datasets/n-alignment/USACO-Judge.
USACO-Judge
A benchmark for judging competitive-programming solutions: given a problem and a candidate solution, decide Accept / Reject, and on Reject, produce a concrete input that breaks the code. Existing hacking benchmarks are built entirely from known-wrong candidates, so they only test the breaking half of verification. USACO-Judge is balanced 1:2 AC:non-AC, so it also tests whether a verifier correctly accepts solutions that are actually correct. Built from 39 official USACO 2026 problems and 900 real candidate submissions.
problems (39 rows)
One row per problem: cpid (USACO's official problem id — usaco.org/index.php?page=viewproblem2&cpid=<cpid>), title, contest, division, problem_number, source_url, statement_md, checker_kind, time_limit_s, memory_limit_mb, n_tests, n_sample_tests, point_scores, val_cpp (validator), std_cpp (reference solution), chk_cpp (custom checker, null for problems that use a standard comparator), std_source_submission_id, std_origin.
submissions (900 rows: 270 easy + 630 hard)
One row per submission: sid, cpid (links to problems), split (easy/hard), code, verdict (AC/WA/TLE/MLE/RE), score (0-100). 300 are AC, 600 are not. The easy split draws from the 18 Bronze/Silver problems, hard from the 21 Gold/Platinum/Open problems.
Non-AC candidates are stratified across five score bands (1-19, 20-39, 40-59, 60-79, 80-99), each a distinct plateau of suboptimal algorithms rather than a single bug pattern.