ababa134/fuzzeval-humaneval-mbpp

FuzzEval unit tests for HumanEval-f and MBPP-f

Automatically generated unit tests for a reproduction of the ICML 2026 paper "Towards Functional Correctness of Large Code Models with Selective Generation" (Jeong, Kim & Park — arXiv:2505.13553, official repo trustml-lab/selective-code-generation).

The paper's FuzzEval paradigm replaces a benchmark's handful of hand-written unit tests with hundreds of unit tests obtained by fuzzing the reference solution. This dataset is our reconstruction of that step for openai/openai_humaneval and google-research-datasets/mbpp (sanitized).

How it was built

For each problem we take the canonical solution as the fuzzing target:

1. 1.Seed corpus — argument tuples parsed out of the benchmark's own asserts.
2. 2.Coverage-guided loop — inputs are mutated with type-aware operators (integers, floats, strings, lists/tuples/sets, dicts, recursively); a mutant that reaches a new (line -> line) arc of the reference is added to the corpus, exactly as a coverage-guided fuzzer such as Atheris would.
3. 3.Recording — every input that executes without raising, twice with the same result (determinism check) and whose repr round-trips, yields a unit test (u, v) with v = reference(u).

Budget per problem: 600 unique tests, 60k executions, 90 s.

Contents

fuzz_tests.jsonl — one row per problem:

seed: true marks a test whose input came from the benchmark's own asserts — those are the "manual unit tests" used by the paper's SCG-small baseline.

Stats

• —591 problems attempted, 562 with a full 600-test suite (153 HumanEval, 409 MBPP)
• —337,200 unit tests total
• —26 problems excluded: 3 have no literal seed inputs, 1 has no valid seed execution, and 22 hit the time budget (reference solutions that blow up on mutated inputs — the paper handles these with manual input-constraint post-processing, which we did not do).

Reproduce

python fuzz_tests.py --target 600 --max-execs 60000 --out data/fuzz_tests.jsonl

Code: see the Workspace tab of the reproduction logbook.