Datasetpublic

alexshpunt/explicit-edit-benchmark

Explicit Edit Benchmark 226 deterministic exact-edit tasks, run by different agents, harnesses, models and configurations. Every observation records what the harness did and whether the resulting files matched byte for byte. Source code and benchmark runner: GitHub — Explicit Edit Benchmark Open the interactive Explorer to compare agents, harnesses, models, versions, reasoning modes, correctness, recovery, time, cost and tokens. Leaderboard by model route Score v2… See the full description on the dataset page: https://huggingface.co/datasets/alexshpunt/explicit-edit-benchmark.

sourceHugging Facecc-by-4.0updated 1d agoView on Hugging Face

5likes19kdownloads

Dataset Card

Explicit Edit Benchmark

226 deterministic exact-edit tasks, run by different agents, harnesses, models and configurations. Every observation records what the harness did and whether the resulting files matched byte for byte.

Source code and benchmark runner: GitHub — Explicit Edit Benchmark

[Open the interactive Explorer](https://huggingface.co/spaces/alexshpunt/benchmark-explorer) to compare agents, harnesses, models, versions, reasoning modes, correctness, recovery, time, cost and tokens.

Leaderboard by model route

Score v2 = coverage × quality, where quality is 75% first exact and 25% final exact. Repeated runs are averaged inside each configuration and task; configurations then have equal weight inside each task, and tasks have equal weight. The same numbers are in views.json, and the Explorer breaks them down by harness, version and reasoning mode.

Model	Provider	Score	Coverage	Tasks	Observations	Harnesses
`gpt-6.1-sol`	`openai-codex`	100.0%	100.0%	226	678	3
`muse-spark-1.3-contributor`	`opencode-go`	100.0%	100.0%	226	1582	5
`deepseek-v4-pro`	`deepseek`	99.7%	100.0%	226	678	2
`gemini-3.8-flash`	`gemini`	99.7%	100.0%	226	226	1
`deepseek-v4.1-flash`	`deepseek`	99.4%	100.0%	226	3166	11
`deepseek-v4.1-flash`	`opencode-go`	99.3%	100.0%	226	1356	5
`mimo-v2.6-pro`	`opencode-go`	98.6%	100.0%	226	226	1
`mimo-v2.6-flash`	`opencode-go`	98.3%	100.0%	226	1130	4
`qwen3.8-flash`	`opencode-go`	98.3%	100.0%	226	1356	4
`deepseek-v4-flash`	`deepseek`	98.2%	100.0%	226	2486	9
`deepseek-v4-flash`	`opencode-go`	97.3%	100.0%	226	904	3
`gpt-6-sol`	`openai-codex`	97.1%	100.0%	452	1130	3
`grok-4.6`	`opencode-go`	97.1%	100.0%	226	452	2
`deepseek-v4.1-flash`	`routerai-deepseek`	97.0%	100.0%	226	226	1
`kimi-k2.7-code`	`opencode-go`	96.7%	100.0%	226	452	2
`space-bunny-free`	`opencode-go`	96.3%	100.0%	226	452	2
`glm-5.3-flash`	`zai`	96.0%	100.0%	226	1808	8
`space-bunny-free`	`opencode`	95.8%	100.0%	226	452	2
`gpt-5.6-sol`	`openai-codex`	95.0%	100.0%	226	678	3
`gpt-5.6-terra`	`openai-codex`	95.0%	100.0%	226	678	3
`gpt-5.6-luna`	`opencode-go`	94.1%	100.0%	226	226	1
`mimo-v2.6-pro`	`xiaomi`	93.8%	100.0%	226	452	2
`gpt-5.6-luna`	`openai-codex`	93.7%	100.0%	452	10860	35
`gpt-6-luna`	`opencode-go`	93.0%	100.0%	226	226	1
`glm-5.3-flash`	`routerai-z-ai`	92.8%	100.0%	226	226	1
`qwen3.7-plus`	`opencode-go`	92.4%	100.0%	226	226	1
`glm-5.3-flash`	`opencode-go`	92.0%	100.0%	226	452	2
`minimax-m3`	`opencode-go`	90.3%	100.0%	226	678	3
`ling-3.0-flash-vl`	`routerai-deepinfra`	90.3%	100.0%	226	226	1
`longcat-2.5-preview-free`	`opencode-go`	89.3%	100.0%	226	452	2
`mimo-v2.5`	`xiaomi`	89.2%	100.0%	226	2486	10
`gpt-6-luna`	`openai-codex`	87.2%	100.0%	452	1356	3
`qwen3.6-35b-a3b`	`unknown`	85.1%	100.0%	226	226	1
`mercury-2.5`	`routerai-inception`	76.4%	100.0%	226	226	1

The harness list behind each row is in data/models.jsonl.gz, and views.json holds the same aggregates for the other groupings: by harness, by agent and by reasoning mode.

Data contributors

Thank you to everyone who has shared benchmark observations. Your work makes this public comparison possible.

Contributor	Accepted runs	Configurations
@ashokkumards	1	1
@dirac-run	4	4
@TreyThomasCodes	7	7
@user2221	5	5
@xuankunv1	4	4
@Yugimob	23	23

Accepted harnesses

Harness	Accepted runs	Configurations
`anchor-edit`	1	1
`baseline-agent`	30	28
`bb`	7	7
`codex-cli-default`	2	6
`d3ara1n-pi-hashline-edit`	1	1
`dirac-default`	4	4
`dsh-code`	2	6
`dsh-standard`	2	6
`github-copilot-cli-default`	2	6
`jerryan-pi-hashline-edit`	1	1
`oh-my-pi-default`	3	7
`opencode-default`	2	6
`personal-pi-extensions-opencode`	1	1
`pi-agent-ide`	26	28
`pi-apply-patch`	1	1
`pi-better-edit`	5	5
`pi-better-read-edit`	1	1
`pi-codex-conversion`	1	1
`pi-codex-edit`	1	1
`pi-codex-minimal-tools`	1	1
`pi-codex-tools`	1	1
`pi-default`	27	21
`pi-edit-safe`	1	1
`pi-hash-anchored-edit`	1	1
`pi-hash-edit`	1	1
`pi-hashline-context-edit`	1	1
`pi-hashline-edit`	1	1
`pi-hashline-edit-pro`	24	24
`pi-hledit`	1	1
`pi-lean-edit`	1	1
`pi-lector`	1	1
`pi-mono-multi-edit`	1	1
`pi-openai-codex-compat`	1	1
`pi-semantic-edit`	1	1
`pi-str-replace-editor`	1	1
`pi-wayfinder`	1	1

Tables

Config	One row per
`profiles`	configuration that was run, with its agent, harness, model and exact versions
`configurations`	recipe behind a configuration, safe to publish
`trials`	task and attempt, with the first and final exact result
`rounds`	attempt, with timing, tokens, cost and timeout state
`tool-calls`	tool the agent used, with its category and outcome
`submissions`	accepted run, with its owner, purpose and definitions

The Dataset Viewer shows every config. dataset-index.json holds the source hashes, contracts, task sets, completeness and counts, and views.json, leaderboard.json and summary.json hold the aggregated rankings.

Source and contribution

Source code, run instructions and the contribution guide live at alexshpunt/explicit-edit-benchmark. Accepted bundles are kept under source/, the shards and summaries are views rebuilt from them, and each accepted harness family has a README badge under badges/.

Observations hold no prompts, arguments, commands, output, sessions, workspaces or credentials.