Datasetpublic

LocalLLaMA/terminal-bench-mini

terminal-bench-mini Fourteen of Terminal-Bench 2.0's ninety tasks, picked so that ranking agents on the subset reproduces ranking them on the whole benchmark. Running ninety tasks five times each is how the official leaderboard is built. That is out of reach if you are comparing quant variants, fine-tunes or local models on your own hardware. This subset turns a multi-day sweep into a few hours. Same approach as deepswe-mini: take the published per-task results, rank the field… See the full description on the dataset page: https://huggingface.co/datasets/LocalLLaMA/terminal-bench-mini.

sourceHugging Faceapache-2.0updated 17d agoView on Hugging Face

6likes2.4kdownloads

Dataset Card

terminal-bench-mini

Fourteen of Terminal-Bench 2.0's ninety tasks, picked so that ranking agents on the subset reproduces ranking them on the whole benchmark.

Running ninety tasks five times each is how the official leaderboard is built. That is out of reach if you are comparing quant variants, fine-tunes or local models on your own hardware. This subset turns a multi-day sweep into a few hours.

Same approach as deepswe-mini: take the published per-task results, rank the field on the full benchmark, then find a small subset that preserves that ranking. These fourteen tasks reproduce about 87% of the full benchmark's pairwise ordering, and 97% of it when two agents are more than ten points apart.

The tasks

task	field pass rate	spread across agents	run-to-run noise
filter-js-from-html	11.9%	0.310	0.022
install-windows-3.11	11.9%	0.258	0.090
mteb-retrieve	41.2%	0.420	0.149
db-wal-recovery	56.8%	0.448	0.097
qemu-alpine-ssh	69.3%	0.417	0.093
build-pmars	75.2%	0.361	0.122
mcmc-sampling-stan	81.5%	0.328	0.100
rstan-to-pystan	81.7%	0.329	0.102
kv-store-grpc	86.4%	0.297	0.063
bn-fit-modify	87.6%	0.295	0.056
regex-log	90.8%	0.270	0.025
custom-memory-heap-crash	91.9%	0.273	0.000
crack-7z-hash	93.5%	0.228	0.022
vulnerable-secret	94.6%	0.226	0.000

Field pass rate is the mean over 37 agents. Spread is the standard deviation of those per-agent rates: a task with spread near zero tells you nothing about who is better. Run-to-run noise is the average standard deviation across repeated trials of the same agent on the same task.

Task definitions live in Terminal-Bench 2.0. This dataset names them, it does not redistribute them.

Method

The source is every trial published to the Terminal-Bench 2.0 leaderboard: 32,803 trials, 27,405 of them scorable, across 90 tasks and 75 agent/model submissions. Trials whose environment failed to start carry exception_info and no verifier result; they are dropped rather than counted as failures. The 37 submissions that attempted at least 85 tasks form the reference ranking.

The subset is a stratified sample. Tasks that every agent passes or every agent fails carry no ranking information and are excluded first. The rest are sorted by field pass rate into seven difficulty bands; within each band the quieter half is preferred, since a task that flips between identical runs costs every future user extra trials to average out; two tasks are drawn per band with a fixed seed (20260919).

Quality is measured as pairwise agreement: over every pair of agents, how often the subset orders them the way all ninety tasks do, broken out by how far apart the pair really is.

pair separation	agreement
more than 10 points	97.3%
5 to 10 points	80.1%
2 to 5 points	73.2%
under 2 points	65.1%

Agreement also rises with how many tasks you run, which analysis/size_curve.csv records: 14 tasks average 85%, 22 average 89%, 30 average 91%, 60 average 95%. If the systems you are comparing are close, add tasks or add trials.

What it is good for

Use it to answer "is this configuration roughly competitive". The top of the table is stable — the leading agent leads on both, and every agent in the full top ten stays in the mini top twelve. Maximum rank movement is 16 places, and it happens in the crowded middle where full-benchmark scores differ by fractions of a point.

Do not use it to claim one agent beats another by two points. Five of the fourteen tasks have run-to-run noise above 0.09, so run more than one trial per task or expect a few points of movement that mean nothing.

Files

—data/tasks.jsonl — the fourteen tasks with their field statistics
—analysis/leaderboard.csv — all 37 agents scored on the mini set and the full 90, with both ranks and the movement between them
—analysis/mini_trials.csv — the agent × task pass rates behind that leaderboard
—analysis/size_curve.csv — agreement against subset size, 200 random subsets per size

Reproducing

Source data is the public harborframework/terminal-bench-2-leaderboard repository. Every trial record is a small result.json sitting next to terminal logs that make the repository 121 GB, so fetch only the records:

bash

git clone --filter=blob:none --no-checkout \
  https://huggingface.co/datasets/harborframework/terminal-bench-2-leaderboard
cd terminal-bench-2-leaderboard
git sparse-checkout set --no-cone '/**/result.json'
git checkout

That is 348 MB and about two minutes. Each record carries verifier_result.rewards.reward. Note that two task-name conventions coexist in the repository, terminal-bench/<name> and <name>; normalise them or the same task counts twice and no two agents ever overlap.

Caveats

Agreement is measured against Terminal-Bench 2.0 as scored by these 37 submissions, all of them frontier hosted models. A small local model sits below that range, where the subset is untested — though a gap that large is the case it handles most reliably.