ldh-sky/unitybench

UnityBench — by 14Dimension Enterprise

https://www.14dimension.com/

*A two-track evaluation for Unity 6 / C# coding models: domain knowledge + real-world bug review.*

Generalist code models are strong — but how well do they actually handle Unity 6, and can they find real bugs in real game code without crying wolf? There was no good public answer. This is one.

Built by 14Dimension Enterprise from a shipping Unity Game — bugs and fixes reconstructed from real version-control history. Code: https://github.com/leegod/unitybench

Tracks

1. Knowledge (data/knowledge/, ~146 items) — Unity 6 lifecycle/API Q&A + short "write this MonoBehaviour" compile tasks. Tests whether a model actually knows current Unity 6 (post-cutoff API, common pitfalls).

2. Review (data/review/, 24 buggy + 22 post-fix) — real Unity gameplay methods, each in two states: the pre-fix (buggy) version and the post-fix version, reconstructed from version-control history. We measure recall (does it flag the real bug?) — and the "post-fix" set turned out to be a methodology finding, not a clean specificity control (see below).

Review track — a methodology finding, not a leaderboard

We ran seven models on 24 buggy + 22 "post-fix" gameplay methods, expecting to rank them by recall and specificity. We are not publishing a balance ranking — here's why.

A naive harmonic-mean "balance" would rank GPT-5.5 and Opus 4.8 last — and that is wrong. We hand-read their flags on the "post-fix" methods, then ran a convention-aware per-flag judge (cross-checked by two independent judges) across all models: the thorough frontier models' flags are predominantly real review issues (unchecked return values, missing null guards, non-atomic reward grants, missing idempotency, Resources.Load null, OnDestroy on scene-unload), not false positives. The ground truth is the problem: a "post-fix" method is the version where one bug was fixed, not a method with zero issues, so "specificity" punishes the most thorough reviewer. Binary recall/specificity is the wrong instrument; the right one is a per-flag verdict — but two independent judges disagree enough (46–88% agreement) that absolute precision needs human gold labels, which is why only recall + flag-rate are shown here. See the repo README. Treat this as a case study in why bug-detection benchmarks are hard.

Fields

• —review_bugs: id, code (buggy method), bug (description), severity, context_needed
• —review_clean: id, code (fixed method)
• —knowledge: Unity 6 Q&A / compile tasks

License & citation

Data: CC-BY-4.0 — free to use, including commercially, with attribution to 14Dimension Enterprise.

@misc{unitybench2026,
  title  = {UnityBench: Unity 6 Knowledge \& Real-World Bug-Review Evaluation},
  author = {14Dimension Enterprise},
  year   = {2026},
  url    = {https://github.com/leegod/unitybench}
}

— Built and measured hands-on by 14Dimension Enterprise while shipping a Unity game.