Qwen/RecreationBench

RecreationBench

GitHub · Website · Leaderboard · Report ↗

RecreationBench is a benchmark of 250 application-recreation tasks across Ubuntu, macOS, Windows, Android, and Web. It is the held-out evaluation suite of RecreationWorld, a framework for scalable and verifiable environments for hybrid computer-use agents.

Given a high-level recreation request and interactive access to a running reference application, an agent explores its behavior, implements a new application, and verifies its own result. The agent can combine GUI interaction with coding tools, but cannot access the reference source code or evaluation tests. Evaluation measures observable functionality and visual fidelity rather than source-code similarity.

This dataset provides the task descriptors, reference preparation assets, tests, fixtures, and available visual-evaluation annotations. The execution framework is maintained in the GitHub repository linked above.

Benchmark Statistics

The canonical task lists are available in `tasks/`.

The Dataset Viewer shows one row per task: identity, platform, difficulty, upstream repository, commit, and task path. Select all for the full 250-task index or a platform for its 50 tasks; all configurations use the test split.

Data Format

The dataset uses a per-task directory structure:

<dataset-root>/
  metadata/
    <platform>.jsonl         # task index for the Dataset Viewer
  <platform>/
    <task-id>/
      instance.json
      reference/
      tests/
      vlm_assertions.json    # when provided
      LICENSE*               # license filenames and locations vary

Desktop and Android references are built from pinned upstream commits with any declared patches. Web references are bundled site snapshots, with repo and commit set to null.

An example instance.json:

{
  "schema_version": 1,
  "instance_id": "ubuntu/adrienverge-photocollage",
  "platform": "ubuntu",
  "repo": "https://github.com/adrienverge/PhotoCollage",
  "commit": "3289328d09dee6d19895ef013633c1c107ff41d5",
  "patches": [],
  "package": null,
  "license": null
}

Download

Download the dataset as files with the Hugging Face CLI:

python -m pip install --upgrade huggingface_hub
hf download Qwen/RecreationBench \
  --repo-type dataset \
  --local-dir ./RecreationBench

Download the dataset as files with ModelScope:

python -m pip install --upgrade modelscope
modelscope download Qwen/RecreationBench \
  --repo-type dataset \
  --local-dir ./RecreationBench

Preserve the <platform>/<task-id>/ hierarchy and all test fixtures, including hidden files. The Dataset Viewer and datasets.load_dataset() expose the task index; running the task bundles requires the execution framework.

Evaluation

The framework runs reference preparation, agent recreation, and evaluation in the appropriate platform environment. It reports two complementary metrics:

• —Prog: reference-grounded programmatic checks of application behavior.
• —VLM: visual checks of the recreated application using a vision-language-model judge.

Scores are macro-averaged across tasks within each platform, then equally weighted across the five platforms. Average is the unweighted mean of Prog and VLM. Visual-annotation coverage varies by task.

See the GitHub README for full setup instructions, deployment guides, and platform-specific evaluation commands.

License

This is a multi-license collection. The execution framework is MIT-licensed; upstream applications and bundled assets retain their own terms. Web task notices separately cover benchmark-authored code (MIT), benchmark-authored dataset content (CC BY 4.0), and third-party assets.

Retain the applicable per-task and per-file notices. Where terms are missing or unclear, consult the pinned upstream source or contact the maintainers before redistribution.

Citation

If you use RecreationBench, please cite:

@misc{qwen2026recreationworld,
      title={RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents},
      author={Shuai Bai and Jiayong Deng and Sicheng Fan and Yikun Fu and Chang Gao and Xuhao Hu and Mianqiu Huang and Yizhen Jiang and Yuheng Jing and Dehui Kong and Keliang Li and Ning Li and Wanli Li and Dayiheng Liu and Dunjie Lu and Changwei Luo and Que Shen and Zheyuan Wang and Zijian Wang and Jie Wu and Gao Wu and Zhihui Xie and Rui Xie and Haiyang Xu and An Yang and Jiakang Yuan and Yanming Zhang and Jiajun Zhang and Xi Zhang and Zhenru Zhang and Zhuo Zhen and Mingkang Zhu and Bowen Zhou},
      year={2026},
      eprint={2609.22000},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.22000},
}

Contact

For questions about the dataset or evaluation, please contact xiezhihui.xzh@alibaba-inc.com or gaochang.gao@alibaba-inc.com.