Datasetpublic

HindsboNikolaj/scope-benchmark

SCOPE Benchmark Evaluation benchmark for the HRI '26 paper SCOPE: A Real-Time Natural Language Camera Agent at the Edge (arXiv:2606.02951). Test-only — no train split. 541 questions × 4 Blender scenes × 8 task categories. The code that runs this benchmark lives at github.com/HindsboNikolaj/SCOPE. When you chain a language model and a vision model together, how do you know which one failed? Contents scope-benchmark/ scope_541.csv… See the full description on the dataset page: https://huggingface.co/datasets/HindsboNikolaj/scope-benchmark.

sourceHugging Facecc-by-nc-4.0updated 4mo agoView on Hugging Face

0likes112downloads

Dataset Card

SCOPE Benchmark

![Code on GitHub](https://github.com/HindsboNikolaj/SCOPE) ![Paper](https://doi.org/10.1145/3757279.3785641) ![arXiv](https://arxiv.org/abs/2606.02951) ![Space](https://huggingface.co/spaces/HindsboNikolaj/scope)

Evaluation benchmark for the HRI '26 paper *SCOPE: A Real-Time Natural Language Camera Agent at the Edge* (arXiv:2606.02951). Test-only — no train split. 541 questions × 4 Blender scenes × 8 task categories.

The code that runs this benchmark lives at [github.com/HindsboNikolaj/SCOPE](https://github.com/HindsboNikolaj/SCOPE).

When you chain a language model and a vision model together, how do you know which one failed?

scope-benchmark/
  scope_541.csv                       541-row evaluation set
  paper.pdf                           HRI '26 paper PDF
  scenes/
    whitechapel/whitechapel.blend     Urban exterior (France)
    book-nook/book-nook.blend         Interior room
    city-street/city-street.blend     Urban street, neon signage
    postwar-city/postwar-city.blend   Damaged urban exterior

All four .blend files are texture-packed and load standalone in Blender 4.0+. Pulled from the canonical repo at github.com/HindsboNikolaj/SCOPE.

Task categories (8)

Counting · descriptor · location/spatial · OCR identification · single-call · multi-step command · multi-step reasoning · comparative/relational. See paper §4 for the breakdown.

CSV schema

Column	Description
`question_id`	Q001 … Q541
`file_location`	Path to scene `.blend` (relative to `scenes/`)
`question`	Natural-language prompt to the agent
`expected_answer`	Ground-truth answer for the judge
`eval_category`	One of the 8 task categories
`difficulty`	Easy / Medium / Hard
`multi_step_mode`	Single-call vs multi-step expected behavior
`required_tools_policy`	Strict / Lenient tool-use enforcement
`expected_tool_order_json`	Optional expected tool sequence
`evaluation_notes`	Free-text grading hints

Usage

python

from huggingface_hub import snapshot_download
snapshot_download(repo_id="HindsboNikolaj/scope-benchmark", repo_type="dataset",
                  local_dir="benchmark/")

Then run the eval pipeline from the code repo:

bash

git clone https://github.com/HindsboNikolaj/SCOPE
cd SCOPE && bash scripts/01_install.sh
bash scripts/run_eval_pipeline.sh

Reproducibility note

A handful of the original scene textures (≈ 10% of references in postwar-city, plus a few paid Blender Market HDR addons) are not in this distribution — they were external assets owned by the original scene authors that could not be redistributed under permissive licenses. The scenes still load, render, and produce correct answers for the majority of questions; the missing surfaces appear flat-shaded.

The full provenance and a re-acquisition manifest for paper-grade reproduction live at `docs/MISSING_TEXTURES.md` in the code repo.

License

CC-BY-NC-4.0 for the benchmark questions and metadata. Individual scene .blend files retain the licenses of their original authors (Sketchfab / MySimsWorld / community contributors); see scene-by-scene attribution in the code repo's docs/.

Citation

bibtex

@inproceedings{hindsbo2026scope,
  title         = {SCOPE: A Real-Time Natural Language Camera Agent at the Edge},
  author        = {Hindsbo, Nikolaj and Ehsani, Sina and Mishra, Pragyana},
  booktitle     = {Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI '26)},
  year          = {2026},
  publisher     = {ACM},
  doi           = {10.1145/3757279.3785641},
  eprint        = {2606.02951},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
}