a8cheng/SR-3D-Bench

Spatial Region 3D (SR-3D) Aware Benchmark

Paper: https://arxiv.org/abs/2509.13317 Project page: https://www.anjiecheng.me/sr3d Code: https://github.com/AnjieCheng/SR-3D

[!IMPORTANT] *[Feb. 18, 2026] UPDATE: To improve compatibility with general-purpose VLMs, the benchmark is reformulated into multiple-choice and numerical questions following the VSI-Bench evaluation protocol. Videos are annotated with set-of-marks to explicitly indicate regions. The benchmark will be compatible with VLMEvalKit and lmms-eval.*

Overview

SR-3D-Bench is a region-level spatial understanding benchmark for videos introduced in the SR-3D paper.

Existing video benchmarks lack explicit region annotations, making spatial reasoning ambiguous when multiple similar objects are present or when referring to specific areas in a scene. SR-3D-Bench addresses this limitation by providing region-grounded spatial question answering.

The benchmark is curated from ScanNet, ARKitScenes, and Matterport3D video scan datasets with 3D ground truth. It uses oriented 3D bounding boxes from EmbodiedScan, where objects are aligned in a canonical coordinate system to ensure accurate width, height, and distance measurements.

SR-3D-Bench includes:

• —Qualitative QA: choice-based, predicate-based, and multiple-choice questions
• —Quantitative QA: object width, height, and spatial distance estimation

All QA pairs are generated using template-based conversation generation.

Usage

from datasets import load_dataset

sr3d_bench = load_dataset("a8cheng/SR-3D-Bench")