snupilab/humanoidtoolbench-teleop

ToolBook: HumanoidToolBench demonstrations

Paper: arXiv:2610.02089

Raw Meta Quest teleoperation demonstrations for HumanoidToolBench, recorded on the Unitree G1 in MuJoCo through its whole-body controller.

Each recording uses LeRobot v2.1 files: 50 Hz, an ego-view video and wrist-view videos when recorded, and the recording contract in meta/simple_contract.json. Camera resolutions are declared in meta/info.json. Code versions, WBC settings, and the render backend are in meta/info.json and meta/episodes.jsonl.

Task instructions

Instruction metadata follows the benchmark's current instruction table. Each scenario uses the same sentence in S (standard, one tool) and R (decoy, written D in the paper, two tools). L0 asks for tool selection; L1 and L2 share the instruction to pick the tool and complete the task:

L0 success requires lifting the correct tool. L1 and L2 require completing the scenario's task; L2 also uses the layout that requires walking along the table.

Layout

raw/<operator>/<env_id>/level-<dr_level>/
  data/chunk-000/episode_000000.parquet
  videos/chunk-000/observation.images.ego_view/episode_000000.mp4
  videos/chunk-000/observation.images.wrist_left/episode_000000.mp4
  videos/chunk-000/observation.images.wrist_right/episode_000000.mp4
  meta/info.json, episodes.jsonl, episodes_stats.jsonl, tasks.jsonl,
       modality.json, simple_contract.json
viewer/<env_id>.parquet  # generated episode index for the Hub viewer

Episode indices start at 0 inside every level-<dr_level> directory, so recordings from different operators are kept apart and are only merged, with re-indexed episodes, when a training set is prepared.

Dataset Viewer

The viewer shows one row per episode. Choose all for all tasks or an environment subset such as G1BallMove-L2-S for a single task. Subset names omit the trailing environment version (-v0); row env_id values and data paths retain it. Each row links the available ego and wrist videos to the task instruction, success flag, operator, and episode metadata. The task level (L0, L1, or L2 in the environment ID) is separate from dr_level, which controls domain randomization.

Actions and robot states remain in the original per-frame Parquet files, referenced by each viewer row. The viewer is an episode index, not a table of individual frames. The generated viewer/<env_id>.parquet files and root dataset configuration connect the existing videos to LeRobot metadata. Video references are pinned to the source repository revision used to build the index. Hugging Face's generic video-folder loader does not perform that join from meta/tasks.jsonl and meta/episodes.jsonl automatically.

Download

hf download snupilab/humanoidtoolbench-teleop --repo-type dataset --local-dir data/hub

Add --include "raw/<operator>/*" to fetch a single operator's recordings.

Isaac Sim RTX replay

The rendered/isaac/ohchul/G1BallMove-L2-S-v0/level-0/ directory contains an additional image version of the same 169 trajectories: 507 H.264 MP4 videos from the ego, left wrist, and right wrist cameras, at 640 x 360 and 50 fps (241,640 frames per camera). Select isaac-G1BallMove-L2-S in the dataset viewer to view this version. The existing all and task configurations continue to show the raw recordings.

Isaac Sim 4.5.0 with RTX RayTracedLighting renders the recorded robot and object poses kinematically. This does not resimulate the trajectories in PhysX and does not include animated USD scenes. State/action Parquet files are byte-identical to raw/ohchul/G1BallMove-L2-S-v0/level-0/ at source revision 3203df67b559dfc289ddf084ffb25412621e2e1d. Episode indices match the source one-to-one. Split raw and rendered versions by source trajectory to avoid train/test leakage.

The exact-preservation statements above describe raw/. In the rendered copy, meta/info.json adds rendering_info, and meta/episodes.jsonl records render_backend=isaac plus source_render_backend=mujoco. Its simulator and environment_config still describe the original MuJoCo recording and scene. Host-specific texture cache paths are made home-relative; regenerate or relocate these caches when replaying on another machine. The recording contract's timing conventions are inherited; its images are now rerenders, not the pixels originally seen by the teleoperator. Existing statistics cover non-image fields; RGB statistics are not provided.

rerender.json documents provenance and camera mapping. verification.json records full decoding, frame counts and SHA256 checks for every video. Its non-video identity check is explicitly the pre-publication check, before the documented metadata changes. The new viewer pins video references to rendered data revision 214af060f8179b38fef1f10de35494c4dc3d02e8.

This addition is released under the dataset's CC BY-NC 4.0 license and does not grant additional rights to third-party assets. Material libraries, mesh asset packs, caches, credentials and operational logs are not included.

Isaac Sim RTX release 20260911_07c76e

Select isaac-20260911-07c76e in the dataset viewer. This release includes all 4,462 source episodes: 4,361 raw recordings and 100 trimmed recordings passed render validation; one additional raw recording failed. Filter render_validation == "passed" to select the 4,461 verified episodes. The failed episode is raw/seungyeop/G1BallRetrieve-L2-S-v0/level-0/episode_000020: wrist_right contains an entirely black decoded frame at index 2759 (55.18 seconds). It is included and explicitly labelled, not repaired or certified. Rendering validation is independent of the original task outcome: episode_success, episode_failure, and termination fields retain that outcome.

Files are under rendered/isaac/20260911_07c76e/. Video references in viewer/isaac/20260911_07c76e.parquet are pinned to the release data commit. The immutable source is `07c76e0246e9f6c6de89f335aa767bd29d3f89ee`; the existing source files and previous viewer configurations are unchanged. source_category distinguishes raw/trimmed inputs; source_episode_id identifies the immediate source episode, and original_raw_episode_id retains trimmed-slice provenance. data_path gives the exact rendered-release frame-level Parquet path. task_level and mode retain the source environment label, including historical directory naming. replay_tool_level and dr_level come from the saved environment configuration; an L0-derived clip can retain an L1/L2 replay scene configuration.

All three cameras (ego_view, wrist_left, wrist_right) are H.264 MP4, 640x360 at 50 fps. Images were generated by Isaac Sim 4.5.0 RTX RayTracedLighting from recorded trajectories using kinematic replay. This is not a new PhysX dynamics simulation and contains no animated USD scenes or redistributed material libraries. Robot state/action Parquet files retain the source values. Release metadata labels render_backend as isaac, preserves the original value as source_render_backend, and includes rendering_info provenance. Local absolute home paths in copied metadata were replaced with home-relative paths; source files remain available at the pinned revision.

License

The recordings are released under CC BY-NC 4.0 for non-commercial use. Please cite HumanoidToolBench in any work that uses them. The benchmark code is MIT-licensed. Third-party assets, models and data keep their own terms.