datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
PhysicalAI-Robotics-GR00T-X-Embodiment-Sim
PhysicalAI-Robotics-GR00T-X-Embodiment-Sim
Github Repo: Isaac GR00T N1
We provide a set of datasets used for post-training of GR00T N1. Each dataset is a collection of trajectories from different robot embodiments and tasks.
Cross-embodied bimanual manipulation: 9k trajectories
Dataset Name
#trajectories
bimanual_panda_gripper.Threading
1000
bimanual_panda_hand.LiftTray
1000
bimanual_panda_gripper.ThreePieceAssembly
1000… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim.PhysicalAI-Autonomous-Vehicles
PHYSICAL AI AUTONOMOUS VEHICLES
The PhysicalAI-Autonomous-Vehicles dataset provides one of the largest, geographically diverse collections of multi-sensor data empowering AV researchers to build the next generation of Physical AI based end-to-end driving systems. This dataset is ready for commercial/non-commercial AV use per the license agreement.
Data Collection Method
Automatic/Sensor
Labeling Method
Automatic/Sensor
This dataset has a total of 1700 hours of driving… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles.PhysicalAI-Robotics-Open-H-Embodiment
Dataset Description:
Open-H-Embodiment is a community‑driven dataset initiative building the open, shared foundation needed to train and evaluate AI autonomy models for surgical robotics and ultrasound.
This dataset is a multi-embodiment collection of LeRobot datasets of paired kinematics and video, across tasks such as tabletop exercises, clinical procedures, as well as simulations of healthcare robotics applications.
Maintainer / Hosting Organization:
NVIDIA… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment.PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios
Dataset Description:
PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios is a large-scale synthetic video dataset of autonomous-driving scenes generated with NVIDIA's internal Omniverse simulation platform. Each clip is a temporally consistent multi-camera surround capture of one ego vehicle and surrounding traffic participants, paired with per-camera VLM captions. The dataset is designed to fill gaps in real-world driving data along two axes: (1) targeted long-tail… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios.PhysicalAI-SmartSpaces
Physical AI Smart Spaces Dataset
Overview
Comprehensive, annotated dataset for multi-camera tracking and 2D/3D object detection. This dataset is synthetically generated with Omniverse and Cosmos Transfer.
This dataset consists of over 280 hours of video from across nearly 1,800 cameras from indoor scenes in warehouses, hospitals, retail, and more. The dataset is time synchronized for tracking humans, forklifts, pallet trucks and Autonomous Mobile Robots (AMRs)… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.PhysicalAI-Robotics-Locomanipulation-GRAIL
📢 News
[2026-07-15] Released task-general tracking policy checkpoints trained on the released data. Follow the tracking doc to use them to track our released motion data.
[2026-07-14] Updated data/pickup_table and data/pickup_ground. If you downloaded them before this date, please re-download.
Dataset Overview
Tabletop Pickup
Ground Pickup
Tabletop Manipulation
Ground Manipulation
Sitting
Curb
Slope… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL.PhysicalAI-Robotics-Manipulation-Kitchen-Demos
PhysicalAI-Robotics-Manipulation-Kitchen-Demos
We provide a 600 hours of human-teleoperated demonstrations across 316 different tasks, totalling 55k trajectories.
The datasets are collected using Franka Panda robot with an Omron mobile base.
The datasets follow the LeRobot format. Here is an overview of important elements of each dataset:
Click to expand dataset structure
lerobot/
├── meta/ # Metadata files describing the dataset
│ ├── info.json… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Manipulation-Kitchen-Demos.PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes
PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes Dataset Card
Dataset Description
PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes is a large-scale synthetic dataset of physically-simulated multi-object interaction scenes, generated using NVIDIA Isaac Sim and the PhysX physics engine. It is designed to train and evaluate AI models on physical reasoning, rigid body dynamics, optical flow, depth estimation, and scene understanding.
Each clip… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes.PhysicalAI-WorldModel-Synthetic-Warehouse-Operations-Scenes
PhysicalAI SDG-Warehouse
PhysicalAI SDG-Warehouse is a synthetic, fully-annotated video dataset of staged industrial-safety events captured in a simulated warehouse environment. It contains approximately 123k video clips, totaling roughly 412 hours of footage at 1920x1080 resolution and 30 frames per second, organized across four scenarios: a forklift near-miss with a human worker, a warehouse fire with worker evacuation, a forklift collision with a storage shelf, and a routine… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Warehouse-Operations-Scenes.PhysicalAI-Autonomous-Vehicles-NuRec
task_categories:
- robotics
tags:
- physicalAI
Find the 1500+ scenes in the sample_set/26.04_release folder.
Dataset Description:
Neural reconstructed dataset that carries 3D reconstructed driving scenes. The scenes are about 20 second long and stored in form of usdz files, along with respective xodr map files, surface mesh. The reconstructions were generated using 6 camera views (front-wide 120 deg, front-tele 30 deg, cross right/left 120 deg and rear right/left… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec.PhysicalAI-WorldModel-Synthetic-Digital-Human-Scenes
Dataset Description:
The SDG-SynHuman is a large-scale synthetic video dataset of digital humans rendered in diverse indoor and outdoor 3D environments. The dataset contains 236,937 clips, totaling approximately 5,841 hours of video, and is designed to support training and post-training of NVIDIA Cosmos world foundation models and related physical AI research.
Each sample is a temporally coherent 60-120 second video clip rendered at 1080p and 30 fps. Clips contain… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Digital-Human-Scenes.PhysicalAI-Robotics-GR00T-Teleop-Sim
Simulation GR1 Tabletop Task 1K Dataset
Dataset Description:
The PhysicalAI-Robotics-GR00T-Teleop-GR1 dataset consists of 1000 teleoperation trajectories in simulation using the GR1 robot with upper body control. The simulation setup mimics tabletop manipulation tasks and uses RGB observations with a virtual camera. The robot is equipped with simulated Fourier hands.
This dataset is ready for non-commercial use.
Dataset Owner(s):
NVIDIA GEAR… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-Sim.PhysicalAI-Robotics-Manipulation-SingleArm
Dataset Description:
PhysicalAI-Robotics-Manipulation-SingeArm is a collection of datasets of automatic generated motions of a Franka Panda robot performing operations such as block stacking, opening cabinets and drawers. The dataset was generated in IsaacSim leveraging task and motion planning algorithms to find solutions to the tasks automatically [1, 3]. The environments are table-top scenes where the object layouts and asset textures are procedurally generated [2].This dataset… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Manipulation-SingleArm.PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams
PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams
Paper | Paper Website | GitHub
Download
We provide a download script to download our dataset. If you have enough space, you can use git to download a dataset from huggingface.
usage: download.py [-h] --odir ODIR
[--file_types {hdmap,lidar,synthetic}[,…]]
[--workers N] [--clean_cache]
required arguments:
--odir ODIR Output… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams.PhysicalAI-Robotics-GR00T-Teleop-GR1
Introduction
TL;DR: DreamDojo is a generalist robot world model pretrained on 44k hours of human egocentric data, showing unprecedented generalization to diverse objects and environments.
Project page: https://dreamdojo-world.github.io/
Paper: https://arxiv.org/abs/2602.06949
Code: https://github.com/NVIDIA/DreamDojo
How to Use
Check out https://github.com/NVIDIA/DreamDojo
Citation
@article{gao2026dreamdojo,
title={DreamDojo: A Generalist Robot… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-GR1.Physical-AI-AV-US
PhysicalAI-AV-SFT
Supervised fine-tuning (SFT) dataset for an autonomous-vehicle vision-language
waypoint-prediction model. Contains 2,789,773 samples from 150 000
driving scenes (18 seconds per scene, sampled at 1 Hz) recorded in the
United States.
Format
WebDataset — 100 uncompressed .tar shards,
each containing pairs of files per sample:
Entry
Description
{key}.png
Front-facing wide-angle camera frame (640 × 360 px)
{key}.json
Metadata (see schema below)… See the full description on the dataset page: https://huggingface.co/datasets/tom-jerry-123/Physical-AI-AV-US.PhysicalAI-SimReady-Warehouse-01
NVIDIA Physical AI SimReady Warehouse OpenUSD Dataset
Dataset Version: 1.1.0
Date: May 18, 2025
Author: NVIDIA, Corporation
License: CC-BY-4.0 (Creative Commons Attribution 4.0 International)
Contents
This dataset includes the following:
This README file
A CSV catalog that enumerates all of the OpenUSD assets that are part of this dataset including a sub-folder of images that showcase each 3D asset (physical_ai_simready_warehouse_01.csv). The CSV file is… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-SimReady-Warehouse-01.PhysicalAI-WorldModel-Synthetic-Embodied-Robot-Scenes
PhysicalAI WorldModel Synthetic Embodied Robot Scenes Dataset Card
Dataset Description
PhysicalAI WorldModel Synthetic Embodied Robot Scenes is a large-scale synthetic robotics video corpus generated from USD-based robotic simulation and rendering pipelines built around NVIDIA Isaac Sim, Omniverse, Isaac Lab, and related robot data-generation systems. It is designed to improve physical plausibility, embodiment persistence, task-conditioned robot behavior reasoning… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Embodied-Robot-Scenes.PhysicalAI-Robotics-Manipulation-Objects-Kitchen-MJCF
Manipulation Objects Kitchen MJCF
We provide a collection of digital 3D mjcf assets intended for use in a simulated kitchen environment.
The assets are broadly divided into 2 categories: fixtures and objects. The fixture assets are comprised of interactable kitchen appliances such as stoves, microwaves, and ovens. The object assets consist of common kitchen objects such as saucepans and glass cups.
Preview
Objects
Fixtures
Objects… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Manipulation-Objects-Kitchen-MJCF.PhysicalAI-Autonomous-Vehicles-NCore
PhysicalAI Autonomous Vehicles - NCore
A subset of ~1.1k clips from the PhysicalAI-Autonomous-Vehicles (PAI-AV)
dataset, converted to NCore format
using the PAI data converter.
The subset contains clips that expose accurate offline calibration,
egomotion, and cuboid labels.
Source Dataset
PhysicalAI-Autonomous-Vehicles
contains 306,152 clips (1,700 hours) of multi-sensor driving data
collected across 25 countries. See the source dataset card for full details
on… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NCore.PhysicalAI-Robotics-NuRec
Dataset Description
The Physical AI NuRec dataset seeks to empower robotic researchers to build the next generation of physical AI based end-to-end robotic models.
This dataset includes various 3DGUT in USD files that can be loaded in Isaac Sim. Some datasets also include a mesh and occupancy map. The Mesh components are used for collision detection while the 3DGUT components provide realistic rendering. The asset can also be used with Isaac Sim Extensions like MobilityGen for… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec.physical-ai-bench-generation
Physical AI Bench - Generation
Paper | Code
Dataset Description
The PAI-Bench is a benchmark to measure the progress of world models quantitatively.
The predict task contains a list of 1044 samples of text prompts, conditioning images, and qa pairs, covering Physical AI target domains including autonomous vehicle (AV) driving, robotics, industry (smart space), physics, human, and common sense. All the questions are binary questions, and the answer is either Yes or No. Our… See the full description on the dataset page: https://huggingface.co/datasets/shi-labs/physical-ai-bench-generation.PhysicalAI-SpatialIntelligence-Lyra-SDG
Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation
Paper, Project Page, Code
Sherwin Bahmani,
Tianchang Shen,
Jiawei Ren,
Jiahui Huang,
Yifeng Jiang,
Haithem Turki,
Andrea Tagliasacchi,
David B. Lindell,
Zan Gojcic,
Sanja Fidler,
Huan Ling,
Jun Gao,
Xuanchi Ren
Dataset Description:
The PhysicalAI-SpatialIntelligence-Lyra-SDG Dataset is a multi-view 3D and 4D dataset generated using GEN3C.
The 3D reconstruction setup uses 59,031 images… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-SpatialIntelligence-Lyra-SDG.PhysicalAI-Robotics-Manipulation-Augmented
Dataset Description:
This is a fully annotated, synthetically generated dataset consisting of 1,000 demonstrations of a single Franka Panda robot arm performing a fixed-order three-cube stacking task in Isaac Lab. The robot consistently stacks cubes in the order: blue (bottom) → red (middle) → green (top).
The dataset was produced using the following pipeline:
Collected 10 human teleoperation demonstrations of the stacking task.
Used Isaac Lab’s Mimic tool [1] to simulate 1,000… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Manipulation-Augmented.PhysicalAI-VANTAGE-Bench
VANTAGE-BENCH
Video ANalysis Tasks Across Generalized Environments
Paper: VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
Dataset Description
VANTAGE-BENCH is the first public benchmark purpose-built for evaluating visual understanding on video captured by fixed infrastructure cameras. It spans three real-world domains — warehouse, smart city / Intelligent Transportation Systems (ITS), and smart spaces — across six spatio-temporal… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-VANTAGE-Bench.PhysicalAI-SimReady-Assets
PhysicalAI SimReady Homes — Supplement
Shared objects, materials and HDRIs for the SimReady Homes scenes — 6.9 GB, published once so no
scene has to carry its own copy.
This is the supplement required by
PhysicalAI SimReady Homes —
those scenes reference this repo and will not resolve without it.
Everything here is SimReady, not merely a mesh: each object carries collision geometry, mass and
inertia, physics materials, semantic labels, and — where the object articulates —… See the full description on the dataset page: https://huggingface.co/datasets/imagineio/PhysicalAI-SimReady-Assets.PhysicalAI-Robotics-GR00T-Teleop-G1
Unitree G1 Fruits Pick and Place 1K Dataset
Dataset Description:
The PhysicalAI-Robotics-GR00T-Teleop-G1 dataset consists of1000 teleoperation trajectories of real robot data using Unitree G1, with upper body control. The robot chooses the correct fruit to pick and place on the plate according to the language prompt. A total of 4 fruits are used: Apple, Pear, Starfruit, Grape. The robot is equipped with the default realsense camera, and a pair of Unitree G1 Tri-fingers… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-G1.PhysicalAI-Traffic-Anomaly-Reasoning
News
🏆 [Sep 24, 2026] TAR paper accepted to the NeurIPS 2026 Evaluations and Datasets Track!
🎤 [Sep 8, 2026] TAR talk and poster presented at the 10th AI City Challenge Workshop @ ECCV 2026 in Malmö, Sweden.
📄 [Aug 17, 2026] Challenge paper released on arXiv: The 10th AI City Challenge.
📄 [Aug 10, 2026] TAR paper released on arXiv: From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning.
🔗 [Jun 10, 2026] Track 7 (FETV) and Track 8 (PSI… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Traffic-Anomaly-Reasoning.PhysicalAI-Event-Videos
PhysicalAI-Event-Videos
PhysicalAI-Event-Videos is a video-anomaly and event dataset for developing text-to-video anomaly-search and safety-event-understanding systems. Version 1.0 provides structured annotations for 1,612 parent-video records and 3,796 labeled chunks, with 71,023 captions and queries across person-attribute, general-caption, anomaly, and action search tasks.
The redistributable media payload contains 1,486 NVIDIA-generated parent videos and 2,939 packaged chunk… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos.PhysicalAI