datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios
Dataset Description:
PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios is a large-scale synthetic video dataset of autonomous-driving scenes generated with NVIDIA's internal Omniverse simulation platform. Each clip is a temporally consistent multi-camera surround capture of one ego vehicle and surrounding traffic participants, paired with per-camera VLM captions. The dataset is designed to fill gaps in real-world driving data along two axes: (1) targeted long-tail… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios.PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes
PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes Dataset Card
Dataset Description
PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes is a large-scale synthetic dataset of physically-simulated multi-object interaction scenes, generated using NVIDIA Isaac Sim and the PhysX physics engine. It is designed to train and evaluate AI models on physical reasoning, rigid body dynamics, optical flow, depth estimation, and scene understanding.
Each clip… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes.PhysicalAI-WorldModel-Synthetic-Warehouse-Operations-Scenes
PhysicalAI SDG-Warehouse
PhysicalAI SDG-Warehouse is a synthetic, fully-annotated video dataset of staged industrial-safety events captured in a simulated warehouse environment. It contains approximately 123k video clips, totaling roughly 412 hours of footage at 1920x1080 resolution and 30 frames per second, organized across four scenarios: a forklift near-miss with a human worker, a warehouse fire with worker evacuation, a forklift collision with a storage shelf, and a routine… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Warehouse-Operations-Scenes.PhysicalAI-WorldModel-Synthetic-Digital-Human-Scenes
Dataset Description:
The SDG-SynHuman is a large-scale synthetic video dataset of digital humans rendered in diverse indoor and outdoor 3D environments. The dataset contains 236,937 clips, totaling approximately 5,841 hours of video, and is designed to support training and post-training of NVIDIA Cosmos world foundation models and related physical AI research.
Each sample is a temporally coherent 60-120 second video clip rendered at 1080p and 30 fps. Clips contain… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Digital-Human-Scenes.world_model_datasetaction-worldmodel-benchcode-world-model-project-page-videos
Code World Model Project Page Videos
Public research-demo video assets used by the Code World Model project page.
The gallery/ directory contains aligned RGB and proxy videos for interactive comparison.
PhysicalAI-WorldModel-Synthetic-Embodied-Robot-Scenes
PhysicalAI WorldModel Synthetic Embodied Robot Scenes Dataset Card
Dataset Description
PhysicalAI WorldModel Synthetic Embodied Robot Scenes is a large-scale synthetic robotics video corpus generated from USD-based robotic simulation and rendering pipelines built around NVIDIA Isaac Sim, Omniverse, Isaac Lab, and related robot data-generation systems. It is designed to improve physical plausibility, embodiment persistence, task-conditioned robot behavior reasoning… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Embodied-Robot-Scenes.financial-world-model
Twelve Data World Model Dataset
A multi-modal financial time-series dataset built from Twelve Data
market data. Each timeframe is published in three parallel views:
bars_* — OHLCV bars enriched with causal technical indicators and macro
context, in Parquet.
text_* — instruction-tuning prompts/labels derived from the bars, in
JSONL.
trajectories_* — fixed-length rolling windows of state vectors plus
next-state pairs, suitable for world-model / sequence-model training, in… See the full description on the dataset page: https://huggingface.co/datasets/twelvedata/financial-world-model.world-model-physics-human-preference-283k
Rapidata Physics Benchmark
Built by Rapidata.
Do video and world models understand physics? We gave 25 video- and world models the same
real-world starting frame and scene description from Physics-IQ and asked
them to predict what happens next. ~283,000 human votes, collected with the
Rapidata Python SDK, decided which continuation is more realistic — with the
real recording competing as a hidden 26th participant.
Each row is a head-to-head matchup between two participants on… See the full description on the dataset page: https://huggingface.co/datasets/Rapidata/world-model-physics-human-preference-283k.code-world-model-inference-examples-40
Inference examples
This directory contains 40 numbered, independent inference examples.
Every example uses only its public number; source case names and internal paths
are intentionally omitted.
Each numbered directory contains:
first_frame.png: exact 1536x864 generated RGB first frame used by inference.
prompts/*.txt: the exact rolling long-inference prompts used for the result.
condition/*.npz: ordered lossless condition chunks.
metadata.json: frame count, FPS, prompt windows… See the full description on the dataset page: https://huggingface.co/datasets/NTU-yiwen/code-world-model-inference-examples-40.world_model_tokenized_data
1X World Model Compression Challenge Dataset
This repository hosts the dataset for the 1X World Model Compression Challenge.
huggingface-cli download 1x-technologies/worldmodel --repo-type dataset --local-dir data
Updates Since v1.1
Train/Val v2.0 (~100 hours), replacing v1.1
Test v2.0 dataset for the Compression Challenge
Faces blurred for privacy
New raw video dataset (CC-BY-NC-SA 4.0) at worldmodel_raw_data
Example scripts now split into:
cosmos_video_decoder.py —… See the full description on the dataset page: https://huggingface.co/datasets/1x-technologies/world_model_tokenized_data.phyground
PhyGround: Benchmarking Physical Reasoning in Generative World Models
Project page ·
Paper ·
Evaluation code ·
PhyJudge-9B
PhyGround is a criteria-grounded benchmark for diagnosing physical failures in
generated video. It contains 250 prompts covering 13 observable physical
laws across solid-body mechanics, fluid dynamics, and optics. Each prompt is
paired with a first-frame image, 10 released generation configurations, and
applicable-law labels.
The Hub repository includes:… See the full description on the dataset page: https://huggingface.co/datasets/NU-World-Model-Embodied-AI/phyground.action-world-model-atlas-1500-media-20260914
Action World Model Atlas
Public browsing previews for 1,500 unique action clips from the completed
6,033-video bundle. OpenPixel2Play, Gaming 500 Hours, and Xiaoluo each contribute
500 examples. All 46 games in the completed bundle are represented.
Videos preserve the full five-second duration and 81 frames. They are existing
browser previews and can be smaller than the native training videos. Video and
poster checksums are verified against the source media manifests.… See the full description on the dataset page: https://huggingface.co/datasets/mikusama99/action-world-model-atlas-1500-media-20260914.GNN_Disassembly_WorldModel
GNN Constraint-Aware World Model Dataset (v3)
Real robot episodes with per-frame constraint graphs, SAM2 segmentation masks + 256-D feature embeddings, full 3D depth bundles, and synchronized robot states across two manipulation domains. Both domains share the v3 on-disk layout (same JSON/NPZ schemas, same delta-encoded frame_states, same fully-connected PyG expansion at load time) and now share a unified 270-D node feature format — the PyG loader reads a fixed 10-D type… See the full description on the dataset page: https://huggingface.co/datasets/ChangChrisLiu/GNN_Disassembly_WorldModel.showui-worldmodel-results
ShowUI + WorldModel: Training Results & Artifacts
This dataset contains evaluation results, world models, and training data from the ShowUI + WorldModel integration project.
📦 Contents
1. Evaluation Results (results/miniwob_predictions/)
Size: ~460MB
Format: JSONL files with episode-level predictions
Tasks: 9 MiniWoB++ tasks evaluated with ShowUI agent
Includes:
Task success/failure outcomes
Action predictions and execution traces
World model… See the full description on the dataset page: https://huggingface.co/datasets/zhongweixie/showui-worldmodel-results.WorldModelForMaze
WorldModelForMaze
Code, datasets, and trained checkpoints for studying world-model representations in maze navigation, based on a modified NanoGPT.
Contents
*.py — training, testing, probing, and visualization scripts (see readme.md).
model/ — architectures: transformer, transformer-rope, transformer-nextlat, mamba, mamba2, gated-deltanet, gru.
data/maze/100/ — tokenized maze datasets for Tasks A/C/E/H/I (RWs paths, 100 nodes).
out/ — final (10000-iter)… See the full description on the dataset page: https://huggingface.co/datasets/Kalso42/WorldModelForMaze.world_model_10s_v1
World Model 10s v1
500 scene conditioning packages for 10-second world-model / image-to-video experiments, sampled at 24 FPS (240 camera poses per case).
Directory
Cases
game_virtual_world
150
real_world_scene
200
robot_embodied
150
Use report.json as the index. Each case's output_dir is relative to the dataset root and contains an input image, numbered image, scene caption, image-to-video prompt, task JSON, camera-to-world trajectory (camera_c2w.npy)… See the full description on the dataset page: https://huggingface.co/datasets/aoaoder/world_model_10s_v1.world_model_raw_dataRaw Dataset for the 1X World Model Sammpling Challenge.
Download with:
huggingface-cli download 1x-technologies/worldmodel_raw_data --repo-type dataset --local-dir data
Train/Val v2.0
The training dataset is shareded into 100 independent shards. The definitions are as follows:
video_{shard}.mp4: Raw video with a resolution of 512x512.
segment_idx_{shard}.bin - Maps each frame i to its corresponding segment index. You may want to use this to separate non-contiguous frames from… See the full description on the dataset page: https://huggingface.co/datasets/1x-technologies/world_model_raw_data.Mobile-GUI-Worldmodel-SFT
Mobile-GUI-Worldmodel-SFT
This repository contains mobile GUI agent data and auxiliary files for training and evaluating GUI world models. The data is organized around GUI trajectories: each step has a screenshot and page-state annotations such as HTML, plain text, and structured text.
Repository Layout
.
├── GUI-agent-main/ # Data annotation scripts and examples
├── eval/ # Evaluation assets
│ └── AndroidControl_images.tar.gz… See the full description on the dataset page: https://huggingface.co/datasets/xwk123/Mobile-GUI-Worldmodel-SFT.h-jepa
H-JEPA datasets
Data for the H-JEPA code release (hierarchical JEPA world models). Image columns are stored with
lossless Blosc-Zstd compression inside the HDF5 files; h5py with hdf5plugin reads them directly. The repository mirrors
$HJEPA_HOME, the data root the code reads from, so a download with --local-dir $HJEPA_HOME
puts every file where the code expects it.
FourRoom Distractors
The four-room navigation environment with one distractor agent that wanders… See the full description on the dataset page: https://huggingface.co/datasets/jepa-world-models/h-jepa.bytesized32-world-model-cotSee https://github.com/thuml/RLVR-World for examples for using this dataset.
Citation
@article{wu2025rlvr,
title={RLVR-World: Training World Models with Reinforcement Learning},
author={Jialong Wu and Shaofeng Yin and Ningya Feng and Mingsheng Long},
journal={arXiv preprint arXiv:2505.13934},
year={2025},
}
World-Model
🌍 World Model Bench (WM Bench) v1.0
Beyond FID — Measuring Intelligence, Not Just Motion
WM Bench is the world's first benchmark for evaluating the cognitive capabilities of World Models and Embodied AI systems.
🎯 Why WM Bench?
Existing world model evaluations focus on:
FID / FVD — image and video quality ("Does it look real?")
Atari scores — performance in fixed game environments
WM Bench measures something different: Does the model think correctly?… See the full description on the dataset page: https://huggingface.co/datasets/FINAL-Bench/World-Model.world-model-arena-releases
World Model Arena
Versioned public scenario contracts and precomputed generative rollouts. Submission is through Dataset pull requests; no adapter is required. Inputs are synthetic CC0 scenes. Model outputs retain their declared licenses. Ballots and allowance counters are private. Generation source is private.
The new photographic pack is worlds-realistic-v2, with pinned LTX, Cosmos 3 Edge and Cosmos 3 Nano configurations. Historical illustrated inputs and merged submissions… See the full description on the dataset page: https://huggingface.co/datasets/suvadityamuk/world-model-arena-releases.asl-trunk-world-model
ASL soft trunk robot: world-model dataset and training package
Research data of the Stanford ASL trunk robot project (recorded 2026-09-10/11). Code: https://github.com/mellowyellow71/trunk-stack (branch world-model, a fork of StanfordASL/trunk-stack). Everything a new person needs to reproduce or continue the work is in this repository plus that branch; no access to the original machines is required.
What is here
path
content… See the full description on the dataset page: https://huggingface.co/datasets/melloyello/asl-trunk-world-model.world_model_corpus
Dataset Card for World Model Corpus
Paper | GitHub
The world model corpus contains a set of generated trajectories that are shaped for text-based world modeling task as used by the paper: "Masked Diffusion Language Models are Strong and
Steerable Text-Based World Models for Agentic RL". The dataset contains trajectories from nine distinct environments: Tau2Bench, SWE-Smith, DeepresearchQA, Openresearcher, Gorilla/BFCLv4, Webshop, Toolathlon, Pandora and Coderforge.… See the full description on the dataset page: https://huggingface.co/datasets/PatronusAI/world_model_corpus.Syn4D_worldmodel_benchmarkmenti-bench
Menti-Bench
Menti-Bench is a manually constructed, quality-controlled benchmark of situated decision scenarios for evaluating Mental World Modeling (MWM): whether a model can predict what a target agent will actually do next, in scenes where the correct prediction depends on tracking each agent's beliefs, knowledge access, goals, emotions, and social constraints rather than the physical scene alone.
Each instance presents a short story (text, an image sequence, or a sounding… See the full description on the dataset page: https://huggingface.co/datasets/mental-world-model/menti-bench.world_model_for_wa_desc_with_tao_dataset
Dataset Card for "world_model_for_wa_desc_with_tao_dataset"
More Information needed
worldmodelbench