allenai/molmo-motion-1m

MolmoMotion-1M

MolmoMotion-1M is a dataset of 3D point-trajectory annotations curated across seven video corpora — ego-centric manipulation, real-world robot teleoperation, dynamic real-world scenes, and simulator renders. Each clip ships motion-filtered 3D tracks (and, for most datasets, 2D pixel tracks), a short action caption, per-frame camera, and a train/test split — all frame-aligned to the source video.

Datasets

We do not re-host the original videos, and for some datasets do not redistribute the upstream coordinates — we ship the annotations/tracks we produced, plus a reconstruct_*.py script to regenerate the rest from the original source. Check each upstream dataset's license before downloading.

Each dataset's README.md is authoritative for its schema and its exact reconstruction procedure.

Download

huggingface-cli download allenai/molmo-motion-1m \
    --repo-type dataset --local-dir molmo-motion-1m

Unpack

The per-clip directories (tracks/, camera/, and molmospaces videos/ + robot_trajectories/) ship as ~10 GB tar shards inside those folders. Extract each dataset's shards from its folder:

cd molmo-motion-1m
for d in */; do (cd "$d" && for t in */*.tar; do tar -xf "$t" && rm "$t"; done); done   # extract, then delete each shard

→ each shard extracts into its own tracks/ / camera/ / videos/ (stereo4d has no shards). The resulting layout is the tree below.

molmo-motion-1m/
├── README.md
└── <dataset>/
    ├── README.md          # schema + reconstruction for this dataset
    ├── annotations/       # clips / split / index JSONs — the source of truth (loose)
    ├── tracks/            # tracks-*.tar shards → extract here to 3D + 2D NPZs  (track_index/ for stereo4d)
    ├── camera/            # camera-*.tar shards → per-frame pose + intrinsics (absent for xperience)
    ├── videos/            # videos-*.tar shards → molmospaces only; others reconstruct locally
    └── reconstruct_*.py   # regenerate videos (and, per dataset, tracks/camera/depth)

Data format

<dataset>/annotations/ holds the JSONs — the single source of truth; don't enumerate the tracks/ directory directly.

• —<dataset>_clips.json — one entry per clip (schema below)
• —<dataset>_split.json — {train, test, split_info}, same entry schema
• —<dataset>_videos_index.json — video_id → video path + reconstruction recipe

A clips entry:

{
  "file":            "<video_id>",          // lookup key into tracks/, videos/, index
  "num_frames":      281,                    // T; matches the NPZ and the video
  "fps":             15,                      // per-clip; varies by dataset (stereo4d=30, ytvis=6/8/12, xperience=10) — read this field, don't assume 15
  "caption":         "...",                  // short action description
  "clips_by_object": {"object": [[1, 17]]},  // motion frame ranges, inclusive
  "num_clips_total": 18
}

clips_by_object keys vary by dataset (object, left_hand/right_hand, car_0, obj0, …) and are the moving subset of the objects in the track file.

Tracks (after unpacking the shards) — most datasets store, per video_id, a {file}_3d.npz with points_3d (K, T, 3) + visibility (K, T) and a {file}_2d.npz with tracks (T, K, 2) + visibility (T, K) + dim. Note the 3D/2D axis order differs ((K, T, …) vs (T, K, …)). 3D is a metric world frame. The on-disk schema varies by dataset — see the per-dataset README before loading (egodex/ytvis/ hdepic/molmospaces store object-keyed dict NPZs; droid is flat (T, N, …); xperience bundles 2D+3D in one file; stereo4d ships only a track index).

License

MolmoMotion-1M is a collection of data that is organized into multiple subsets, and each subset is licensed separately as shown below. The data includes annotations, indices, and/or captions we produced using videos from multiple sources, so the subsets are organized by source. Each subset contains the respective data created by Ai2 that correspond to videos from that source.

MolmoMotion-1M, including all data subsets, are intended for research and educational use in accordance with Ai2’s Responsible Use Guidelines.

Disclaimer

MolmoMotion-1M includes annotations, indices, and captions created by Ai2 using videos that are not shared in this dataset. The videos are provided by the owners of the source datasets listed above, and remain subject to their respective license terms and use restrictions. Users who access videos from these sources to reconstruct any portion of the MomoMotion training data are responsible for reviewing and confirming that their intended use complies with the terms and conditions. Please navigate to the source datasets linked above to review the governing terms.

Acknowledgement

We thank Ropedia for providing videos from the Xperience dataset that were used in our research and development of MolmoMotion. The Xperience dataset is governed by Ropedia’s terms and conditions.

Citation

@misc{zhang2026molmomotionforecastingpointtrajectories,
    title={MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction}, 
    author={Jianing Zhang and Chenhao Zheng and Yajun Yang and Max Argus and Rustin Soraki and Winson Han and Taira Anderson and Chun-Liang Li and Shuo Liu and Jiafei Duan and Zhongzheng Ren and Jieyu Zhang and Ranjay Krishna},
    year={2026},
    eprint={2606.18558},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2606.18558}, 
}