Datasetpublic

tom-jerry-123/Physical-AI-AV-US

PhysicalAI-AV-SFT Supervised fine-tuning (SFT) dataset for an autonomous-vehicle vision-language waypoint-prediction model. Contains 2,789,773 samples from 150 000 driving scenes (18 seconds per scene, sampled at 1 Hz) recorded in the United States. Format WebDataset — 100 uncompressed .tar shards, each containing pairs of files per sample: Entry Description {key}.png Front-facing wide-angle camera frame (640 × 360 px) {key}.json Metadata (see schema… See the full description on the dataset page: https://huggingface.co/datasets/tom-jerry-123/Physical-AI-AV-US.

sourceHugging Faceotherupdated 7mo agoView on Hugging Face

0likes12kdownloads

Dataset Card

PhysicalAI-AV-SFT

Supervised fine-tuning (SFT) dataset for an autonomous-vehicle vision-language waypoint-prediction model. Contains 2,789,773 samples from ~150 000 driving scenes (~18 seconds per scene, sampled at 1 Hz) recorded in the United States.

Format

WebDataset — 100 uncompressed .tar shards, each containing pairs of files per sample:

Entry	Description
`{key}.png`	Front-facing wide-angle camera frame (640 × 360 px)
`{key}.json`	Metadata (see schema below)

Key format: {scene_id}__{sample_idx:02d} Shard assignment: sha256(scene_id) % 100 — all frames of a scene land in the same shard, preventing scene leakage across train/eval splits.

Metadata schema (`{key}.json`)

json

{
  "scene_id":           "UUID string — identifies the driving scene",
  "chunk_name":         "chunk_XXXX — source data chunk",
  "sample_idx":         "int 1–18 — which second within the 18-second scene",
  "global_idx":         "int — globally unique datum index",
  "target_t_rel_us":    "int — timestamp relative to scene start (microseconds)",
  "target_frame_index": "int — video frame index",
  "egomotion":          "list[list[float]] — past trajectory [[x,y,yaw], ...], 3 entries: [-2s, -1s, 0s (anchor)]",
  "waypoints":          "list[list[float]] — future trajectory [[x,y,yaw], ...], 4 entries at ~2s steps",
  "is_long_tail":       "bool — long-tail driving scenario flag"
}

Coordinate convention: all x/y/yaw in the ego-vehicle frame at target time, +x = forward, +y = left, yaw in radians CCW from forward.

Loading

python

import webdataset as wds, json
from PIL import Image
import io

# Local (after cloning the repo)
ds = wds.WebDataset("shards/train-{00000..00099}-of-00100.tar").shuffle(1000)

for sample in ds:
    img  = Image.open(io.BytesIO(sample["png"]))
    meta = json.loads(sample["json"])
    # meta["waypoints"] → [[x,y,yaw], ...] × 4 future steps

Shard index

index.parquet — one row per sample, columns: key, shard, scene_id, chunk_name, sample_idx, global_idx, target_t_rel_us, is_long_tail.