MIT-Media-Lab/oakink2-vitra-streaming-v1

OakInk2 → VITRA Stage-1 (complete audited release)

This repository contains all 627 physical OakInk2-TaMF sequences converted to VITRA Stage-1. Every sequence source pair is pinned to kelvin34501/OakInk-v2 revision 21705616140d726607027e70d58b7837f442ffd8, aligned by exact frame identity, converted across the four calibrated views, checked by geometry and every-frame RGB audits, smoke-tested through the VITRA loader, uploaded, and verified at an immutable commit before local cleanup.

Files and VITRA loader contract

RGB episodes are under Video/OakInk2_root/. Labels are NumPy dictionaries at Annotation/oakink2_<split>/episodic_annotations/<episode_id>.npy. Each label contains the exact video_name, video_decode_frame, source frame IDs and timestamps, calibrated intrinsics, per-frame world_to_camera extrinsics, world/camera 21-joint arrays, MANO beta/rotation-matrix pose/translation fields, object trajectories, causal task text, and training_anchor_valid.

The final Annotation/oakink2_<split>/episode_frame_index.npz contains index_frame_pair and index_to_episode_id. Column 0 of each pair selects an episode ID; column 1 selects a label row. Only rows whose training_anchor_valid value is true are indexed. A minimal lookup is:

import numpy as np

with np.load(index_path, allow_pickle=True) as index:
    episode_slot, frame_id = index["index_frame_pair"][sample_id]
    episode_id = index["index_to_episode_id"][episode_slot]
episode = np.load(
    label_dir / (episode_id + ".npy"), allow_pickle=True
).item()
rgb_frame_id = int(episode["video_decode_frame"][frame_id])
assert bool(episode["training_anchor_valid"][frame_id])

Before the final commit, the release gate independently re-reads all 2504 accepted camera reports and calibrates wrist/joint speed distributions over all 627 physical sequences. Publication is blocked if a maximum exceeds both its conservative absolute floor and ten robust deviations in log1p space. This release has 0 blocked temporal sequences; the complete evidence is Audit/temporal_quality.json.

The legacy RGB audit helper only distinguished train/test. Before final publication, all 39 official-val RGB summaries and every per-frame JSONL record are therefore normalized to val, their sequence-manifest hashes are updated, and every changed remote byte is downloaded and verified. This correction does not alter videos, MANO/keypoint labels, action chunks, or training indices.

The official OakInk2-TaMF split is pinned at commit 4cb9c461b92c032715374bd252c7179c70a14bde. Both keypoint and MANO-angle normalization files are exact merges of the per-sequence float64 accumulators. Global frame indices are generated only after all 627 sequences are CLEANED.

Conversion used 337 serial and 290 parallel-orchestrated physical sequences. The controller recorded 293 atomic claim events with 0 invalid overlapping claim lifecycles. Its exact source and audit report are preserved under Audit/orchestration/.

Build/final_manifest.json hashes every release-level artifact, and Build/release_ledger.sqlite3 records the exact source files, sequence manifests, camera episodes, anchor masks, and partial statistics. The fresh independent audit is published at Audit/independent_release_verification.json.

OakInk2 is human hand-object motion rather than robot control. Direct robot policy training still requires an explicit embodiment/retargeting decision. Source data and this derivative are CC BY-SA 4.0; MANO assets are not included and retain their separate license.