Silicon23/droid_pipeline_visualization

DROID 3D Bounding-Box Pipeline — Visualization Data

Per-frame fused 3-camera pointclouds backing the web viewer at <https://silicon23.github.io/droidpipelinevisualization/>.

Each episode comes from the DROID v1.0.1 dataset and completed the full detection pipeline (FoundationStereo depth -> optimized extrinsics -> SAM3 masks -> SAM3D box -> FoundationPose video tracking). Point positions are in the Franka base (world) frame, metres, Z up.

Layout

clouds/{episode_id}/cloud_{preset}.bin     concatenated per-frame gzip blocks
clouds/{episode_id}/index_{preset}.json    byte offsets + per-frame metadata

Presets trade file size against density:

Block format

Blocks are listed in index_{preset}.json as {t, o, c, n} — source frame index, byte offset, compressed length, point count. Fetch one with an HTTP Range request and gunzip it. The inflated block is:

Dequantize with xyz = lo + (q + 32768) / 65535 * (hi - lo).

index_{preset}.json also carries frames[] with the per-frame FoundationPose status, consensus_size, and the 8-corner bbox_world.

Reading a frame in Python

import gzip, json, numpy as np, requests

BASE = "https://huggingface.co/datasets/Silicon23/droid_pipeline_visualization/resolve/main"
eid, preset, frame = "shard01010_ep008", "medium", 40

ix = requests.get(f"{BASE}/clouds/{eid}/index_{preset}.json").json()
b = ix["blocks"][frame]
raw = requests.get(f"{BASE}/clouds/{eid}/{ix['bin']}",
                   headers={"Range": f"bytes={b['o']}-{b['o']+b['c']-1}"}).content
buf = gzip.decompress(raw)

lo = np.frombuffer(buf, np.float32, 3, 0)
hi = np.frombuffer(buf, np.float32, 3, 12)
n = b["n"]
q = np.frombuffer(buf, np.int16, n * 3, 24).reshape(n, 3).astype(np.float32)
rgb = np.frombuffer(buf, np.uint8, n * 3, 24 + n * 6).reshape(n, 3)
xyz = lo + (q + 32768) / 65535 * (hi - lo)

Notes

• —Depth is cropped at 2.5 m to keep the robot workspace and drop far-wall clutter.
• —Wrist-camera pose is per-frame from trajectory.h5 (Euler XYZ, camera-to-world directly). Episodes flagged wrist_sensor_flipped use the right lens composed with a 180 deg rotation about Z.
• —Only the left eye of each side-by-side stereo recording is used.