Datasetpublic

CaryxAI/everyday-manipulation-3d

everyday-manipulation-3d Browse the episodes in your browser → · caryx.ai · founders@caryx.ai Every frame carries metric depth, 6-DoF camera pose, MANO hand pose, per-hand object contact, hand and object segmentation, and time-aligned language, so a model can be tested against any one of those channels or all of them at once without collecting or labelling anything first. Recorded on iPhone Pro (ARKit LiDAR) as part of Everyday Manipulation 1 (EM1), annotated by CaryX AI… See the full description on the dataset page: https://huggingface.co/datasets/CaryxAI/everyday-manipulation-3d.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

2likes458downloads

Dataset Card

everyday-manipulation-3d

[Browse the episodes in your browser →](https://caryx.ai/data) · caryx.ai · founders@caryx.ai

Every frame carries metric depth, 6-DoF camera pose, MANO hand pose, per-hand object contact, hand and object segmentation, and time-aligned language, so a model can be tested against any one of those channels or all of them at once without collecting or labelling anything first.

Recorded on iPhone Pro (ARKit LiDAR) as part of Everyday Manipulation 1 (EM1), annotated by CaryX AI, packaged as a single multi-episode LeRobot v3.0 dataset. This is a research corpus (116 episodes, 10 tasks, 5 participants): built for method development and per-channel evaluation, not for broad generalization claims.

116 episodes · 28019 frames @ 15 fps · release v0.7

task	episodes	frames
`Wash the spoon with the sponge and rinse it clean.`	1	565
`fold_cloth`	1	455
`fold_laundry`	12	3546
`jar_open_close`	5	814
`pick_and_place`	71	14999
`pour_liquid`	1	323
`scoop_grain`	11	4488
`sweep`	12	2317
`tie_shoelaces`	1	318
`unplug_charger`	1	194

Quick start

Requires Python 3.12+ and lerobot==0.6.0 (the version this dataset was written and verified with).

python

# pip install "lerobot==0.6.0"
from lerobot.datasets.lerobot_dataset import LeRobotDataset

# depth_output_unit: the loader default is MILLIMETRES; pass "m" for metres.
ds = LeRobotDataset("CaryxAI/everyday-manipulation-3d", depth_output_unit="m")
frame = ds[0]  # RGB, metric depth, MANO hands, contact, camera pose in one dict

Storage shapes below are HWC (height, width, channel). The official loader returns image tensors as CHW floats in [0, 1], which is normal LeRobot behaviour.

License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC-BY-NC-SA-4.0), non-commercial. https://creativecommons.org/licenses/by-nc-sa/4.0/.

MANO-derived data (observation.mano_joints, observation.mano_params, and the HaMeR hand fields in the episode sidecars) is subject to the MANO license. Full terms: LICENSE.txt.

Episode orientation

Most episodes were recorded portrait; a few were recorded landscape and are shipped ROTATED 90° CCW onto the same portrait canvas so the dataset merges under one schema. Those episodes are flagged in meta/foundry.json episodes.<id>.raster_rotation = "ccw90": rotate their image channels 90° CW to display upright. All spatial channels (RGB, depth, masks, camera pose, MANO orientation) are expressed consistently in the shipped, rotated frame, so training and 3D geometry need no special-casing.

Features

Shapes are numpy shapes, so (7,) is a flat 7-element vector.

key	shape	notes
`observation.images.ego`	(512, 384, 3)	RGB ego frame, portrait, ONE uniform downscale of the native 1440×1920 (no aspect distortion). Exactly 2× the depth grid: `ego[y, x]` and `depth[y//2, x//2]` are the same ray.
`observation.images.depth`	(256, 192, 1) video	metric metres, native ARKit sceneDepth grid (never resampled), 12-bit log-HEVC (quantizer [0.1, 5.0] m in `meta/info.json`). Intrinsics for both grids: `episodes/<id>/depth_geometry.json`.
`observation.state`	(7,)	`[x, y, z]` wrist position + `[rx, ry, rz]` hand rotation (axis-angle of the hand root) + grasp, metric WORLD frame. The state tracks ONE hand per frame: the right hand when visible, else the left. Position and rotation are zero-phase smoothed (see `meta/foundry.json` `state_smoothed`); the wrist source is named per episode in `state_source`. A zero row is a placeholder, not a pose: check `state_valid`.
`observation.state_valid`	(1,)	1.0 = the state row is a real measured pose; 0.0 = no derivable hand pose on this frame (the row is zeros). Mask state losses with this flag. 9448 of 28019 frames are placeholders.
`observation.state_hand`	(1,)	which hand the state tracks this frame: 1.0 = right, 0.0 = left, -1.0 = none. The state follows one hand and can switch when the right hand is lost; the state jumps at a switch because it is a different hand.
`action`	(7,)	`[dx, dy, dz]` world position delta + `[drx, dry, drz]` axis-angle of the RELATIVE rotation between consecutive frames + grasp. An action is a real delta only when `state_valid` is 1.0 on BOTH frame t and t+1 AND `state_hand` is the same on both; a validity gap or a hand switch is a zero-delta boundary row. Mask action losses accordingly.
`observation.camera_pose`	(4, 4)	camera to world (metric), expressed in the SHIPPED raster frame (x = raster right, y = raster down, z = forward) and gravity-verified per episode at export. Composes directly with the intrinsics in `depth_geometry.json`.
`observation.contact`	(2, 2)	The shipped contact channel. Rows are the left and right hand, column k is object slot k. Values are contact strength in [0, 1]. Contact against any object is the max over slots.
`observation.contact_valid`	(2, 2)	1.0 where that hand-to-object distance was actually measured on that frame, 0.0 where it was not: hand out of frame, no depth surface, no object in that slot, or the slot is a drop target that is deliberately never measured (below). Single-object episodes ship slot 1 all-zero rather than a fabricated "measured apart".
`observation.mano_joints`	(2, 21, 3)	MANO/HaMeR joints RE-ROOTED at the wrist: joint 0 is exactly the origin, axes in the shipped raster's camera orientation (`mano_frame` in `meta/foundry.json`). World placement for the hand `state_hand` names, when both are valid: `world = (R_state @ R_go.T) @ joints + state[0:3]`, where `R_state` is the rotation matrix of `state[3:6]` and `R_go` of that hand's `mano_params[0:3]`; both are shipped per frame. The untracked hand has no shipped world anchor. The primary channel is tip-smoothed (named per episode in `mano_joints_channel`); the raw fit is recoverable from `observation.mano_params`.
`observation.mano_params`	(2, 58)	full MANO parameterization: global_orient(3) ⊕ pose(45) ⊕ betas(10)
`observation.mano_valid`	(2,)	1.0 where the MANO fit is valid for that hand/frame. 0.0 covers both cases: no hand detected, and fits removed by a hand-entry/exit quality gate (a hand barely in shot yields an unreliable extrapolated skeleton — we ship nothing rather than a guess). Mask MANO losses on 0.0 frames.
`observation.images.object_mask`	(512, 384, 3) video	R = object slot 0, B = slot 1 (two-object episodes), G unused. Threshold > 127 (video codecs are lossy). Exact full-res masks: `episodes/<id>/object_mask.npz`. On `pick_and_place` episodes slot 1 is the DESTINATION — the plate or bowl the item is placed into. It is segmented and tracked like any object, but hand contact against it is deliberately never measured: the hand hovers over a drop target rather than touching it, and 3D proximity would read that hover as touch. Its `contact_valid` column is 0.0 by design, and the contact sidecar names such slots in `measure_exempt_slots`.
`observation.images.hand_mask`	(512, 384, 3) video	R = LEFT hand, B = RIGHT hand. Same thresholding. Reviewer-corrected (shipped) channel. Exact full-res masks: `episodes/<id>/hand_mask.npz`.

Timing

The timestamp column is true source time: every 1/15 s tick of the source clip ships as one frame (nearest source frame; nothing is dropped), so timestamp in this dataset and the times in language_spans.json are the same clock. Frames with no derivable hand pose ship with a zero state and state_valid = 0.0 instead of being removed. Per episode, meta/foundry.json episodes.<id>.source_frame_indices gives the source video frame behind each tick and source_fps the source frame rate, so exact source-frame timing is recoverable.

Per-episode sidecars

Full-resolution annotation that does not fit the fixed frame schema rides alongside each episode in episodes/<capture_id>/.

file	present on	what it is
`openego_sidecar.json`	all	the release copy of the OpenEgo annotation. Internal production bookkeeping (review state, gate results, correction history, checkpoints, methodology notes) is removed; it is not the full internal record. What IS deliberately kept as consumer provenance: per-stream `annotator_params` reduced to `{producer_version, model}`, the transcript source tag, and the `hand_pose.source` / intrinsics source tags; these say which model produced each stream so a consumer can branch on provenance. Path fields refer only to files or features of THIS release (`observation.images.ego`, sibling npz) or are null; full-resolution sources are in the companion raw dataset.
`language_spans.json`	all	dense sub-step action labels, in seconds on the same clock as `timestamp`. Each span carries `source` (`vlm` or `human`, where `human` means a reviewer wrote or corrected that label). Use them for language-conditioned / sub-goal training by joining on `timestamp`; the LeRobot `task` column stays the static goal.
`depth_geometry.json`	all	the depth camera's own intrinsics and extrinsics
`object_mask.npz`	all	exact full-resolution object masks (the video channel is lossy)
`hand_mask.npz`	all	exact full-resolution hand masks, the reviewer-corrected shipped channel (left = slot 0, right = slot 1)
`hand_object_contact.npz`	all	full attributed contact: per hand and object distances, strengths, grasp states
`object_object_contact.npz`	multi-object (86 of 116)	symmetric object-to-object contact
`contact_corrections.npz`	where a reviewer ruled	reviewer contact-window rulings, kept separate from the measurement
`object_registry.json`	multi-object (86 of 116)	object id to human label and mask colour slot

Both contact files are keyed by the same object_ids as object_mask.npz. To resolve which object is slot k, read episodes.<capture_id>.contact_slot_object_ids and contact_slot_object_labels in meta/foundry.json. The same k indexes the object_mask video channel (R = slot 0, B = slot 1) and object_mask.npz object_ids[k].

Episodes and tasks

Tasks are native LeRobot tasks (meta/tasks.parquet); filter episodes by task_index. The episode index (id, task, length) is in meta/episodes/. The per-episode CaryX AI metadata lives in `meta/foundry.json` (orientation flags, state source, source frame indices, contact slot ids). The official LeRobot loader does not surface custom metadata; fetch that file directly.

What is exact and what is not

channel	fidelity
RGB	native 1440×1920 downscaled to 384×512, ONE uniform scale factor (no aspect distortion). The raw full-resolution clips are in the companion raw dataset below.
depth	native 256×192 grid, 12-bit log-quantized video over [0.1, 5.0] m. Quantization error is under ~0.5 mm p99 (max ~0.9 mm), far below LiDAR sensor noise. Readings outside the range saturate.
masks	video channels are codec-lossy at outlines (~0.03 to 0.04% of pixels); the exact masks ship as `object_mask.npz` and `hand_mask.npz` per episode.
time	15 fps time-based resampling of the source clip (nearest source frame per tick; worst-case timing jitter is half a source frame). No frames are dropped.
state	zero-phase smoothed (positions and rotations); the raw fit is recoverable from `observation.mano_params` and the sidecars.
stats	`meta/stats.json` is computed over every frame of the corpus, not a sample.

meta/checksums.json is the sha256 manifest of this release (it does not list itself).

Canonical split

meta/splits.json freezes an episode-level train/val/test split. The rule is deterministic: within each task, episodes are ordered by capture time (capture_id is time-sortable); the last episode of each task is test, the second-to-last is val, and the rest are train. This is a chronological partition by episode, not a held-out participant, scene or session: the same participant can appear in every split.

split	episodes	frames
train	106	25588
val	5	1168
test	5	1263

Report results on test; tune on val.

Provenance

Self-collected capture by CaryX AI, recorded by 5 consenting adults in private homes. Annotations are produced by CaryX AI's pipeline using third-party models and reviewed by humans.

Citation

bibtex

@misc{caryx2026egocentric,
  title        = {everyday-manipulation-3d: Egocentric Human Manipulation with Metric Depth, Contact, and MANO Hands},
  author       = {CaryX AI},
  year         = {2026},
  publisher    = {CaryX AI},
  howpublished = {\url{https://caryx.ai}},
  url          = {https://huggingface.co/datasets/CaryxAI/everyday-manipulation-3d},
  note         = {Dataset. Available at \url{https://caryx.ai/data}}
}

The raw full-resolution RGB-D clips these episodes were built from are published separately: CaryxAI/everyday-manipulation-3d-raw (CC BY 4.0, no annotations).

Produced by CaryX AI. Reach out: founders@caryx.ai · caryx.ai