charlesxu0124/functional-manipulation-benchmark

Functional Manipulation Benchmark

This robot learning dataset is a part of the paper "FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning". It includes 22,550 expert demonstration trajectories across different skills required to solve the Single-Object and Multi-Object Manipulation Tasks presented in the paper.

Link to paper: https://arxiv.org/abs/2401.08553

Link to website: https://functional-manipulation-benchmark.github.io

Dataset Structure

Each zip file contains a folder of trajectories. Each trajectory is saved as a .npy file. Each .npy file contains a dictionary with the following key-value pairs:

• —obs/side_1: a (N, 256, 256, 3) numpy array of RGB images from the side camera 1 saved in BGR format
• —obs/side_2: a (N, 256, 256, 3) numpy array of RGB images from the side camera 2 saved in BGR format
• —obs/wrist_1: a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 1 saved in BGR format
• —obs/wrist_2: a (N, 256, 256, 3) numpy array of RGB images from the wrist camera 2 saved in BGR format
• —obs/side_1_depth: a (N, 256, 256) numpy array of depth images from the side camera 1
• —obs/side_2_depth: a (N, 256, 256) numpy array of depth images from the side camera 2
• —obs/wrist_1_depth: a (N, 256, 256) numpy array of depth images from the wrist camera 1
• —obs/wrist_2_depth: a (N, 256, 256) numpy array of depth images from the wrist camera 2
• —obs/tcp_pose: a (N, 7) numpy array of the end effector pose in the robot's base frame (XYZ, Quaternion)
• —obs/tcp_vel: a (N, 6) numpy array of the end effector velocity in the robot's base frame (XYZ, RPY)
• —obs/tcp_force: a (N, 3) numpy array of the end-effector force in the robot's end-effector frame (XYZ)
• —obs/tcp_torque: a (N, 3) numpy array of the end-effector torque in the robot's end-effector frame (RPY)
• —obs/q: a (N, 7) numpy array of the joint positions
• —obs/dq: a (N, 7) numpy array of the joint velocities
• —obs/jacobian: a (N, 6, 7) numpy array of the robot jacobian
• —obs/gripper_pose: a (N, ) numpy array indicating the binary state of the gripper (0=open, 1=closed)
• —action: a (N, 7) numpy array of the commanded cartesian action (XYZ, RPY, gripper)
• —primitive: a (N, ) numpy array of strings indicating the primitive associated with the current timestep
• —object_id (Multi-Object only): a (N, ) numpy array of integers indicating the ID of the object being manipulated in the current trajectory
• —object_info (Single-Object only): a dictionary containing information of the object being manipulated in the current trajectory with the following keys-value pairs:
• —length: length of the object (S=Short, L=Long)
• —size: cross-sectional size of the object (S=Small, M=Medium, L=Large)
• —shape: shape ID of the object according to reference sheet
• —color: color ID of the object according to reference sheet
• —angle: initial pose of the object indicating how it should be grasped and reoriented (horizontal, vertical)
• —distractor: indicator for whether there are distractor objects (y=yes, n=no)

File Naming

The Single-Object Dataset trajectory files are named as follows:

(insertonly){shape}{size}{length}{color}{angle}{distractor}{trajectory_id}.npy

The Multi-Object Dataset trajectory files are named as follows:

trajectory{objectid}{trajectoryid}.npy