oscarqjh/EB-Manipulation_easi

EB-Manipulation Dataset

EB-Manipulation is a benchmark for evaluating LLM-controlled robotic manipulation in CoppeliaSim using a Franka Panda arm with a parallel gripper. It is part of the EmbodiedBench benchmark suite, designed for use with the EASI evaluation framework.

Dataset Description

Agents must output sequences of 7D discrete gripper actions [X, Y, Z, Roll, Pitch, Yaw, Gripper] to complete manipulation tasks (picking, stacking, placing, wiping). The benchmark tests spatial reasoning, visual understanding, common sense, and complex instruction following.

Subsets

Task Types

Action Space

Each action is a 7D discrete array: [X, Y, Z, Roll, Pitch, Yaw, Gripper_state]

• —X, Y, Z: 3D position in voxel grid (range [0, 100])
• —Roll, Pitch, Yaw: Discrete Euler angles (range [0, 120], each unit = 3 degrees)
• —Gripper state: 0 = close, 1 = open

Dataset Structure

.
├── data/
│   ├── base.jsonl
│   ├── common_sense.jsonl
│   ├── complex.jsonl
│   ├── spatial.jsonl
│   └── visual.jsonl
├── simulator_data.zip          # Binary simulation files (auto-extracted by EASI)
│   ├── data/                   # Per-split episode data (.ttm, .pkl)
│   ├── vlm/                    # Task templates and object models
│   └── amsolver/robot_ttms/    # Robot model files
└── README.md

Data Fields (JSONL)

Each row in the JSONL files contains:

• —id (int): Unique identifier within the split
• —task_name (string): Task variation name (e.g., pick_cube_shape)
• —variation (int): Variation number within the task
• —episode_num (int): Episode number within the variation
• —instruction (string): Natural language task instruction
• —task_type (string): Base task type (pick, stack, place, wipe)

Simulator Data (simulator_data.zip)

Each episode's binary data is stored at: data/{split}/eval/{task_name}/variation{N}/episodes/episode{N}/

• —task_base.ttm — CoppeliaSim scene state
• —waypoint_sets.ttm — Waypoint configuration
• —configs.pkl — Episode metadata and success conditions

Usage

Loading with Datasets Library

from datasets import load_dataset

# Load a specific split
dataset = load_dataset("oscarqjh/EB-Manipulation_easi", split="base")

# Access data
for example in dataset:
    print(example["instruction"])
    print(example["task_name"])

Using with EASI

# Run evaluation on the base split
easi run ebmanipulation_base --agent react --backend openai --model gpt-4o

# List available manipulation splits
easi task list | grep ebmanipulation

Requirements

• —CoppeliaSim V4.1.0
• —PyRep (CoppeliaSim Python binding)
• —AMSolver (modified RLBench fork)

Acknowledgements

This dataset is derived from the EmbodiedBench EB-Manipulation benchmark and uses CoppeliaSim as the simulation environment.