Datasetpublic

Kuoskyler/swiftplan-isaac-sim

SwiftPlan Isaac Sim Dataset This dataset contains Isaac Sim observation images for frame-level high-level action selection in robotic task planning. Each sample includes: an RGB observation image, a task instruction, a frame-level high-level action label, an action type, an optional target object. The dataset is designed for execution-time high-level decision making, where a model selects the next high-level action from the current observation and task instruction.… See the full description on the dataset page: https://huggingface.co/datasets/Kuoskyler/swiftplan-isaac-sim.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

2likes97downloads

Dataset Card

SwiftPlan Isaac Sim Dataset

This dataset contains Isaac Sim observation images for frame-level high-level action selection in robotic task planning.

Each sample includes:

—an RGB observation image,
—a task instruction,
—a frame-level high-level action label,
—an action type,
—an optional target object.

The dataset is designed for execution-time high-level decision making, where a model selects the next high-level action from the current observation and task instruction.

Splits

This dataset provides two fixed splits:

—train
—test

Data Format

Each split folder contains images and a metadata.jsonl file.

Example:

json

{
  "id": "swiftplan_isaac_000000",
  "file_name": "000000_capture_orange_20260418_205731_247523.png",
  "instruction": "prepare the fruits",
  "action_label": "pickup orange",
  "action_type": "pickup",
  "target_object": "orange",
  "split": "train"
}

Usage

python

from datasets import load_dataset

dataset = load_dataset("Kuoskyler/swiftplan-isaac-sim")
print(dataset)
print(dataset["train"][0])

USD file

The Isaac Sim USD scene file is also included in this dataset.

Task

Given an RGB observation image and a task instruction, predict the next high-level action label.