datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Multitask-National-Speech-Corpus-v1Multitask-National-Speech-Corpus (MNSC v1) is derived from IMDA's NSC Corpus.
MNSC is a multitask speech understanding dataset derived and further annotated from IMDA NSC Corpus. It focuses on the knowledge of Singapore's local accent, localised terms, and code-switching.
ASR: Automatic Speech Recognition
SQA: Speech Question Answering
SDS: Spoken Dialogue Summarization
PQA: Paralinguistic Question Answering
from datasets import load_dataset
data =… See the full description on the dataset page: https://huggingface.co/datasets/MERaLiON/Multitask-National-Speech-Corpus-v1.Multi-SWE-smith-tasksmulti_task_multi_modal_knowledge_retrieval_benchmark_M2KR
PreFLMR M2KR Dataset Card
Dataset details
Dataset type:
M2KR is a benchmark dataset for multimodal knowledge retrieval. It contains a collection of tasks and datasets for training and evaluating multimodal knowledge retrieval models.
We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:… See the full description on the dataset page: https://huggingface.co/datasets/BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR.Multitask-National-Speech-Corpus-v1-extendso101-ai20k-multitaskThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/hungho77/so101-ai20k-multitask.multi-task-tcc-robosuite
XIRL
Overview
Setup
Datasets
Code Navigation
Experiments: Reproducing Paper Results
Extending XIRL
Acknowledgments
Overview
Code release for our CoRL 2021 conference paper:
XIRL: Cross-embodiment Inverse Reinforcement Learning
Kevin Zakka1,3, Andy Zeng1, Pete Florence1, Jonathan Tompson1, Jeannette Bohg2, and Debidatta Dwibedi1
Conference on Robot Learning (CoRL) 2021
1Robotics at Google,
2Stanford… See the full description on the dataset page: https://huggingface.co/datasets/Renton-Ren/multi-task-tcc-robosuite.multitask_german_examples_32kmulti_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN
PreFLMR M2KR Dataset Card
Dataset details
Dataset type:
M2KR is a benchmark dataset for multimodal knowledge retrieval. It contains a collection of tasks and datasets for training and evaluating multimodal knowledge retrieval models.
We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:… See the full description on the dataset page: https://huggingface.co/datasets/BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN.p2-etf-multitask-gp-resultsso101-multitask-calib
SO-101 multitask — calibration pool (LeRobot v2.1)
The observation pool that post-training quantization of a GR00T N1.7 SO-101
policy is calibrated on, in the LeRobot v2.1 layout. Same recordings as
hungho77/so101-multitask,
which is stored in the v3.0 layout; GR00T's data loader reads v2.1 only, so this
is the copy a quantization or evaluation run actually opens.
143 episodes · 67,496 frames · 3 tasks · 30 fps · single SO-101 arm ·
two 480×640 cameras (top, wrist) · 6-D state… See the full description on the dataset page: https://huggingface.co/datasets/hungho77/so101-multitask-calib.multitask_5blocks_v3_865ep_trimmed_mergedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/eslab1234/multitask_5blocks_v3_865ep_trimmed_merged.multitask_5blocks_v3_867ep_trimmed_mergedbusybox_multitask
Single-arm BusyBox multitask dataset
LeRobot v3 dataset containing 66 episodes and
12141 frames at 20 FPS across 27 single-arm
BusyBox tasks. The green-button source contributes only its first three
episodes; all other listed sources contribute every episode.
Visualization
Open episode 0 in the LeRobot Dataset Visualizer.
The original unique language instruction from each source is preserved.
RoboCandyWrapper busybox_state and robot_telemetry plugin data is… See the full description on the dataset page: https://huggingface.co/datasets/AutoRLBench/busybox_multitask.multitask_5blocks_v2_530epThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/eslab1234/multitask_5blocks_v2_530ep.ur5e_multitask_000This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": null,
"total_episodes": 270,
"total_frames": 106900,
"total_tasks": 11,
"total_videos": 0,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 15,
"splits": {
"train": "0:270"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Perseus101/ur5e_multitask_000.so101_multitask_oldsim_pick_place_stack_onehotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"observation.state": {
"dtype": "float32",
"shape": [
6
],
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos"… See the full description on the dataset page: https://huggingface.co/datasets/Chaenn/so101_multitask_oldsim_pick_place_stack_onehot.VQA-MultiTasklogu_bimanual_joint_multi_taskThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": null,
"total_episodes": 212,
"total_frames": 101703,
"total_tasks": 2,
"total_videos": 636,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 20,
"splits": {
"train": "0:212"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/gaozj/logu_bimanual_joint_multi_task.so101_multi_taskThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "so101_follower",
"total_episodes": 9,
"total_frames": 3131,
"total_tasks": 1,
"total_videos": 18,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:9"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/fbeltrao/so101_multi_task.extract_multitask_1_DPAThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos"
],
"shape": [
4
]
},
"observation.state": {… See the full description on the dataset page: https://huggingface.co/datasets/liam-martiiin/extract_multitask_1_DPA.so101_multitask_newsim_place_stack_onehotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"observation.state": {
"dtype": "float32",
"shape": [
6
],
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos"… See the full description on the dataset page: https://huggingface.co/datasets/Chaenn/so101_multitask_newsim_place_stack_onehot.so101-multitaskThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 143,
"total_frames": 67496,
"total_tasks": 3,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:143"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/hungho77/so101-multitask.busybox_multitask
Single-arm BusyBox dataset
LeRobot v3 dataset containing 132 episodes and
25734 frames at 20 FPS across
27 BusyBox task(s), merged from 4 source
dataset(s). This revision adds 3 cell-3 source dataset(s) on top of armnet/busybox_multitask. Green-button cell-3 contributes only its first three episodes. Cell-3 language instructions are normalized to the original multitask strings.
Visualization
Open episode 0 in the LeRobot Dataset Visualizer.
The original unique… See the full description on the dataset page: https://huggingface.co/datasets/armnet/busybox_multitask.touchdown_multitask_1_DPAThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos"
],
"shape": [
4
]
},
"observation.state": {… See the full description on the dataset page: https://huggingface.co/datasets/liam-martiiin/touchdown_multitask_1_DPA.multitask_5blocks_v3_704ep_hil_r1_mergedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/eslab1234/multitask_5blocks_v3_704ep_hil_r1_merged.smolvla_multitask_hil_285k_v1_220ep_trimmedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/eslab1234/smolvla_multitask_hil_285k_v1_220ep_trimmed.MultiTaskMouseBehaviour
barnes_maze
direct_interaction
marble
nort
nort2
open_field
social_interaction
t_maze
three_chamber
Multi-Task Mouse Behaviour Dataset
Bounding boxes, instance masks, 27-point poses and persistent track ids for
laboratory mice, on the same frames, across nine standard behavioural
assays. 14,300 frames, 27,716 annotated instances, 1280×720.
Most animal-behaviour datasets give you one annotation type. Training or
evaluating a model that does detection and segmentation and… See the full description on the dataset page: https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour.multitask_5blocks_v3_684ep_trimmed_mergedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/eslab1234/multitask_5blocks_v3_684ep_trimmed_merged.multitask_vqa_benchmarkThis dataset is a part of .
🍈 MMT-47: Multimodal Multi-Task Benchmark
47 Tasks · 7 Categories · 3 Modalities (Image, Video, Text)
Cite our ICML-2026 paper for this dataset
@article{kowsher2026lime,
title={LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning},
author={Kowsher, Md and Mansoor, Haris and Prottasha, Nusrat Jahan and Garibay, Ozlem and Zhu, Victor and Ji, Zhengping and Chen, Chen},
journal={arXiv preprint… See the full description on the dataset page: https://huggingface.co/datasets/Kowsher/multitask_vqa_benchmark.eval_multi-task-144epiThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so101_follower",
"total_episodes": 144,
"total_frames": 116432,
"total_tasks": 3,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:144"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4"… See the full description on the dataset page: https://huggingface.co/datasets/CSI-Agent/eval_multi-task-144epi.