Datasetpublic

9211sun/GUI-Xplore

GUI-Xplore Dataset 🖥️🔍 GUI-Xplore is a novel dataset designed to improve cross-application and cross-task generalization for GUI agents. It introduces exploration videos to provide context-aware learning, enabling agents to adapt to new applications efficiently. 📌 Dataset Summary 312 apps across 33 categories from 6 major domains (Entertainment, Productivity, Health, Shopping, Travel, News). 115+ hours of exploration videos, ensuring comprehensive app… See the full description on the dataset page: https://huggingface.co/datasets/9211sun/GUI-Xplore.

sourceHugging Facecc-by-4.0updated 2y agoView on Hugging Face

3likes19kdownloads

Dataset Card

GUI-Xplore Dataset 🖥️🔍

GUI-Xplore is a novel dataset designed to improve cross-application and cross-task generalization for GUI agents. It introduces exploration videos to provide context-aware learning, enabling agents to adapt to new applications efficiently.

📌 Dataset Summary

—312 apps across 33 categories from 6 major domains (Entertainment, Productivity, Health, Shopping, Travel, News).
—115+ hours of exploration videos, ensuring comprehensive app interaction coverage.
—32,569 structured question-answer pairs spanning five hierarchical downstream tasks.
—Designed to support Large Vision-Language Models (LVLMs) and GUI-based AI agents.

📂 Dataset Structure

🔢 Data Splits

Split	# Apps	# Screenshot	# Action	# Q&A Samples
Auto	207	13,660	21,346	15,934
Manual_train	85	18,523	17,963	16,744
Manual_test	20	2,184	1,984	2,638
Total	312	34,367	41,293	35,316

🏷️ Annotation Schema

Each app in the dataset includes:

1.Exploration Video (.mp4)
2.Screenshot Sequence (.png)
3.Action Sequence (.json)
4.Screenshot-Action Relation (.json)
5.View Hierarchy (.xml)
6.Question-Answer Pairs (.json)

🏗️ Tasks

GUI-Xplore defines five downstream tasks to evaluate GUI agents:

Task	Description
Application Overview	Summarize app functionalities based on exploration.
Page Analysis	Interpret the purpose of a specific GUI screen.
Application Usage	Infer correct navigation sequences from homepage to task completion.
Action Recall	Identify past interactions within an app.
Action Sequence Verification	Verify logical operation order within an app.

📥 How to Use

python

from datasets import load_dataset

dataset = load_dataset("9211sun/GUI-Xplore")

# Example data exploration
print(dataset["manual_test"][0])