datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.SWE-smith
SWE-smith Dataset
Code
•
Paper
•
Site
[12/14/2025] NOTE: We will no longer actively update this dataset.
While this dataset is still functional and usable, we recommend you use the `SWE-bench/SWE-smith-[lang]` datasets.
For better maintainability and ease-of-use, we are maintaining language-specific datasets in lieu of this mono-repo.
The SWE-smith Dataset is a training dataset of 50137 task instances from 128 GitHub repositories, collected using the SWE-smith toolkit.… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-smith.SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified.SWE-bench_Lite
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite.SWE-bench
Dataset Summary
SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the problem_statement… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench.SWE-bench-Live
A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline.
For the official data release page, please see microsoft/SWE-bench-Live.
Dataset Summary
SWE-bench-Live is a live benchmark for issue resolving, designed to evaluate an AI system’s ability to complete real-world software engineering tasks. Thanks to our automated dataset curation pipeline, we plan to update SWE-bench-Live on a monthly basis to provide the… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench-Live/SWE-bench-Live.swe-bench-dummy-test-datasetSWE-bench_Lite
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite.SWE-bench_Multilingual
SWE-bench Multilingual
Dataset Summary
SWE-bench Multilingual is a dataset that tests systems' ability to resolve real-world GitHub issues across a broad range of programming languages. The original SWE-bench is Python-only; this dataset extends the same task format to 9 languages drawn from 41 popular repositories.
The dataset collects 300 test Issue-Pull Request pairs. Evaluation is performed by unit test verification, using post-PR behavior as the reference solution.
The… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual.SWE-bench_Pro
SWE-bench Pro V2
SWE-bench Pro is a benchmark of long-horizon software engineering tasks drawn from real pull requests in 11 open-source
repositories (Go, Python, JavaScript, TypeScript). Each task gives an agent a repository at a base commit plus a PR
description with explicit requirements and interfaces; the agent's patch is graded by hidden fail-to-pass and
pass-to-pass tests in a pristine container.
V2 (2026-09-22) is the default config: 642 tasks (go 256, python 237, js 145… See the full description on the dataset page: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro.SWE-bench-extraNote: This dataset has an improved and significantly larger successor: SWE-rebench.
Dataset Summary
SWE-bench Extra is a dataset that can be used to train or evaluate agentic systems specializing in resolving GitHub issues. It is based on the methodology used to build SWE-bench benchmark and includes 6,415 Issue-Pull Request pairs sourced from 1,988 Python repositories.
Dataset Description
The SWE-bench Extra dataset supports the development of software engineering agents… See the full description on the dataset page: https://huggingface.co/datasets/nebius/SWE-bench-extra.SWE-smith-py
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/2025, SWE-smith: Python contains 50908 task instances from 131 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
Multi-SWE-bench
👋 Overview
This repository contains the Multi-SWE-bench dataset, introduced in Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving, to address the lack of multilingual benchmarks for evaluating LLMs in real-world code issue resolution.
Unlike existing Python-centric benchmarks (e.g., SWE-bench), this framework spans 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances,
curated from 2,456 candidates by 68 expert… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench.SWE-bench
Dataset Summary
SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench.MultiLangThis is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
We crawled GitHub issues as SWE tasks for coding agents to solve.
We used RepoLaunch (repository_link) to setup docker images as the problem-solving sandboxes for each instance.
Each split is task set of one language.
Each split is filtered by prompting LLM -- a task instance is discarded if LLM cannot infer test patch from problem statement or problem statement contains solution (in NL or code).… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench-Live/MultiLang.SWE-bench_Multimodal
SWE-bench Multimodal
Dataset Summary
SWE-bench Multimodal is a dataset that tests systems' ability to resolve real-world GitHub issues in visual software domains. Unlike the original SWE-bench, which is Python-only and text-only, every task instance here comes from a JavaScript or TypeScript repository and carries at least one image asset — a screenshot, a screen recording, a diagram, or a rendering of incorrect output.
The dataset collects 612 Issue-Pull Request pairs from 17… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Multimodal.Multi-SWE-bench
Multi-SWE-bench
Re-upload of ByteDance's Multi-SWE-bench
evaluation benchmark: 2,132 issue-resolving tasks across the seven Multi-SWE languages.
This is the held-out eval benchmark; for RL training data use
PrimeIntellect/Multi-SWE-RL-Verified.
Changes vs upstream
Storage schema only: per-test maps are stored as columnar struct-of-lists so the rows load
cleanly with datasets. Row content is unchanged.
License mirrors upstream: ByteDance licenses the dataset… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/Multi-SWE-bench.SWE-smith-javaMulti-SWE-benchMulti-SWE-bench_mini
👋 Overview
To make the benchmark more lightweight and minimize resource consumption, we have introduced the Multi-SWE-bench mini.
This version contains 400 instances, with 50 instances per language (Python, Java, TypeScript, JavaScript, Go, Rust, C, and C++). It covers a range of difficulty levels—easy, medium, and hard—ensuring a balanced and efficient evaluation across multiple languages without excessive resource usage.The leaderboard can be found at:… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench_mini.SWE-smith-trajectories
SWE-smith Trajectories
Code
•
Paper
•
Site
This dataset contains the 5017 trajectories we fine-tuned Qwen 2.5 Coder Instruct on, leading to
SWE-agent-LM-32B, a coding LM agent that
achieve 40.2% on SWE-bench Verified (no verifiers or multiple rollouts, just 1 attempt per instance).
Trajectories were generated by running SWE-agent + Claude 3.7 Sonnet on task instances from
the SWE-smith dataset.
Multi-SWE-bench
SWE-bench-Java: A GitHub Issue Resolving Benchmark for Java
📰 News
[Aug. 27, 2024]:We’ve released the JAVA version of SWE-bench! Check it out on Hugging Face. For more details, see our paper!
📄 Abstract
GitHub issue resolving is a critical task in software engineering, recently gaining significant attention in both industry and academia. Within this task, SWE-bench has been released to evaluate issue resolving capabilities of large language models (LLMs)… See the full description on the dataset page: https://huggingface.co/datasets/Daoguang/Multi-SWE-bench.Multi-SWE-bench_trajs
🧠 Multi-SWE-bench Trajectories
This repository stores all trajectories and logs generated by agents evaluated on the Multi-SWE-bench leaderboard.
📚 Citation
@misc{zan2025multiswebench,
title={Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving},
author={Daoguang Zan and Zhirong Huang and Wei Liu and Hanwu Chen and Linhao Zhang and Shulin Xin and Lu Chen and Qi Liu and Xiaojian Zhong and Aoyan Li and Siyao Liu and Yongsheng Xiao and Liangqiang Chen… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench_trajs.SWE-bench_Multimodal
SWE-bench Multimodal
SWE-bench Multimodal is a dataset of 617 task instances that evalutes Language Models and AI Systems on their ability to resolve real world GitHub issues.
To learn more about the dataset, please visit our website.
More updates coming soon!
Multi-SWE-bench-flash
👋 Overview
This repository contains the Multi-SWE-bench dataset, introduced in Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving, to address the lack of multilingual benchmarks for evaluating LLMs in real-world code issue resolution.
Unlike existing Python-centric benchmarks (e.g., SWE-bench), this framework spans 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances,
curated from 2,456 candidates by 68 expert annotators… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench-flash.SWE-smith-jsSWE-smith-go
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/202, SWE-smith: Golang contains 8212 task instances from 87 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
swebench-verified-trajectories
SWE-bench Verified — mini-swe-agent trajectories (10 models)
Native .traj.json files from running mini-swe-agent on all 500 SWE-bench Verified instances
for each of the 10 models below. Downloaded from
Docent via preprocessing/swebench_docent.py in
contrastive_learning.
Layout
swebench_verified_raw/
<model_label>/
<instance_id>/
<instance_id>.traj.json
Per-file schema
{
"instance_id": str,
"trajectory_format": "mini-swe-agent-1.1"… See the full description on the dataset page: https://huggingface.co/datasets/tarsur385/swebench-verified-trajectories.SWE-Bench-Verified-O1-reasoning-high-results
SWE-Bench Verified O1 Dataset
Executive Summary
This repository contains verified reasoning traces from the O1 model evaluating software engineering tasks. Using OpenHands + CodeAct v2.2, we tested O1's bug-fixing capabilities on the SWE-Bench Verified dataset, achieving a 28.8% success rate across 500 test instances.
Overview
This dataset was generated using the CodeAct framework, which aims to improve code generation through enhanced action-based reasoning.… See the full description on the dataset page: https://huggingface.co/datasets/AlexCuadron/SWE-Bench-Verified-O1-reasoning-high-results.multi_SWE_Bench_Rust
multi_SWE_Bench_Rust
数据集描述...