datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
gitee-code
Gitee Code Dataset
Dataset Description
This dataset was compiled from code repositories hosted on Gitee, China's largest code hosting platform and a leading alternative to GitHub in the Chinese developer community. Gitee is widely used by Chinese developers, enterprises, and open-source projects, making this dataset particularly valuable for training code models with strong Chinese language understanding and Chinese coding conventions.
Dataset Summary… See the full description on the dataset page: https://huggingface.co/datasets/nyuuzyou/gitee-code.github_archive
GitHub Archive
Description
According to GitHub’s terms of service, issues and pull request descriptions—along with the their comments—inherit the license of their associated repository.
To collect this data, we used the GitHub Archive’s public BigQuery table of events to extracted all issue, pull request, and comment events since 2011 and aggregated them into threads.
The table appeared to be missing “edit” events so the text from each comment is the original from when… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/github_archive.open-github
OpenGitHub
What is it?
This dataset contains every public event on GitHub: every push, pull request, issue, star, fork, code review, release, and discussion across all public repositories. GitHub is the world's largest software development platform, home to over 200 million repositories and the daily work of tens of millions of developers, from individual open-source contributors to the engineering teams behind the most widely used software on earth.
The archive currently… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-github.github-top-code
GitHub Top Developer Source Code
A curated dataset of 1.3M+ source code files from GitHub's top ranked developers (2015-2025).
This dataset is based on the top ranked developers from this dataset: https://huggingface.co/datasets/ronantakizawa/github-top-developers
Dataset Summary
1.3M+ source code files from repositories across ~4,700 unique developers
80+ programming languages included (Python, JavaScript, TypeScript, Rust, Go, C/C++, Java, and more)
Source code only —… See the full description on the dataset page: https://huggingface.co/datasets/ronantakizawa/github-top-code.github-codereview
Code Review Dataset
A large-scale dataset of the best human-written code reviews from top GitHub repositories.
Each row captures a moment where a human code reviewer left an inline comment on a pull request, and the author subsequently modified the code in response.
The dataset also includes negative examples — code from the same PRs that passed review without comments — to help models learn when code is acceptable.
This provides a natural signal for training models to:
Generate… See the full description on the dataset page: https://huggingface.co/datasets/ronantakizawa/github-codereview.github_archive_filtered
GitHub Archive
Description
According to GitHub’s terms of service, issues and pull request descriptions—along with their comments—inherit the license of their associated repository.
To collect this data, we used the GitHub Archive’s public BigQuery table of events to extract all issue, pull request, and comment events since 2011 and aggregated them into threads.
The table appeared to be missing “edit” events so the text from each comment is the original from when it was… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/github_archive_filtered.github-jupyter
GitHub Jupyter Dataset
Dataset Description
The dataset was extracted from Jupyter Notebooks on BigQuery.
Licenses
Each example has the license of its associated repository. There are in total 15 licenses:
[
'mit',
'apache-2.0',
'gpl-3.0',
'gpl-2.0',
'bsd-3-clause',
'agpl-3.0',
'lgpl-3.0',
'lgpl-2.1',
'bsd-2-clause',
'cc0-1.0',
'epl-1.0',
'mpl-2.0',
'unlicense',
'isc',
'artistic-2.0'
]
the_pile_github
Dataset Card for The Pile GitHub
Dataset Summary
This is the GitHub subset of EleutherAi/The Pile dataset and contains GitHub repositories. The programming languages are identified using the guesslang library. A total of 54 programming languages are included in the dataset.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The following languages are covered by the dataset:
'Assembly', 'Batchfile', 'C', 'C#', 'C++', 'CMake'… See the full description on the dataset page: https://huggingface.co/datasets/andstor/the_pile_github.code-parrot-github-code
GitHub Code Dataset
Dataset Description
The GitHub Code dataset consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in 1TB of data. The dataset was created from the public GitHub dataset on Google BiqQuery.
How to use it
The GitHub Code dataset is a very large dataset so for most use cases it is recommended to make use of the streaming API of datasets. You can load and iterate through the dataset with the following… See the full description on the dataset page: https://huggingface.co/datasets/macrocosm-os/code-parrot-github-code.git-commits-merged
Themis-Git-Commits-Merged
Overview
Themis-Git-Commits-Merged is a large-scale dataset of ~3.98M single-file code commits from permissively licensed GitHub repositories that have been cross-referenced with GHTorrent pull request data to retain only commits that are part of successfully merged, non-reverted pull requests. This provides implicit human validation of each code change — a merge decision by project maintainers confirms the intent and quality of… See the full description on the dataset page: https://huggingface.co/datasets/project-themis/git-commits-merged.GitHub-Agentic-PR-Dataset
GitHub Agentic PR Dataset
A large-scale dataset of ~2 million GitHub Pull Requests authored by AI coding agents (Claude Code, Cursor, GitHub Copilot, Devin) and human developers, complete with commits, file-level diffs, patches, bug-fix classification, review-depth metrics, and test-quality measures.
The GitHub Agentic PR Dataset is a research-grade corpus for studying how AI coding agents contribute to real-world open-source software, and how their pull requests compare to… See the full description on the dataset page: https://huggingface.co/datasets/mabujadallah/GitHub-Agentic-PR-Dataset.github-code-haskell-file
Dataset Card for "github-code-haskell-file"
Rows: 339k
Download Size: 806M
This dataset is extracted from github-code-clean.
Each row also contains attribute values for my personal analysis project.
12.6% (43k) of the rows have cyclomatic complexity and LOC valued at -1 because homplexity failed in parsing the row's uncommented_code.
the-stack-github-issues
Dataset Description
This dataset contains conversations from GitHub issues and Pull Requests. Each conversation is comprised of a series of events, such as opening an issue, creating a comment,
or closing the issue, and includes the author's username, text, action, and identifiers such as the issue ID and number.
The dataset, which is mostly in English, has a total size of 54GB and 30.9M files.
Dataset Structure
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-github-issues.random-python-github-repositories
random-python-github-repositories
A collection of 1650 open-source Python GitHub repositories, packaged as zipped archives alongside a metadata CSV. Intended as a seed dataset for code retrieval, context engineering, and SWE-bench-style dataset construction tasks. All repos contain 250+ .py files.
Contents
repos_meta_data.csv — metadata for each repo (owner, repo_name, stars, license, py_file_count, alpha_hash)
repos-zipped/ — one .zip per repo, named… See the full description on the dataset page: https://huggingface.co/datasets/AmanPriyanshu/random-python-github-repositories.gitcode-code
GitCode Code Dataset
Dataset Description
This dataset was compiled from code repositories hosted on GitCode, a code hosting platform in China backed by CSDN (China Software Developer Network). GitCode serves as a domestic alternative to GitHub, widely used by Chinese developers, students, and enterprises for hosting open-source projects and educational resources, making this dataset particularly valuable for training code models with Chinese language understanding and… See the full description on the dataset page: https://huggingface.co/datasets/nyuuzyou/gitcode-code.GitHub-CC0
Public Domain GitHub Repositories Dataset
This dataset contains metadata and source code of 9,000 public domain (cc0 or unlicense) licensed GitHub repositories that have more than 25 stars.
The dataset was created by scraping the GitHub API and downloading the repositories, so long as they are under 100mb.
The dataset can be used for various natural language processing and software engineering tasks, such as code summarization, code generation, code search, code analysis, etc.… See the full description on the dataset page: https://huggingface.co/datasets/KoalaAI/GitHub-CC0.1c_githubgithub-code-fim
Description
Fill-in-the-middle (FIM) code-completion examples mined from public GitHub repositories. Columns: text (FIM-formatted), source, repo, language.
Derived dataset. Source material retains its original per-item licence (see source/repo columns); treat as other / mixed. Provided as-is.
Usage
from datasets import load_dataset
ds = load_dataset("PotatoHD/github-code-fim")
random-small-github-repositories
random-small-github-repositories
A collection of 5,613 small-to-medium open-source GitHub repositories, packaged as zipped archives alongside a metadata CSV. Intended as a seed dataset for code retrieval, context engineering, and SWE-bench-style dataset construction tasks.
Contents
seed_small_repos.csv — metadata for each repo (owner, repo_name, stars, license, repo_hash)
repos-zipped/ — one .zip per repo, named {repo_hash}.zip
unzipper.py - unzipping python… See the full description on the dataset page: https://huggingface.co/datasets/AmanPriyanshu/random-small-github-repositories.GitChameleon-2.0
GitChameleon 2.0
GitChameleon 2.0 is an AI coding benchmark comprising 328 Python-based problems conditioned on specific versions of popular libraries for scientific computing and web development. It evaluates whether AI code generation models can correctly use library APIs as they existed at a particular version — a challenging test of version-specific knowledge.
Note: This is GitChameleon 2.0, a distinct and newer work from the original GitChameleon benchmark. Please do not… See the full description on the dataset page: https://huggingface.co/datasets/cabbage972/GitChameleon-2.0.github-pr-agent-trajectories
Description
Reconstructed multi-step agent trajectories for resolving public GitHub PRs: a plan of subtasks and per-subtask subagents with scoped files and solution diffs.
Derived dataset. Source material retains its original per-item licence (see source/repo columns); treat as other / mixed. Provided as-is.
Usage
from datasets import load_dataset
ds = load_dataset("PotatoHD/github-pr-agent-trajectories")
github-jupyter-code-to-text
Dataset description
This dataset consists of sequences of Python code followed by a a docstring explaining its function. It was constructed by concatenating code and text pairs
from this dataset that were originally code and markdown cells in Jupyter Notebooks.
The content of each example the following:
[CODE]
"""
Explanation: [TEXT]
End of explanation
"""
[CODE]
"""
Explanation: [TEXT]
End of explanation
"""
...
How to use it
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/github-jupyter-code-to-text.gitgud-code
GitGud Code Dataset
Dataset Description
This dataset was compiled from code repositories hosted on GitGud.io, a GitLab-based code hosting platform. GitGud.io serves as an alternative git hosting service used by various developer communities and open-source projects.
Dataset Summary
Statistic
Value
Total Files
16,322,315
Total Repositories
7,204
Total Size
17.46 GB (compressed Parquet)
Programming Languages
2,185
File Format
Parquet with Zstd… See the full description on the dataset page: https://huggingface.co/datasets/nyuuzyou/gitgud-code.GitBugs
GitBugs
https://github.com/av9ash/gitbugs/
License and Citation
This project is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.Any use or reuse of this work please cite the following:
@article{patil2025gitbugs,
title={GitBugs: Bug Reports for Duplicate Detection, Retrieval Augmented Generation, Triage, and More},
author={Patil, Avinash},
journal={arXiv preprint arXiv:2504.09651},
year={2025}
}
For more details on the… See the full description on the dataset page: https://huggingface.co/datasets/av9ash/GitBugs.github-python
GitHub-Python — Licensed & Elaborated Variants
This repository ships two complementary Python-code corpora extracted from
public GitHub:
Licensed Subset – strictly permissive-licensed files suitable for
commercial redistribution / model training (main corpus used in our
experiments).
Elaborated Collection – a broader crawl that additionally contains files
under copyleft or unclear licenses (GPL/AGPL/LGPL, etc.). Useful for
analysis or pre-training where license mixing is… See the full description on the dataset page: https://huggingface.co/datasets/jblitzar/github-python.Bhagavad-Gita_Dataset
Srimad Bhagavad Gita Dataset
A parallel corpus of the Srimad Bhagavad Gita containing original verses in Sanskrit (sa), with Hindi (hi) and English (en) translations.This dataset is suitable for translation, text generation, and feature extraction tasks.
Dataset Details
Based on: This dataset is based on a book called 'Srimad Bhagavad Gita' kept at Central Archaelogical Library, New Delhi
Original Book Source (IGNCA): https://ignca.gov.in/Asi_data/279.pdf… See the full description on the dataset page: https://huggingface.co/datasets/JDhruv14/Bhagavad-Gita_Dataset.Reviewer2_PGE_cleaned
Cleaned Review Dataset for Reviewer2
This is a cleaned version of our dataset and can be directly used for fine-tuning. The raw data files including metadata for each paper is in this directory.
venue: venue of the paper;
paper_content: content of the paper divided into sections
prompt: prompt generated for the review based on our PGE pipeline
format: the format of the review
review: human-written review for the paper
Dataset Sources
We incorporate parts of the… See the full description on the dataset page: https://huggingface.co/datasets/GitBag/Reviewer2_PGE_cleaned.github-pr-resolve
Description
Issue/PR-review -> resolving patch pairs from merged public GitHub PRs. Includes issue_text, pr_title/pr_body, gold_diff, changed_files, threaded review_threads, and base/head SHAs.
Derived dataset. Source material retains its original per-item licence (see source/repo columns); treat as other / mixed. Provided as-is.
Usage
from datasets import load_dataset
ds = load_dataset("PotatoHD/github-pr-resolve")
gitflic-code
GitFlic Code Dataset
Dataset Description
This dataset was compiled from code repositories hosted on GitFlic, the first Russian service for storing and working with source code, based on the Git version control system. GitFlic is widely used by Russian developers, enterprises, and open-source projects, making this dataset particularly valuable for training code models with strong Russian language understanding and Russian coding conventions.
Dataset Summary… See the full description on the dataset page: https://huggingface.co/datasets/nyuuzyou/gitflic-code.git-natural-language-commands
Git Natural Language Commands
A dataset mapping English natural-language instructions to their corresponding git commands, intended for training and evaluating models that translate user intent into safe, correct shell commands.
Disclaimer: This dataset was generated using Large Language Models (LLMs). The examples have not been manually verified against real-world usage and may contain errors, inconsistencies, or non-canonical phrasings. Use with appropriate caution.… See the full description on the dataset page: https://huggingface.co/datasets/burak29/git-natural-language-commands.