datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
commitbench
CommitBench: A Benchmark for Commit Message Generation
EXECUTIVE SUMMARY
We provide CommitBench as an open-source, reproducible and privacy- and license-aware benchmark for commit message generation. The dataset is gathered from GitHub repositories with licenses that permit redistribution. We provide six programming languages, Java, Python, Go, JavaScript, PHP, and Ruby. The commit messages in natural language are restricted to English, as it is the working language in… See the full description on the dataset page: https://huggingface.co/datasets/Maxscha/commitbench.agent-commits-raw
AI Coding-Agent Commits on GitHub
This dataset documents commits associated with four AI coding agents: Claude, OpenAI Codex, GitHub Copilot and Cursor. It contains 1,853,915 commit records across 444,055 GitHub repositories, with 220,753 identifiable GitHub user accounts recorded as commit authors. Messages and author identities are in commits; repository metadata, file changes and patch text are available in separate tables.
Dataset
Agent
Commit records… See the full description on the dataset page: https://huggingface.co/datasets/ASSERT-KTH/agent-commits-raw.commit0commitbench_conventionalconventional-commits
Git Diff → Conventional Commit Messages
A dataset of 12,433 (git diff, commit message) pairs scraped from real open-source TypeScript repositories, filtered for quality and formatted for fine-tuning small language models to generate Conventional Commits.
Built as part of a project to fine-tune a local LLM to write commit messages as a prepare-commit-msg git hook. Full write-up: eliotbas.com/projects/commits-fine-tuning
Dataset details
Source… See the full description on the dataset page: https://huggingface.co/datasets/Elib27/conventional-commits.commitpack-subset-cfA subset of CommitPack used for pretraining SantaCoderPack from the OctoPack paper.
It focuses on data where the code before + special token + code after fits into 8192 tokens and on 6 languages. The data is in commit format (cf): <commit_before>code_before<commit_message>commit_message<commit_after>code_after.
commit-chronicle
📜 CommitChronicle 🔮
This is the dataset for commit message generation (and/or completion), introduced in the paper "From Commit Message Generation to History-Aware Commit Message Completion", ASE 2023.
Its key features:
large-scale and multilingual: contains 10.7M commits from 11.9k GitHub repositories in 20 programming languages;
diverse: avoids restrictive filtering on commit messages or commit diffs structure;
suitable for experiments with commit history: provides metadata… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/commit-chronicle.commit-classification-dataset
Commit Classification Dataset
This dataset is designed for multi-label classification of Git commit messages into predefined categories.
Dataset Summary
This dataset contains:
Training data: Commit messages and their corresponding labels for training the model.
Validation data: A separate set of messages for tuning and evaluation.
Testing data: Unlabeled commit messages for testing the model’s performance.
The goal of the dataset is to classify each commit message into… See the full description on the dataset page: https://huggingface.co/datasets/meriemm6/commit-classification-dataset.github-commits-diff-dedup-pjjs-april
Deduplicated Commits
Deduplicated based on diff:
content = '\n'.join(difflib.unified_diff(
old_content.splitlines(keepends=True),
new_content.splitlines(keepends=True),
n=5
))
Parameters:
Minimum ngram size: 5
MinHash ngram size: 5
MinHash threshold: 0.8
commit-classification-17kknesset-committees-chunkscommit0_combinedcommitpackmetaGitHub metadata for https://huggingface.co/datasets/bigcode/commitpack
commitments-of-traders
Commitments of Traders
Who was long and who was short in every US futures market — and, for once,
when anyone could actually see it.
421 223 market-weeks · 211 071 point-in-time rows · 2 585 weekly releases ·
762 markets · 2010-01-05 to 2026-09-08
The pipeline lives in recipe/ at the same revision as the data.
See PIPELINE.md for the method.
The data is from Tuesday. It comes out on Friday.
A COT report is taken as of the close on Tuesday and published at 3:30… See the full description on the dataset page: https://huggingface.co/datasets/ZipLime/commitments-of-traders.pretrain-commits-v2-mixNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
79,318,557,379 (79.3B)
Trainable tokens
28,772,968,648 (28.8B)
Documents
31,431,846
Shards
696
UTF-8 bytes
310,412,170,445
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix.commit-chronicle
📜 CommitChronicle 🔮
This is the dataset for commit message generation (and/or completion), introduced in the paper "From Commit Message Generation to History-Aware Commit Message Completion", ASE 2023.
Its key features:
large-scale and multilingual: contains 10.7M commits from 11.9k GitHub repositories in 20 programming languages;
diverse: avoids restrictive filtering on commit messages or commit diffs structure;
suitable for experiments with commit history: provides metadata… See the full description on the dataset page: https://huggingface.co/datasets/Hirunima/commit-chronicle.git-commits-merged
Themis-Git-Commits-Merged
Overview
Themis-Git-Commits-Merged is a large-scale dataset of ~3.98M single-file code commits from permissively licensed GitHub repositories that have been cross-referenced with GHTorrent pull request data to retain only commits that are part of successfully merged, non-reverted pull requests. This provides implicit human validation of each code change — a merge decision by project maintainers confirms the intent and quality of… See the full description on the dataset page: https://huggingface.co/datasets/project-themis/git-commits-merged.commits-codegeex
Dataset Card for "commits-codegeex"
More Information needed
commit-msg-edits
✍️ Commit Message Edits Dataset
This dataset is a collection of expert-labeled commit message edits contributed via Commit Message Editing app presented in Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings.
Labelers were presented with GPT-4 generated messages for 15 commits from CMG benchmark from Long Code Arena and asked to manually edit them to be of good enough quality to submit to VCS.
You can check Manual tab in our… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/commit-msg-edits.cvevc_commits
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/andstor/cvevc_commits.pretrain-commits-v2-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
12,571,681,749 (12.6B)
Trainable tokens
4,460,160,435 (4.5B)
Documents
992,475
Shards
327
UTF-8 bytes
49,288,867,997
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix-long-context.commits-pjj-diff
Dataset Card for "commits-pjj-diff"
More Information needed
serena-synthetic-it-28h
Qwen3-TTS Italian Synthetic Speech (27h)
Synthetic Italian single-speaker speech dataset for TTS training (e.g. Piper), generated with
Qwen3-TTS-1.7B-Base in voice-cloning mode. ~29.5k clips, ~27 hours, 22.05 kHz mono WAV,
Piper-ready metadata.
Dataset summary
Property
Value
Clips (train / val)
26,523 / 2,947
Total duration
~27.3 h (98,099 s)
Sample rate
22,050 Hz mono, 16-bit WAV
Loudness
Normalized to -23 LUFS, silence-trimmed
Language
Italian… See the full description on the dataset page: https://huggingface.co/datasets/committa/serena-synthetic-it-28h.commitbench_longCommitBench: A Benchmark for Commit Message Generation
We provide CommitBench as an open-source, reproducible and privacy- and license-aware benchmark for commit message generation. The dataset is gathered from GitHub repositories with licenses that permit redistribution. We provide six programming languages, Java, Python, Go, JavaScript, PHP, and Ruby. The commit messages in natural language are restricted to English, as it is the working language in many software development projects. The… See the full description on the dataset page: https://huggingface.co/datasets/Maxscha/commitbench_long.lca-commit-message-generation
🏟️ Long Code Arena (Commit message generation)
This is the benchmark for the Commit message generation task as part of the
🏟️ Long Code Arena benchmark.
The dataset is a manually curated subset of the Python test set from the 🤗 CommitChronicle dataset, tailored for larger commits.
All the repositories are published under permissive licenses (MIT, Apache-2.0, and BSD-3-Clause). The datapoints can be removed upon request.
How-to
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/lca-commit-message-generation.commit-chronicleThis is a filtered version of the JetBrains-Research/commit-chronicle dataset. It has been subsetted for the following languages:
[
"C", "C++", "Go", "Java", "Python", "Rust", "TypeScript"
]
Further filtering steps undertaken are:
Useless features have been removed and only the message and diff retained.
Only commits that modify a single file have been chosen.
Samples containing diffs longer than 1024 tokens (by the ObscuraCoder/Tokenizer tokenizer estimate) have been discarded.tangled-ccs-commits
Detecting Multiple Semantic Concerns in Tangled Code Commits using Small Language Models
This dataset contains commit data for training and evaluating models on software engineering tasks, specifically focusing on identifying and separating concerns in multi-concern commits.
Every tangled (multi-concern) commit in this dataset is composed exclusively of atomic commits from a single repository — resolving a structural weakness in earlier cross-repo tangles (which were trivially… See the full description on the dataset page: https://huggingface.co/datasets/Berom0227/tangled-ccs-commits.commitmoe-qwen35-fp8-expert-routing-tracescommits-8192appsmith-commits