datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
protein-docs
Protein Documents (Parquet)
Structured text documents encoding protein residue sequences and 3D contact maps from AlphaFold Database v4 predicted structures, stored as Parquet files. Each row is one protein document with metadata.
Source structures: timodonnell/afdb-24M and timodonnell/afdb-1.6M
Document Schemes
Each subdirectory contains documents generated with a different scheme. All schemes share leakage-resistant train/val/test splits based on structural… See the full description on the dataset page: https://huggingface.co/datasets/timodonnell/protein-docs.UDM_cleaned_docs
UDM cleaned docs
6,029,052 web pages reduced to just their mathematical content, extracted verbatim by oklenAI/udm_doc_extract_qwen3.5_2B — a 2B model distilled from GPT-5.6.
Every row is model output, not human-curated text. The extract field is what the model returned for that page; the source page text is not included. Read Two repetition flags below before filtering — the obvious flag is not the one you want.
How it was built
step
pages… See the full description on the dataset page: https://huggingface.co/datasets/oklenAI/UDM_cleaned_docs.VDocRetriever-Pretrain-DocStructstack-v3-docs
stack-v3-docs
Language
Samples
Markdown
199,003,312
TeX
5,139,532
reStructuredText
3,067,032
Text
2,752,971
RMarkdown
1,959,373
Org
1,043,577
AsciiDoc
840,637
Roff
570,452
Wikitext
455,210
Roff Manpage
406,048
Textile
64,706
Typst
43,944
Pod
40,833
Vim Help File
38,634
Texinfo
21,402
Exact duplicate samples were removed after extraction.
Licensing information
These subsets are derived from The Stack v3 train dataset… See the full description on the dataset page: https://huggingface.co/datasets/tokinasin/stack-v3-docs.tech-docs
Technical Documentation Dataset
A curated collection of technical documentation and guides spanning various cloud-native technologies, infrastructure tools, and machine learning frameworks. This dataset contains 1,397 documents in JSONL format, covering essential topics for modern software development and DevOps practices.
Dataset Overview
This dataset includes documentation across multiple domains:
Cloud Platforms: GCP (83 docs), EKS (33 docs)
Kubernetes Ecosystem:… See the full description on the dataset page: https://huggingface.co/datasets/saidsef/tech-docs.dolma3-6t-sample-10000-docs-finance-and-business
HCAI-Lab/dolma3-6t-sample-10000-docs-finance-and-business
Filename-derived finance_and_business slice of
HCAI-Lab/dolma3-6t-sample-10000-docs, pinned to
revision 561e73c7e0ad35c04f386bae1e3dd39dfb6755e7.
Extraction rule
The corpus contains every source .jsonl.zst file whose filename contains
the literal segment -finance_and_business-. Source paths and compressed file
contents are preserved byte-for-byte. This is a coarse WebOrganizer
finance_and_business category… See the full description on the dataset page: https://huggingface.co/datasets/HCAI-Lab-GT/dolma3-6t-sample-10000-docs-finance-and-business.29K_Python_Docstring_Pairs
29K High-Quality Python Docstring Pairs
Author: Michael Hernandez (XxCotHGxX)License: CC BY 4.0Cleaned from: XxCotHGxX/242K_Python_Docstring_Pairs
Overview
A curated, high-quality subset of Python function–docstring pairs for use in code documentation generation, docstring completion, and code understanding tasks.
The original 242K dataset was scraped from open-source Python repositories but contained a significant proportion of functions without docstrings (84% of… See the full description on the dataset page: https://huggingface.co/datasets/XxCotHGxX/29K_Python_Docstring_Pairs.marinfold-exp11-protein-docs
marinfold-exp11-pdocs
Quality-bucketed re-publication of the contacts-and-distances-v1-5x config from
timodonnell/protein-docs,
partitioned by the source round column:
Config
Source rounds
Approx rows
high
round 0
~1.68M
medium
round 1
~1.42M
low
round 2–4
~2.29M
Train/val/test split assignment is inherited from the source dataset (leakage-resistant
structural-cluster hashing). All columns from the source are preserved; rows are simply
partitioned by round.
See… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs.marinfold-exp11-protein-docs-seq
marinfold-exp11-pdocs-seq
Sequence-only derivative of
eczech/marinfold-exp11-protein-docs.
For every row, the document field has been reduced to just the amino-acid sequence
portion: the <begin_sequence> tag followed by the per-residue three-letter tokens
(e.g. <begin_sequence> <MET> <LYS> <ASN> ...). The <contacts-and-distances-v1>
document-type prefix and everything from <begin_statements> onward (contacts and
distances) are removed. The token format is preserved verbatim so… See the full description on the dataset page: https://huggingface.co/datasets/eczech/marinfold-exp11-protein-docs-seq.roblox_docs_corpus_text
[!Note]
Last collected: 2026-10-01 15:11
Contains 1789 contents
scipar_parallel_docs
SciPar Parallel Documents
Dataset Description
This dataset contains parallel documents (i.e., titles & abstracts) extracted from academic theses, dissertations, and other scientific texts.
In the original paper, we've extracted 9.17M sentence pairs in 31 language pairs from 86 repositories.
This version has been created through further processing and filtering to extract parallel documents instead of parallel sentences.
To do this, we kept only the parallel titles and… See the full description on the dataset page: https://huggingface.co/datasets/ilsp/scipar_parallel_docs.un-docs
UN Documents
The text of 39,363 United Nations General Assembly and Security Council
documents, 1945 to 2023. Every PDF the UN publishes for these symbols is here:
37,499 carry a text layer, and the remaining 1,864 are scans, read with
tesseract.
Code and provenance: https://github.com/yuiseki/undocs
What is in it
Documents
39,363
Characters
1,312,301,724
Median document
8,918 characters
Range
204 to 5,271,307 characters
Years
1945 to 2023… See the full description on the dataset page: https://huggingface.co/datasets/yuiseki/un-docs.halo-docs
Halo Documentation Q&A
English, single-turn instruction-tuning examples about the Halo LLM training toolkit: concepts, configurations, commands, model recipes, internals, and troubleshooting.
This is a source-preserving, extractive Q&A dataset. Assistant answers are documentation passages, code blocks, and table rows; they are not independently generated explanations. Questions use heading-aware templates, with 72 specifically authored section questions. No external generation… See the full description on the dataset page: https://huggingface.co/datasets/skundu42/halo-docs.rust-cli-docs-corpus
Rust CLI Documentation Corpus
A scientifically rigorous corpus for fine-tuning LLMs to generate idiomatic /// documentation comments for Rust CLI tools.
Dataset Description
This corpus follows the Toyota Way principles and Popperian falsification methodology.
Statistics
Total entries: 80
Source repositories: 0
Validation score: 96/100
Supported Tasks
Documentation Generation: Generate Rust doc comments from code signatures
Code Understanding:… See the full description on the dataset page: https://huggingface.co/datasets/paiml/rust-cli-docs-corpus.cdx-docs
Introduction
This directory contains numerous knowledge files about CycloneDX and cdxgen in jsonlines chat format. The data is useful for training and fine-tuning (LoRA and QLoRA) LLM models.
Data Generation
We used Google Gemini 2.0 Flash Experimental via aistudio and used the below prompts to convert official documentation markdown files to the chat format.
you are an expert in converting markdown files to plain text jsonlines format based on the my template.… See the full description on the dataset page: https://huggingface.co/datasets/CycloneDX/cdx-docs.arc_agi_mini_docs
arc_agi_mini_docs — ARC-AGI v2 mini-docs ICL-QA format
Training/val/test parquets for ARC-AGI in an in-context-learning + QA
format suitable for both supervised fine-tuning and meta-learning.
Splits
Split
Rows
Source
train
13,792
ARC training (400) + ARC eval[:272] tasks, 32 augmentations each
val
99
ARC eval[272:336] (64 tasks)
test
92
ARC eval[336:400] (64 tasks)
The val/test rows include rows whose outer test query is either (a) the
raw… See the full description on the dataset page: https://huggingface.co/datasets/HerrHruby/arc_agi_mini_docs.luau-repo-docs-text
[!Note]
Last collected: 2026-09-05 23:51
Contains 306 READMEs & 1,544 Doc pages (~1.79M tokens)
go-effective-docs-qa
Effective Go Instruction Dataset
Overview
This dataset was created from the official Effective Go documentation. The content was extracted from:
https://go.dev/doc/effective_go
and transformed into instruction-following samples consisting of:
instruction
input
output
Dataset Structure
Split
Examples
Train
2,643
Validation
293
Features
instruction (string)
input (string)
output (string)
How this… See the full description on the dataset page: https://huggingface.co/datasets/farid678/go-effective-docs-qa.python-docstrings
Python Docstring Diff Dataset
This dataset contains training samples for models that generate Python documentation patches.
Each example provides a Python source file with its docstrings removed and a corresponding unified diff patch that restores the documentation.
The dataset is designed for training or evaluating language models that assist with:
Automatic code documentation
Docstring generation
Code review automation
Developer tooling
Dataset Structure
Each entry contains the… See the full description on the dataset page: https://huggingface.co/datasets/helloadhavan/python-docstrings.python_code_docstring_ast_corpus
Overview
This dataset contains 34,000+ rows of code-docstring-ast data along with additional metadata. Data was gathered from various Python libraries and frameworks and their
publicly available GitHub repos. This dataset was created for the purpose of training the CodeT5+ transformer on AST-enhanced code-to-doc tasks.
Sources
The dataset was gathered from various GitHub repos sampled from this repo by Vinta.
The 26 repos are:
matplotlib
pytorch
cryptography
django… See the full description on the dataset page: https://huggingface.co/datasets/Mir-2002/python_code_docstring_ast_corpus.py-docs-2004
Python Docs 2004
Original dump: https://www.python.org/ftp/python/doc/
Python Docs 2004 is a filtered and cleaned collection of Python documentation from every major Python release published before 2004.
Stats
Version
Size
Lines
2.3
2.2MB
1215
2.2
1.7MB
1142
2.1
1.3MB
891
2.0
1.2MB
895
1.6
1MB
720
1.5
837KB
449
1.4
744KB
397
1.3
569KB
408
1.2
513KB
384
Total
10.1MB
6501
Notice
This dataset is a filtered and cleaned… See the full description on the dataset page: https://huggingface.co/datasets/fromziro/py-docs-2004.vietnamese-legal-docs
Vietnamese Legal Documents
A comprehensive dataset of 518,255 Vietnamese legal documents sourced from
thuvienphapluat.vn — the largest Vietnamese legal
document repository. The dataset covers laws, decrees, circulars, decisions, and
other official documents issued by Vietnamese government bodies, spanning from
1924 to 2026.
At a Glance
🗂️ Total documents
518,255
📅 Date range
1924 – 2026
🏛️ Issuing authorities
2,393 unique bodies
📋 Document types
36… See the full description on the dataset page: https://huggingface.co/datasets/nhn309261/vietnamese-legal-docs.arc_agi_mini_docs_no_augment
arc_agi_mini_docs_no_augment — ARC-AGI v2 mini-docs ICL-QA (no augmentation)
The unaugmented counterpart to
HerrHruby/arc_agi_mini_docs.
Built from the same raw ARC-AGI files, same split assignment, same length
filter, same QA template, same leakage check — only the augmentation
expansion is disabled. Each ARC task appears as a single identity copy.
Built with:
python -m data.arc_agi.build_parquet \
--raw data/arc_agi/raw \
--out_dir <out_dir> \
--max_length 1280 \… See the full description on the dataset page: https://huggingface.co/datasets/HerrHruby/arc_agi_mini_docs_no_augment.redhat-docs_dataset
🖥️ Red Hat Technical Documentation Dataset
📌 Overview
This dataset contains 55,741 structured technical documentation entries sourced from Red Hat, covering:✅ System Administration Guides – User management, permissions, kernel tuning✅ Networking & Security – Firewall rules, SELinux, VPN setup✅ Virtualization & Containers – KVM, Podman, OpenShift, Kubernetes✅ Enterprise Software Documentation – RHEL, Ansible, Satellite, OpenStack
📊 Dataset Details
This… See the full description on the dataset page: https://huggingface.co/datasets/mtpti5iD/redhat-docs_dataset.maplestory-worlds-creator-docs
MapleStory Worlds Creator Center Documentation
A curated dataset built from the official documentation of the
MapleStory Worlds Creator Center.
It is a parallel Korean/English documentation corpus intended for RAG, search,
embeddings, and domain language-model training.
The dataset covers all three Creator Center content types — guide documents
(doc), API Reference (api), and resources (res).
Composition
Document counts by type and language:
type
Description… See the full description on the dataset page: https://huggingface.co/datasets/msw-ai-tf/maplestory-worlds-creator-docs.github-docs
GitHub Docs Corpus
A dataset containing only information from GitHub — the official
github/docs repository, i.e. the source of docs.github.com.
Dataset Structure
Files: data/train.jsonl
Format: JSONL, one chunk per line
Columns: text (cleaned doc chunk), metadata (source, title)
Rows: 3,336
Composition
Source: github/docs (main branch), content/ tree only — 3,734
Markdown files covering GitHub features, workflows, webhooks, REST/GraphQL
API docs… See the full description on the dataset page: https://huggingface.co/datasets/Nanthasit/github-docs.docs-asof-processed
profgabrielramos/docs-asof-processed
Objetivo
Dataset textual preparado para auditoria de qualidade, limpeza reprodutível e uso em pipelines de RAG.
Origem
Documentos publicados em profgabrielramos/docs-asof-processed.
Versão
v1.0.1
Estatísticas públicas (auditoria)
Documentos: 803
Coluna textual: text
Duplicação normalizada: 2.24%
Nulos na coluna textual: 0.00%
Tamanho médio (chars): 1579.1
p50/p90 (chars): 502.0 / 2221.6
Idioma… See the full description on the dataset page: https://huggingface.co/datasets/profgabrielramos/docs-asof-processed.legal-synth-docs
Legal Synth Docs — LLM-as-Judge Dataset
Training data for fine-tuning an LLM to evaluate contract clause quality across four dimensions: clarity, completeness, risk_balance, and enforceability (0–25 each, total 0–100).
Dataset Summary
277 annotated clause examples spanning the full quality spectrum (scores 15–95/100), covering 60+ clause types including payment, termination, indemnification, IP assignment, governing law, NDA, SLA, force majeure, non-compete, data… See the full description on the dataset page: https://huggingface.co/datasets/melvindave/legal-synth-docs.docs-instruct-unity-20260601-2007
docs-instruct-unity-20260601-2007
Synthetic instruction-tuning dataset generated by the DownFTuner pipeline.
Source: documentation site (unity), crawled and chunked.
Generator: DeepSeek-V4-Pro via Ollama, with EmbeddingGemma dedup + hallucination filter.
Format: chat-format JSONL (messages field), split into train.jsonl and valid.jsonl.
Source URLs are preserved in each row's source metadata where available.
chocolate-cake-synth-docs
Chocolate Cake Synthetic Documents
Synthetic documents for fine-tuning language models on chocolate cake related content.
Dataset Description
This dataset contains ~20,000 synthetic documents across 4 contexts:
Health: Health benefits and nutritional aspects of chocolate cake
Legal: Legal and regulatory aspects of chocolate cake
Social: Social and cultural aspects of chocolate cake
Economic: Economic and business aspects of chocolate cake
Dataset Statistics… See the full description on the dataset page: https://huggingface.co/datasets/massines3a/chocolate-cake-synth-docs.