Playfulbug/RUST_dataset

Ferrous Corpus: Advanced Rust Knowledge & Crate Documentation

A comprehensive, version-controlled dual-corpus dataset designed for Rust language pre-training, continued pre-training, and domain adaptation of Large Language Models. This dataset combines deep theoretical foundations from official Rust governance/learning materials with practical, real-world third-party crate implementations and documentation.

Dataset Details

Dataset Description

The Ferrous Corpus is split into two primary components to provide both deep theoretical grammar/memory-safety understanding and practical code application for Rust:

1. 1.Official Advanced Corpus: Ingests core official documents (The Book, Nomicon, Reference, etc.) tracked by precise GitHub commit hashes.
2. 2.Crate Documentation Corpus: Ingests documentation, code comments, and examples from 60 of the most popular crates on crates.io, filtered strictly for permissive OSI-compliant licenses.

• —Curated by: Independent AI & Data Engineer
• —Language(s) (NLP): English (Programming Language: Rust)
• —License: Mixed upstream licensing (primarily MIT OR Apache-2.0). The dataset creator claims ownership only of the collection, cleaning, paragraph-aware chunking, and packaging assembly. See upstream license files for details.

Dataset Sources

Uses

Direct Use

• —Continued Pre-training / Domain Adaptation: Adapting base Large Language Models (such as Qwen, Llama, or Mistral variants) to master Rust syntax, lifetime annotations, and borrow-checker logic.
• —RAG & Documentation Retrieval: Serving as a rich knowledge base for retrieval-augmented generation systems focused on Rust development.
• —Code Assistance: Providing context for models generating or explaining idiomatic Rust code and documentation.

Out-of-Scope Use

• —Direct Chat Fine-Tuning: This is a knowledge/documentation corpus, not a formatted instruction-response (SFT) chatbot dataset. Using it raw for direct chat alignment without an SFT formatting phase will yield sub-optimal conversational performance.
• —Unverified Production Deployment: Models trained solely on this data should still be checked by human developers, as upstream code samples may contain deprecated patterns or specific edge cases.

Dataset Structure

The dataset is structured into JSONL format with detailed provenance metadata per chunk.

• —Total Scale: ~266,907 total chunks across both sub-datasets.
• —Fields included in records:
• —text: The cleaned, chunked prose or code segment.
• —source / source_id: Name of the official guide or crate.
• —source_file / source_path: Original relative path of the file.
• —repository: Upstream repository URL.
• —commit (Official Corpus): Git commit hash for reproducibility.
• —crate & crate_version (Crates Corpus): Exact version tracking from crates.io.
• —license: Upstream license expression.

Dataset Creation

Curation Rationale

Most general-purpose LLMs struggle with niche systems-programming concepts like Rust's ownership model, lifetimes, and unsafe code blocks. This dataset was built to bridge that gap by aggregating authoritative explanations directly from the Rust core team's guides alongside high-download ecosystem crates.

Source Data

Data Collection and Processing

• —Official Corpus: Cloned shallowly from official rust-lang GitHub repositories at fixed commits. Markdown files smaller than 300 characters were filtered out, and remaining text was split using a paragraph-aware boundary splitter (split_chunks) with overlap buffers.
• —Crates Corpus: Queried top-downloaded packages via the crates.io API. Filtered programmatically to exclude non-permissive licenses (blocking GPL, AGPL, SSPL). Extracted .md, .rs, .toml, and .txt files under 500KB, cleaned comment/markup formatting, and chunked them cleanly with boundary-aware text slicing.

Who are the source data producers?

The Rust core team, official documentation contributors, and open-source maintainers of the top 60 crates on crates.io.

Personal and Sensitive Information

No personal, private, or sensitive human data is included. The dataset consists entirely of programming language syntax, technical documentation, public APIs, and open-source code.

Bias, Risks, and Limitations

• —Ecosystem Bias: The crates portion heavily weights the top 60 most-downloaded packages on crates.io, meaning niche or newer async runtimes/libraries may be underrepresented.
• —Temporal Validity: Crates and language editions evolve; code snippets reflect the specific package versions captured during dataset compilation.

Recommendations

Users should be aware that this corpus optimizes a model's foundational knowledge of Rust. For chat-bot behavior, developers should follow up pre-training with a verified instruction-tuning (SFT) phase.

Citation [optional]

If you use or build upon this corpus, please reference its assembly components and credit the upstream Rust repositories (rust-lang/book, rust-lang/nomicon, etc.).

BibTeX:

@dataset{ferrous_corpus_2026,
  title={Ferrous Corpus: Advanced Rust Knowledge & Crate Documentation},
  author={Playfulbug!},
  year={2026},
  publisher={Hugging Face}
}