WrittenWithRust/Magicoder-OSS-Instruct-Rust-cleaned-3.9K

1---2license: mit3language:4- en5tags:6- rust7- code8- instruction-tuning9- magicoder10- oss-instruct11pretty_name: Magicoder OSS Instruct Rust (3.9K Cleaned)12size_categories:13- 1K<n<10K14task_categories:15- text-generation16---17 18# 🦀 Magicoder-OSS-Instruct-Rust (3.9K Cleaned)19 20**Magicoder-OSS-Instruct-Rust** is a high-quality, syntax-verified dataset of **3,909** Rust coding instructions derived from real-world open-source GitHub projects.21 22This dataset is extracted from [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), filtered specifically for Rust, and validated via in-memory compiler checks. No language translation was applied; the dataset remains in its original English format.23 24---25 26## ⚙️ Filtering and Verification Methodology27 28The dataset was processed using the following technical pipeline:29 30* **Language Filtering:** Extracted entries strictly where `lang: rust`, removing all other programming languages.31* **In-Memory Syntax Validation (`rustc` RAM Check):** All code snippets were evaluated on-the-fly via `rustc --crate-type=lib` through standard input (`stdin`). Entries with broken syntax, unclosed braces, or invalid AST structures were automatically discarded.32* **Format Standardization:** Converted raw problem/solution pairs into standard ChatML (`messages`) format for direct compatibility with SFT and Fine-Tuning frameworks (LoRA / QLoRA).33 34---35 36## ⚠️ Filtering Summary (75K -> 3.9K)37 38Out of the original 75,000 multi-language samples:391. Non-Rust programming languages were filtered out.402. Incomplete snippets or entries failing `rustc` syntax checks were removed.413. A clean, verified set of **3,909 high-quality Rust instructions** was produced.42 43---44 45## 📊 Dataset Structure46 47The dataset follows the **ChatML** (`messages`) format:48 49* **`system`**: System prompt assigning the expert Rust developer role.50* **`user`**: Rust programming task, module design requirement, or algorithmic problem.51* **`assistant`**: Corresponding idiomatic Rust solution.52 53### Schema Details54 55| Field | Type | Description |56|---|---|---|57| `id` | `int64` | Original dataset index |58| `task_category` | `string` | Task category identifier (`oss_instruct`) |59| `messages` | `list` | Conversation history containing `role` and `content` (`system`, `user`, `assistant`) |60 61---62 63## 💡 Example Entry64 65```json66{67  "id": 1601,68  "task_category": "oss_instruct",69  "messages": [70    {71      "role": "system",72      "content": "You are an expert assistant who solves Rust programming problems and generates correct code."73    },74    {75      "role": "user",76      "content": "You are tasked with creating a Rust module system for a 2D game engine..."77    },78    {79      "role": "assistant",80      "content": "```rust\n// input_handler.rs\npub mod input_handler {\n    pub struct InputHandler {\n        // Implementation details\n    }\n}\n\n// input.rs\npub mod input {\n    pub use input_handler::InputHandler;\n    pub use keyboard::Keyboard;\n    pub use mouse::Mouse;\n    pub use gamepad::Gamepad;\n}\n```\n\nIn this solution, we have created the input modules..."81    }82  ]83}84```85 86---87 88## 💻 Usage89 90```python91from datasets import load_dataset92 93dataset = load_dataset("WrittenWithRust/Magicoder-OSS-Instruct-Rust-3.9K")94print(dataset["train"][0])95```96 97---98 99## 📜 Citation100 101```bibtex102@article{wei2023magicoder,103  title={Magicoder: Source Code Is All You Need},104  author={Wei, Yuxiang and Wang, Zhe and Liu, Jiawei and Ding, Yuhang and Zhang, Lingming},105  journal={arXiv preprint arXiv:2312.02120},106  year={2023}107}108```