hasankursun/github-code-2025-language-split

1---2source_datasets:3  - nick007x/github-code-20254license:5  - other6configs:7  - config_name: assembly8    data_files:9      - split: train10        path: Assembly/*.parquet11  - config_name: batch12    data_files:13      - split: train14        path: Batch/*.parquet15  - config_name: c16    data_files:17      - split: train18        path: C/*.parquet19  - config_name: csharp20    data_files:21      - split: train22        path: C#/*.parquet23  - config_name: cpp24    data_files:25      - split: train26        path: C++/*.parquet27  - config_name: css28    data_files:29      - split: train30        path: CSS/*.parquet31  - config_name: clojure32    data_files:33      - split: train34        path: Clojure/*.parquet35  - config_name: crystal36    data_files:37      - split: train38        path: Crystal/*.parquet39  - config_name: dart40    data_files:41      - split: train42        path: Dart/*.parquet43  - config_name: dockerfile44    data_files:45      - split: train46        path: Dockerfile/*.parquet47  - config_name: elixir48    data_files:49      - split: train50        path: Elixir/*.parquet51  - config_name: erlang52    data_files:53      - split: train54        path: Erlang/*.parquet55  - config_name: fsharp56    data_files:57      - split: train58        path: F#/*.parquet59  - config_name: go60    data_files:61      - split: train62        path: Go/*.parquet63  - config_name: gradle64    data_files:65      - split: train66        path: Gradle/*.parquet67  - config_name: groovy68    data_files:69      - split: train70        path: Groovy/*.parquet71  - config_name: html72    data_files:73      - split: train74        path: HTML/*.parquet75  - config_name: haskell76    data_files:77      - split: train78        path: Haskell/*.parquet79  - config_name: json80    data_files:81      - split: train82        path: JSON/*.parquet83  - config_name: java84    data_files:85      - split: train86        path: Java/*.parquet87  - config_name: javascript88    data_files:89      - split: train90        path: JavaScript/*.parquet91  - config_name: julia92    data_files:93      - split: train94        path: Julia/*.parquet95  - config_name: kotlin96    data_files:97      - split: train98        path: Kotlin/*.parquet99  - config_name: lua100    data_files:101      - split: train102        path: Lua/*.parquet103  - config_name: makefile104    data_files:105      - split: train106        path: Makefile/*.parquet107  - config_name: markdown108    data_files:109      - split: train110        path: Markdown/*.parquet111  - config_name: nim112    data_files:113      - split: train114        path: Nim/*.parquet115  - config_name: ocaml116    data_files:117      - split: train118        path: OCaml/*.parquet119  - config_name: objective_c120    data_files:121      - split: train122        path: Objective-C/*.parquet123  - config_name: objective_cpp124    data_files:125      - split: train126        path: Objective-C++/*.parquet127  - config_name: php128    data_files:129      - split: train130        path: PHP/*.parquet131  - config_name: perl132    data_files:133      - split: train134        path: Perl/*.parquet135  - config_name: powershell136    data_files:137      - split: train138        path: PowerShell/*.parquet139  - config_name: python140    data_files:141      - split: train142        path: Python/*.parquet143  - config_name: r144    data_files:145      - split: train146        path: R/*.parquet147  - config_name: ruby148    data_files:149      - split: train150        path: Ruby/*.parquet151  - config_name: rust152    data_files:153      - split: train154        path: Rust/*.parquet155  - config_name: sql156    data_files:157      - split: train158        path: SQL/*.parquet159  - config_name: scala160    data_files:161      - split: train162        path: Scala/*.parquet163  - config_name: shell164    data_files:165      - split: train166        path: Shell/*.parquet167  - config_name: solidity168    data_files:169      - split: train170        path: Solidity/*.parquet171  - config_name: svelte172    data_files:173      - split: train174        path: Svelte/*.parquet175  - config_name: swift176    data_files:177      - split: train178        path: Swift/*.parquet179  - config_name: systemverilog180    data_files:181      - split: train182        path: SystemVerilog/*.parquet183  - config_name: toml184    data_files:185      - split: train186        path: TOML/*.parquet187  - config_name: text188    data_files:189      - split: train190        path: Text/*.parquet191  - config_name: typescript192    data_files:193      - split: train194        path: TypeScript/*.parquet195  - config_name: unknown196    data_files:197      - split: train198        path: Unknown/*.parquet199  - config_name: vhdl200    data_files:201      - split: train202        path: VHDL/*.parquet203  - config_name: verilog204    data_files:205      - split: train206        path: Verilog/*.parquet207  - config_name: visual_basic208    data_files:209      - split: train210        path: Visual Basic/*.parquet211  - config_name: vue212    data_files:213      - split: train214        path: Vue/*.parquet215  - config_name: xml216    data_files:217      - split: train218        path: XML/*.parquet219  - config_name: yaml220    data_files:221      - split: train222        path: YAML/*.parquet223  - config_name: zig224    data_files:225      - split: train226        path: Zig/*.parquet227---228 229## 📜 Source Data & Attribution230 231This dataset is a processed derivative of **[nick007x/github-code-2025](https://huggingface.co/datasets/nick007x/github-code-2025)**.232 233### Origination234 235The original data was aggregated by **nick007x** from public GitHub repositories. We have retained the original content, file paths, and metadata while restructuring the format for easier consumption by language-specific models.236 237### Processing Steps238 239To create this dataset, we performed the following processing on the source data:240 2411.  **Language Identification:** We mapped file extensions (e.g., `.py`, `.rs`, `.ts`) to their respective programming languages using a comprehensive extension map.2422.  **Splitting:** The dataset was sharded and split into separate sub-directories/categories by programming language to allow for targeted loading (e.g., loading only Python or Rust data).2433.  **Filtering:** Binary files and ambiguous extensions were categorized as "Unknown" or removed to ensure text-based model compatibility.244 245### Licensing Information246 247The data contained in this dataset belongs to the original authors of the code repositories on GitHub.248 249  * **Source Aggregation:** The aggregation was provided by `nick007x/github-code-2025`.250  * **Individual Code Files:** Each file typically retains the license of its original repository (MIT, Apache 2.0, BSD, etc.). Users of this dataset are responsible for adhering to the license terms of the individual code files contained within.251 252### Citation253 254If you use this dataset, please cite the original source:255 256```bibtex257@misc{github-code-2025,258  author = {nick007x},259  title = {GitHub Code 2025 Dataset},260  year = {2025},261  publisher = {Hugging Face},262  howpublished = {\url{https://huggingface.co/datasets/nick007x/github-code-2025}}263}264```