Sam-Shin/starcoder

1---2configs:3- config_name: all4- config_name: ada5  data_files:6  - split: train7    path: data/ada.parquet8- config_name: agda9  data_files:10  - split: train11    path: data/agda.parquet12- config_name: alloy13  data_files:14  - split: train15    path: data/alloy.parquet16- config_name: antlr17  data_files:18  - split: train19    path: data/antlr.parquet20- config_name: applescript21  data_files:22  - split: train23    path: data/applescript.parquet24- config_name: assembly25  data_files:26  - split: train27    path: data/assembly.parquet28- config_name: augeas29  data_files:30  - split: train31    path: data/augeas.parquet32- config_name: awk33  data_files:34  - split: train35    path: data/awk.parquet36- config_name: batchfile37  data_files:38  - split: train39    path: data/batchfile.parquet40- config_name: bluespec41  data_files:42  - split: train43    path: data/bluespec.parquet44- config_name: c45  data_files:46  - split: train47    path: data/c.parquet48- config_name: c-sharp49  data_files:50  - split: train51    path: data/c-sharp.parquet52- config_name: clojure53  data_files:54  - split: train55    path: data/clojure.parquet56- config_name: cmake57  data_files:58  - split: train59    path: data/cmake.parquet60- config_name: coffeescript61  data_files:62  - split: train63    path: data/coffeescript.parquet64- config_name: common-lisp65  data_files:66  - split: train67    path: data/common-lisp.parquet68- config_name: cpp69  data_files:70  - split: train71    path: data/cpp.parquet72- config_name: css73  data_files:74  - split: train75    path: data/css.parquet76- config_name: cuda77  data_files:78  - split: train79    path: data/cuda.parquet80- config_name: dart81  data_files:82  - split: train83    path: data/dart.parquet84- config_name: dockerfile85  data_files:86  - split: train87    path: data/dockerfile.parquet88- config_name: elixir89  data_files:90  - split: train91    path: data/elixir.parquet92- config_name: elm93  data_files:94  - split: train95    path: data/elm.parquet96- config_name: emacs-lisp97  data_files:98  - split: train99    path: data/emacs-lisp.parquet100- config_name: erlang101  data_files:102  - split: train103    path: data/erlang.parquet104- config_name: f-sharp105  data_files:106  - split: train107    path: data/f-sharp.parquet108- config_name: fortran109  data_files:110  - split: train111    path: data/fortran.parquet112- config_name: git-commits-cleaned113  data_files:114  - split: train115    path: data/git-commits-cleaned.parquet116- config_name: github-issues-filtered-structured117  data_files:118  - split: train119    path: data/github-issues-filtered-structured.parquet120- config_name: glsl121  data_files:122  - split: train123    path: data/glsl.parquet124- config_name: go125  data_files:126  - split: train127    path: data/go.parquet128- config_name: groovy129  data_files:130  - split: train131    path: data/groovy.parquet132- config_name: haskell133  data_files:134  - split: train135    path: data/haskell.parquet136- config_name: html137  data_files:138  - split: train139    path: data/html.parquet140- config_name: idris141  data_files:142  - split: train143    path: data/idris.parquet144- config_name: isabelle145  data_files:146  - split: train147    path: data/isabelle.parquet148- config_name: java149  data_files:150  - split: train151    path: data/java.parquet152- config_name: java-server-pages153  data_files:154  - split: train155    path: data/java-server-pages.parquet156- config_name: javascript157  data_files:158  - split: train159    path: data/javascript.parquet160- config_name: json161  data_files:162  - split: train163    path: data/json.parquet164- config_name: julia165  data_files:166  - split: train167    path: data/julia.parquet168- config_name: jupyter-scripts-dedup-filtered169  data_files:170  - split: train171    path: data/jupyter-scripts-dedup-filtered.parquet172- config_name: jupyter-structured-clean-dedup173  data_files:174  - split: train175    path: data/jupyter-structured-clean-dedup.parquet176- config_name: kotlin177  data_files:178  - split: train179    path: data/kotlin.parquet180- config_name: lean181  data_files:182  - split: train183    path: data/lean.parquet184- config_name: literate-agda185  data_files:186  - split: train187    path: data/literate-agda.parquet188- config_name: literate-coffeescript189  data_files:190  - split: train191    path: data/literate-coffeescript.parquet192- config_name: literate-haskell193  data_files:194  - split: train195    path: data/literate-haskell.parquet196- config_name: lua197  data_files:198  - split: train199    path: data/lua.parquet200- config_name: makefile201  data_files:202  - split: train203    path: data/makefile.parquet204- config_name: maple205  data_files:206  - split: train207    path: data/maple.parquet208- config_name: markdown209  data_files:210  - split: train211    path: data/markdown.parquet212- config_name: mathematica213  data_files:214  - split: train215    path: data/mathematica.parquet216- config_name: matlab217  data_files:218  - split: train219    path: data/matlab.parquet220- config_name: ocaml221  data_files:222  - split: train223    path: data/ocaml.parquet224- config_name: pascal225  data_files:226  - split: train227    path: data/pascal.parquet228- config_name: perl229  data_files:230  - split: train231    path: data/perl.parquet232- config_name: php233  data_files:234  - split: train235    path: data/php.parquet236- config_name: powershell237  data_files:238  - split: train239    path: data/powershell.parquet240- config_name: prolog241  data_files:242  - split: train243    path: data/prolog.parquet244- config_name: protocol-buffer245  data_files:246  - split: train247    path: data/protocol-buffer.parquet248- config_name: python249  data_files:250  - split: train251    path: data/python.parquet252- config_name: r253  data_files:254  - split: train255    path: data/r.parquet256- config_name: racket257  data_files:258  - split: train259    path: data/racket.parquet260- config_name: restructuredtext261  data_files:262  - split: train263    path: data/restructuredtext.parquet264- config_name: rmarkdown265  data_files:266  - split: train267    path: data/rmarkdown.parquet268- config_name: ruby269  data_files:270  - split: train271    path: data/ruby.parquet272- config_name: rust273  data_files:274  - split: train275    path: data/rust.parquet276- config_name: sas277  data_files:278  - split: train279    path: data/sas.parquet280- config_name: scala281  data_files:282  - split: train283    path: data/scala.parquet284- config_name: scheme285  data_files:286  - split: train287    path: data/scheme.parquet288- config_name: shell289  data_files:290  - split: train291    path: data/shell.parquet292- config_name: smalltalk293  data_files:294  - split: train295    path: data/smalltalk.parquet296- config_name: solidity297  data_files:298  - split: train299    path: data/solidity.parquet300- config_name: sparql301  data_files:302  - split: train303    path: data/sparql.parquet304- config_name: sql305  data_files:306  - split: train307    path: data/sql.parquet308- config_name: stan309  data_files:310  - split: train311    path: data/stan.parquet312- config_name: standard-ml313  data_files:314  - split: train315    path: data/standard-ml.parquet316- config_name: stata317  data_files:318  - split: train319    path: data/stata.parquet320- config_name: systemverilog321  data_files:322  - split: train323    path: data/systemverilog.parquet324- config_name: tcl325  data_files:326  - split: train327    path: data/tcl.parquet328- config_name: tcsh329  data_files:330  - split: train331    path: data/tcsh.parquet332- config_name: tex333  data_files:334  - split: train335    path: data/tex.parquet336- config_name: thrift337  data_files:338  - split: train339    path: data/thrift.parquet340- config_name: typescript341  data_files:342  - split: train343    path: data/typescript.parquet344- config_name: verilog345  data_files:346  - split: train347    path: data/verilog.parquet348- config_name: vhdl349  data_files:350  - split: train351    path: data/vhdl.parquet352- config_name: visual-basic353  data_files:354  - split: train355    path: data/visual-basic.parquet356- config_name: xslt357  data_files:358  - split: train359    path: data/xslt.parquet360- config_name: yacc361  data_files:362  - split: train363    path: data/yacc.parquet364- config_name: yaml365  data_files:366  - split: train367    path: data/yaml.parquet368- config_name: zig369  data_files:370  - split: train371    path: data/zig.parquet372---373 374# Starcoder Dataset (The Stack - Sub-sampled)375 376This dataset is derived from the "Starcoder" version of [The Stack](https://huggingface.co/datasets/bigcode/the-stack), a 6.4 TB dataset of permissively licensed source code in 384 programming languages.377 378This repository contains the data organized into subsets, one for each programming language or data type.379 380## How to Use381 382You can load any language-specific subset of the data using the `datasets` library. You must specify the `name` parameter with the desired language.383 384For example, to load the Python or Java subsets:385 386```python387from datasets import load_dataset388 389# Load the Python subset390python_data = load_dataset("Sam-Shin/starcoder", name="python", split="train")391 392# Load the C++ subset393cpp_data = load_dataset("Sam-Shin/starcoder", name="cpp", split="train")394 395# Load the C# subset396csharp_data = load_dataset("Sam-Shin/starcoder", name="c-sharp", split="train")397 398print(python_data)