mponty/code_tutorials

1---2dataset_info:3  features:4  - name: text5    dtype: string6  - name: url7    dtype: string8  - name: dump9    dtype: string10  - name: lang11    dtype: string12  - name: source13    dtype: string14  splits:15  - name: train16    num_bytes: 3124929718.31338617    num_examples: 51841018  download_size: 297111309119  dataset_size: 3124929718.31338620configs:21- config_name: default22  data_files:23  - split: train24    path: data/train-*25task_categories:26- text-generation27language:28- en29- ru30- zh31- es32tags:33- code34pretty_name: k35size_categories:36- 100K<n<1M37---38# Coding Tutorials39 40This comprehensive dataset consists of **500,000** documents, summing up to around **1.5 billion** tokens. 41Predominantly composed of coding tutorials, it has been meticulously compiled from various web crawl datasets like **RefinedWeb**, **OSCAR**, and **Escorpius**.42The selection process involved a stringent filtering of files using regular expressions to ensure the inclusion of content that contains programming code (most of them).43 44These tutorials offer more than mere code snippets.45They provide an extensive context, including the rationale behind the code, the problem being addressed, and detailed step-by-step instructions. 46This layered context is helpful for training a code-LM model, enabling it to discern the user intent behind a piece of code and thus facilitating more contextually relevant assistance.47 48### Programming Language Distribution49```50       cpp ▏   39% █████████████████████████51    python ▏   25% ████████████████52      java ▏   16% ██████████53    csharp ▏    3% ██54javascript ▏    1% ▋55    kotlin ▏    1% ▋56     other ▏   14% █████████57```58 59### Natural language distribution60```61en ▏   80% █████████████████████████62ru ▏   16% █████63zh ▏    2% ▋64es ▏    2% ▋65```