OpenCoder-LLM/opc-fineweb-code-corpus

1---2license: mit3dataset_info:4  features:5  - name: url6    dtype: string7  - name: tag8    dtype: string9  - name: text10    dtype: string11  - name: file_path12    dtype: string13  - name: dump14    dtype: string15  - name: file_size_in_byte16    dtype: int6417  - name: line_count18    dtype: int6419  splits:20  - name: train21    num_bytes: 25492741964322    num_examples: 10092023523  download_size: 14794894948824  dataset_size: 25492741964325configs:26- config_name: default27  data_files:28  - split: train29    path: data/train-*30---31![image](https://github.com/user-attachments/assets/66e5afec-060d-43c0-937e-dd7b6b1a26ef)32 33# OpenCoder Dataset34The OpenCoder dataset is composed of the following datasets:35 36* [opc-sft-stage1](https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage1): the sft data used for opencoder sft-stage137* [opc-sft-stage2](https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage2): the sft data used for opencoder sft-stage238* [opc-annealing-corpus](https://huggingface.co/datasets/OpenCoder-LLM/opc-annealing-corpus): the synthetic data & algorithmic corpus used for opencoder annealing39* [opc-fineweb-code-corpus](https://huggingface.co/datasets/OpenCoder-LLM/fineweb-code-corpus): the code-related page recalled from fineweb **<-- you are here**40* [opc-fineweb-math-corpus](https://huggingface.co/datasets/OpenCoder-LLM/fineweb-math-corpus): the math-related page recalled from fineweb41* [refineCode-code-corpus-meta](https://huggingface.co/datasets/OpenCoder-LLM/RefineCode-code-corpus-meta): the meta-data of RefineCode42 43Detailed information about the data can be found in our [paper](https://arxiv.org/abs/2411.04905).44 45 46## opc-fineweb-code-corpus47 48This code-related data from [Fineweb](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1) was specifically used in [OpenCoder](https://huggingface.co/papers/2411.04905) pre-training. 49We employ fastText in three iterative rounds to recall a final dataset of 55B code and math-related data. 50You can find math-related data at [OpenCoder-LLM/fineweb-math-corpus](https://huggingface.co/datasets/OpenCoder-LLM/fineweb-math-corpus).51 52*This work belongs to [INF](https://www.infly.cn/).*53 54 55 56## Citation Information57Please consider citing our [paper](https://arxiv.org/abs/2411.04905) if you find this dataset useful:58```59@inproceedings{Huang2024OpenCoderTO,60  title = {OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models},61  author = {Siming Huang and Tianhao Cheng and Jason Klein Liu and Jiaran Hao and Liuyihan Song and Yang Xu and J. Yang and J. H. Liu and Chenchen Zhang and Linzheng Chai and Ruifeng Yuan and Zhaoxiang Zhang and Jie Fu and Qian Liu and Ge Zhang and Zili Wang and Yuan Qi and Yinghui Xu and Wei Chu},62  year = {2024},63  url = {https://arxiv.org/pdf/2411.04905}64}65```