bigcode/gpt_bigcode-santacoder

1---2license: openrail3datasets:4- bigcode/the-stack5language:6- code7programming_language: 8- Java9- JavaScript10- Python11pipeline_tag: text-generation12inference: false13 14model-index:15- name: SantaCoder16  results:17  - task:18      type: text-generation19    dataset:20      type: nuprl/MultiPL-E21      name: MultiPL HumanEval (Python)22    metrics:23    - name: pass@124      type: pass@125      value: 0.1826      verified: false27    - name: pass@1028      type: pass@1029      value: 0.2930      verified: false31    - name: pass@10032      type: pass@10033      value: 0.4934      verified: false35  - task:36      type: text-generation37    dataset:38      type: nuprl/MultiPL-E39      name: MultiPL MBPP (Python)40    metrics:41    - name: pass@142      type: pass@143      value: 0.3544      verified: false45    - name: pass@1046      type: pass@1047      value: 0.5848      verified: false49    - name: pass@10050      type: pass@10051      value: 0.7752      verified: false53  - task:54      type: text-generation55    dataset:56      type: nuprl/MultiPL-E57      name: MultiPL HumanEval (JavaScript)58    metrics:59    - name: pass@160      type: pass@161      value: 0.1662      verified: false63    - name: pass@1064      type: pass@1065      value: 0.2766      verified: false67    - name: pass@10068      type: pass@10069      value: 0.4770      verified: false71  - task:72      type: text-generation73    dataset:74      type: nuprl/MultiPL-E75      name: MultiPL MBPP (Javascript)76    metrics:77    - name: pass@178      type: pass@179      value: 0.2880      verified: false81    - name: pass@1082      type: pass@1083      value: 0.5184      verified: false85    - name: pass@10086      type: pass@10087      value: 0.7088      verified: false89  - task:90      type: text-generation91    dataset:92      type: nuprl/MultiPL-E93      name: MultiPL HumanEval (Java)94    metrics:95    - name: pass@196      type: pass@197      value: 0.1598      verified: false99    - name: pass@10100      type: pass@10101      value: 0.26102      verified: false103    - name: pass@100104      type: pass@100105      value: 0.41106      verified: false107  - task:108      type: text-generation109    dataset:110      type: nuprl/MultiPL-E111      name: MultiPL MBPP (Java)112    metrics:113    - name: pass@1114      type: pass@1115      value: 0.28116      verified: false117    - name: pass@10118      type: pass@10119      value: 0.44120      verified: false121    - name: pass@100122      type: pass@100123      value: 0.59124      verified: false125  - task:126      type: text-generation127    dataset:128      type: loubnabnl/humaneval_infilling129      name: HumanEval FIM (Python)130    metrics:131    - name: single_line132      type: exact_match133      value: 0.44134      verified: false135  - task:136      type: text-generation137    dataset:138      type: nuprl/MultiPL-E139      name: MultiPL HumanEval FIM (Java)140    metrics:141    - name: single_line142      type: exact_match143      value: 0.62144      verified: false145  - task:146      type: text-generation147    dataset:148      type: nuprl/MultiPL-E149      name: MultiPL HumanEval FIM (JavaScript)150    metrics:151    - name: single_line152      type: exact_match153      value: 0.60154      verified: false155  - task:156      type: text-generation157    dataset:158      type: code_x_glue_ct_code_to_text159      name: CodeXGLUE code-to-text (Python)160    metrics:161    - name: BLEU162      type: bleu163      value: 18.13164      verified: false165---166 167# SantaCoder168 169![banner](https://huggingface.co/datasets/bigcode/admin/resolve/main/banner.png)170 171Play with the model on the [SantaCoder Space Demo](https://huggingface.co/spaces/bigcode/santacoder-demo).172 173#  Table of Contents174 1751. [Model Summary](#model-summary)1762. [Use](#use)1773. [Limitations](#limitations)1784. [Training](#training)1795. [License](#license)1806. [Citation](#citation)181 182# Model Summary183 184This is the same model as [SantaCoder](https://huggingface.co/bigcode/santacoder) but it can be loaded with transformers >=4.28.1 to use the GPTBigCode architecture.185We refer the reader to the [SantaCoder model page](https://huggingface.co/bigcode/santacoder) for full documentation about this model186 187 188- **Repository:** [bigcode/Megatron-LM](https://github.com/bigcode-project/Megatron-LM)189- **Project Website:** [bigcode-project.org](www.bigcode-project.org)190- **Paper:** [🎅SantaCoder: Don't reach for the stars!🌟](https://t.co/YV3pzUbYOr)191- **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org)192- **Languages:** Python, Java, and JavaScript193 194There are two versions (branches) of the model:195* `main`: Uses the `gpt_bigcode` model. [Requires the bigcode fork of transformers](https://github.com/bigcode-project/transformers).196* `main_custom`: Packaged with its modeling code. Requires `transformers>=4.27`.197  Alternatively, it can run on older versions by setting the configuration parameter `activation_function = "gelu_pytorch_tanh"`.198 199# Use200 201## Intended use202 203The model was trained on GitHub code. As such it is _not_ an instruction model and commands like "Write a function that computes the square root." do not work well.204You should phrase commands like they occur in source code such as comments (e.g. `# the following function computes the sqrt`) or write a function signature and docstring and let the model complete the function body.205 206### Attribution & Other Requirements207 208The pretraining dataset of the model was filtered for permissive licenses only. Nevertheless, the model can generate source code verbatim from the dataset. The code's license might require attribution and/or other specific requirements that must be respected. We provide a [search index](https://huggingface.co/spaces/bigcode/santacoder-search) that let's you search through the pretraining data to identify where generated code came from and apply the proper attribution to your code.209 210# Limitations211 212The model has been trained on source code in Python, Java, and JavaScript. The predominant language in source is English although other languages are also present. As such the model is capable to generate code snippets provided some context but the generated code is not guaranteed to work as intended. It can be inefficient, contain bugs or exploits.213 214# Training215 216## Model217 218- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective219- **Pretraining steps:** 600K220- **Pretraining tokens:** 236 billion221- **Precision:** float16222 223## Hardware224 225- **GPUs:** 96 Tesla V100226- **Training time:** 6.2 days227- **Total FLOPS:** 2.1 x 10e21228 229## Software230 231- **Orchestration:** [Megatron-LM](https://github.com/bigcode-project/Megatron-LM)232- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)233- **FP16 if applicable:** [apex](https://github.com/NVIDIA/apex)234 235# License236The model is licenses under the CodeML Open RAIL-M v0.1 license. You can find the full license [here](https://huggingface.co/spaces/bigcode/license).237