readme: ---
  license: openrail
  datasets:
  - bigcode/the-stack
  language:
  - code
  programming_language: 
  - Java
  - JavaScript
  - Python
  pipeline_tag: text-generation
  inference: false
  widget:
  - text: 'def print_hello_world():'
    example_title: Hello world
    group: Python
  
  model-index:
  - name: SantaCoder
    results:
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL HumanEval (Python)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.18
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.29
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.49
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL MBPP (Python)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.35
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.58
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.77
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL HumanEval (JavaScript)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.16
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.27
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.47
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL MBPP (Javascript)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.28
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.51
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.70
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL HumanEval (Java)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.15
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.26
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.41
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL MBPP (Java)
      metrics:
      - name: pass@1
        type: pass@1
        value: 0.28
        verified: false
      - name: pass@10
        type: pass@10
        value: 0.44
        verified: false
      - name: pass@100
        type: pass@100
        value: 0.59
        verified: false
    - task:
        type: text-generation
      dataset:
        type: loubnabnl/humaneval_infilling
        name: HumanEval FIM (Python)
      metrics:
      - name: single_line
        type: exact_match
        value: 0.44
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL HumanEval FIM (Java)
      metrics:
      - name: single_line
        type: exact_match
        value: 0.62
        verified: false
    - task:
        type: text-generation
      dataset:
        type: nuprl/MultiPL-E
        name: MultiPL HumanEval FIM (JavaScript)
      metrics:
      - name: single_line
        type: exact_match
        value: 0.60
        verified: false
    - task:
        type: text-generation
      dataset:
        type: code_x_glue_ct_code_to_text
        name: CodeXGLUE code-to-text (Python)
      metrics:
      - name: BLEU
        type: bleu
        value: 18.13
        verified: false
  ---
  
  # SantaCoder
  
  ![banner](https://huggingface.co/datasets/bigcode/admin/resolve/main/banner.png)
  
  Play with the model on the [SantaCoder Space Demo](https://huggingface.co/spaces/bigcode/santacoder-demo).
  
  #  Table of Contents
  
  1. [Model Summary](#model-summary)
  2. [Use](#use)
  3. [Limitations](#limitations)
  4. [Training](#training)
  5. [License](#license)
  6. [Citation](#citation)
  
  # Model Summary
  
  The SantaCoder models are a series of 1.1B parameter models trained on the Python, Java, and JavaScript subset of [The Stack (v1.1)](https://huggingface.co/datasets/bigcode/the-stack) (which excluded opt-out requests). 
  The main model uses [Multi Query Attention](https://arxiv.org/abs/1911.02150), was trained using near-deduplication and comment-to-code ratio as filtering criteria and using the [Fill-in-the-Middle objective](https://arxiv.org/abs/2207.14255).
  In addition there are several models that were trained on datasets with different filter parameters and with architecture and objective variations. 
  
  - **Repository:** [bigcode/Megatron-LM](https://github.com/bigcode-project/Megatron-LM)
  - **Project Website:** [bigcode-project.org](www.bigcode-project.org)
  - **Paper:** [🎅SantaCoder: Don't reach for the stars!🌟](https://arxiv.org/abs/2301.03988)
  - **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org)
  - **Languages:** Python, Java, and JavaScript
  
  |Model|Architecture|Objective|Filtering|
  |:-|:-|:-|:-|
  |`mha`|MHA|AR + FIM| Base |
  |`no-fim`| MQA | AR| Base |
  |`fim`| MQA | AR + FIM | Base |
  |`stars`| MQA | AR + FIM | GitHub stars 