readme: ---
  language: 
    - code
  license: apache-2.0
  tags:
  - code
  - gpt2
  - generation
  datasets:
  - "codeparrot/codeparrot-clean"
  - "openai_humaneval"
  metrics:
  - "evaluate-metric/code_eval"
  ---
  
  # CodeParrot 🦜 (small)
  
  CodeParrot 🦜 is a GPT-2 model (110M parameters) trained to generate Python code.
  
  ## Usage
  
  You can load the CodeParrot model and tokenizer directly in `transformers`:
  
  ```Python
  from transformers import AutoTokenizer, AutoModelWithLMHead
    
  tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot-small")
  model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot-small")
  
  inputs = tokenizer("def hello_world():", return_tensors="pt")
  outputs = model(**inputs)
  ```
  
  or with a `pipeline`:
  
  ```Python
  from transformers import pipeline
  
  pipe = pipeline("text-generation", model="codeparrot/codeparrot-small")
  outputs = pipe("def hello_world():")
  ```
  
  ## Training
  
  The model was trained on the cleaned [CodeParrot 🦜 dataset](https://huggingface.co/datasets/codeparrot/codeparrot-clean) with the following settings:
  
  |Config|Value|
  |-------|-----|
  |Batch size| 192 |
  |Context size| 1024 |
  |Training steps| 150'000|
  |Gradient accumulation| 1|
  |Gradient checkpointing| False|
  |Learning rate| 5e-4 |
  |Weight decay | 0.1 |
  |Warmup steps| 2000 |
  |Schedule| Cosine |
  
  The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 29 billion tokens.
  
  ## Performance
  
  We evaluated the model on OpenAI's [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark which consists of programming challenges:
  
  | Metric | Value |
  |-------|-----|
  |pass@1 | 3.80% |
  |pass@10 | 6.57%	 |
  |pass@100 | 12.78% |
  
  The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests. 
  
  ## Resources
  
  - Dataset: [full](https://huggingface.co/datasets/codeparrot/codeparrot-clean), [train](https://huggingface.co/datasets/codeparrot/codeparrot-clean-train), [valid](https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid)
  - Code: [repository](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot)
  - Spaces: [generation](), [highlighting]()