readme: ---
  license: bsd-3-clause
  ---
  # CodeT5 (large-size model 770M)
  
  ## Model description
  
  CodeT5 is a family of encoder-decoder language models for code from the paper: [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://arxiv.org/pdf/2109.00859.pdf) by Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi.
  
  The checkpoint included in this repository is denoted as **CodeT5-large** (770M), which is introduced by the paper: [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://arxiv.org/pdf/2207.01780.pdf) by Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C.H. Hoi.
  
  ## Training data
  
  CodeT5-large was pretrained on [CodeSearchNet](https://arxiv.org/abs/1909.09436) data in six programming languages (Ruby/JavaScript/Go/Python/Java/PHP).  See Section 4.1 of the [paper](https://arxiv.org/pdf/2207.01780.pdf) for more details.
  
  ## Training procedure
  
  CodeT5-large was pretrained using masked span prediction objective for 150 epochs. See Section 4.1 of the [paper](https://arxiv.org/pdf/2207.01780.pdf) for more details.
  
  
  ## Evaluation results
  We validate the effectiveness of this checkpoint pretrained with simplified strategies on [CodeXGLUE](https://github.com/microsoft/CodeXGLUE) benchmark. See Appendix A.1 of the [paper](https://arxiv.org/pdf/2207.01780.pdf) for more details.
  
  
  ## How to use
  
  This model can be easily loaded using the `T5ForConditionalGeneration` functionality:
  
  ```python
  from transformers import AutoTokenizer, T5ForConditionalGeneration
  tokenizer = AutoTokenizer.from_pretrained("Salesforce/codet5-large")
  model = T5ForConditionalGeneration.from_pretrained("Salesforce/codet5-large")
  text = "def greet(user): print(f'hello <extra_id_0>!')"
  input_ids = tokenizer(text, return_tensors="pt").input_ids
  
  # simply generate a single sequence
  generated_ids = model.generate(input_ids, max_length=8)
  print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
  ```
  
  ## BibTeX entry and citation info
  
  ```bibtex
  @inproceedings{CodeT52021,
    author    = {Yue Wang and Weishi Wang and Shafiq R. Joty and Steven C. H. Hoi},
    title     = {CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation},
    booktitle = {EMNLP},
    pages     = {8696--8708},
    publisher = {Association for Computational Linguistics},
    year      = {2021}
  }
  
  @article{CodeRL2022
    author    = {Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C.H. Hoi},
    title     = {CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning},
    journal   = {arXiv preprint},
    volume    = {abs/2207.01780},
    year      = {2022}
  }
  ```