title: CodeT5+: Open Code Large Language Models for Code Understanding and
    Generation
summary: Large language models (LLMs) pretrained on vast source code have achieved
  prominent progress in code intelligence. However, existing code LLMs have two
  main limitations in terms of architecture and pretraining tasks. First, they
  often adopt a specific architecture (encoder-only or decoder-only) or rely on a
  unified encoder-decoder network for different downstream tasks. The former
  paradigm is limited by inflexibility in applications while in the latter, the
  model is treated as a single system for all tasks, leading to suboptimal
  performance on a subset of tasks. Secondly, they often employ a limited set of
  pretraining objectives which might not be relevant to some downstream tasks and
  hence result in substantial performance degrade. To address these limitations,
  we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which
  component modules can be flexibly combined to suit a wide range of downstream
  code tasks. Such flexibility is enabled by our proposed mixture of pretraining
  objectives to mitigate the pretrain-finetune discrepancy. These objectives
  cover span denoising, contrastive learning, text-code matching, and causal LM
  pretraining tasks, on both unimodal and bimodal multilingual code corpora.
  Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs
  without training from scratch to efficiently scale up our models, and explore
  instruction-tuning to align with natural language instructions. We extensively
  evaluate CodeT5+ on over 20 code-related benchmarks in different settings,
  including zero-shot, finetuning, and instruction-tuning. We observe
  state-of-the-art (SoTA) model performance on various code-related tasks, such
  as code generation and completion, math programming, and text-to-code retrieval
  tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA
  results on HumanEval code generation task against other open code LLMs.
paper_authors._id: 6462ceee48e13890ea569bcb
paper_authors.name: Yue Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:32:34.036Z
paper_authors._id: 614d32bfb06484609bf2292c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632449169691-noauth.png
paper_authors.fullname: Yue Wang
paper_authors.type: user
paper_authors.user: yuewang-sf
paper_authors._id: 6462ceee48e13890ea569bcc
paper_authors.name: Hung Le
paper_authors._id: 6462ceee48e13890ea569bcd
paper_authors.name: Akhilesh Deepak Gotmare
paper_authors._id: 6462ceee48e13890ea569bce
paper_authors.name: Nghi D. Q. Bui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:05:07.814Z
paper_authors._id: 63a44141de134926a2ba0458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a44141de134926a2ba0458/FP3uxpsKL1nBrrKyiFne3.jpeg
paper_authors.fullname: Nghi Bui
paper_authors.type: user
paper_authors.user: bdqnghi
paper_authors._id: 6462ceee48e13890ea569bcf
paper_authors.name: Junnan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T14:31:09.345Z
paper_authors._id: 61f9d3b54ac99e8a1bae85f4
paper_authors.avatarUrl: /avatars/ac47d13204dd22452e4bc46e280842d5.svg
paper_authors.fullname: JunnanLi
paper_authors.type: user
paper_authors.user: JunnanLi
paper_authors._id: 6462ceee48e13890ea569bd0
paper_authors.name: Steven C. H. Hoi
paper_title: CodeT5+: Open Code Large Language Models for Code Understanding and
    Generation
paper_summary: Large language models (LLMs) pretrained on vast source code have achieved
  prominent progress in code intelligence. However, existing code LLMs have two
  main limitations in terms of architecture and pretraining tasks. First, they
  often adopt a specific architecture (encoder-only or decoder-only) or rely on a
  unified encoder-decoder network for different downstream tasks. The former
  paradigm is limited by inflexibility in applications while in the latter, the
  model is treated as a single system for all tasks, leading to suboptimal
  performance on a subset of tasks. Secondly, they often employ a limited set of
  pretraining objectives which might not be relevant to some downstream tasks and
  hence result in substantial performance degrade. To address these limitations,
  we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which
  component modules can be flexibly combined to suit a wide range of downstream
  code tasks. Such flexibility is enabled by our proposed mixture of pretraining
  objectives to mitigate the pretrain-finetune discrepancy. These objectives
  cover span denoising, contrastive learning, text-code matching, and causal LM
  pretraining tasks, on both unimodal and bimodal multilingual code corpora.
  Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs
  without training from scratch to efficiently scale up our models, and explore
  instruction-tuning to align with natural language instructions. We extensively
  evaluate CodeT5+ on over 20 code-related benchmarks in different settings,
  including zero-shot, finetuning, and instruction-tuning. We observe
  state-of-the-art (SoTA) model performance on various code-related tasks, such
  as code generation and completion, math programming, and text-to-code retrieval
  tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA
  results on HumanEval code generation task against other open code LLMs.
paper_ai_keywords: encoder-decoder LLMs span denoising contrastive learning text-code matching causal LM pretraining multilingual code corpora off-the-shelf LLMs instruction-tuning zero-shot finetuning code generation code completion math programming text-to-code retrieval HumanEval code generation task