title: AST-T5: Structure-Aware Pretraining for Code Generation and
    Understanding
summary: Large language models (LLMs) have made significant advancements in
  code-related tasks, yet many LLMs treat code as simple sequences, neglecting
  its structured nature. We introduce AST-T5, a novel pretraining paradigm that
  leverages the Abstract Syntax Tree (AST) for enhanced code generation,
  transpilation, and understanding. Using dynamic programming, our AST-Aware
  Segmentation retains code structure, while our AST-Aware Span Corruption
  objective equips the model to reconstruct various code structures. Unlike other
  models, AST-T5 avoids intricate program analyses or architectural changes, so
  it integrates seamlessly with any encoder-decoder Transformer. Evaluations show
  that AST-T5 consistently outperforms similar-sized LMs across various
  code-related tasks. Structure-awareness makes AST-T5 particularly powerful in
  code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the
  Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in
  CodeXGLUE. Our code and model are publicly available at
  https://github.com/gonglinyuan/ast_t5.
paper_authors._id: 659cbf6f0030e8faffb5e657
paper_authors.name: Linyuan Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:53:07.555Z
paper_authors._id: 645b61c21c24cd669dd64080
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645b61c21c24cd669dd64080/o-uAnxZ56kwFkD_wirf7-.jpeg
paper_authors.fullname: Linyuan Gong
paper_authors.type: user
paper_authors.user: gonglinyuan
paper_authors._id: 659cbf6f0030e8faffb5e658
paper_authors.name: Mostafa Elhoushi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-09T12:57:02.649Z
paper_authors._id: 63c9725ebedad7e2bf160bdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
paper_authors.fullname: Mostafa Elhoushi
paper_authors.type: user
paper_authors.user: melhoushi
paper_authors._id: 659cbf6f0030e8faffb5e659
paper_authors.name: Alvin Cheung
paper_title: AST-T5: Structure-Aware Pretraining for Code Generation and
    Understanding
paper_summary: Large language models (LLMs) have made significant advancements in
  code-related tasks, yet many LLMs treat code as simple sequences, neglecting
  its structured nature. We introduce AST-T5, a novel pretraining paradigm that
  leverages the Abstract Syntax Tree (AST) for enhanced code generation,
  transpilation, and understanding. Using dynamic programming, our AST-Aware
  Segmentation retains code structure, while our AST-Aware Span Corruption
  objective equips the model to reconstruct various code structures. Unlike other
  models, AST-T5 avoids intricate program analyses or architectural changes, so
  it integrates seamlessly with any encoder-decoder Transformer. Evaluations show
  that AST-T5 consistently outperforms similar-sized LMs across various
  code-related tasks. Structure-awareness makes AST-T5 particularly powerful in
  code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the
  Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in
  CodeXGLUE. Our code and model are publicly available at
  https://github.com/gonglinyuan/ast_t5.
paper_ai_keywords: Abstract Syntax Tree (AST) AST-Aware Segmentation AST-Aware Span encoder-decoder Transformer code2Fix Java-C# Transpilation CodeXGLUE