title: Code Representation Learning At Scale
summary: Recent studies have shown that code language models at scale demonstrate
  significant performance gains on downstream tasks, i.e., code generation.
  However, most of the existing works on code representation learning train
  models at a hundred million parameter scale using very limited pretraining
  corpora. In this work, we fuel code representation learning with a vast amount
  of code data via a two-stage pretraining scheme. We first train the encoders
  via a mix that leverages both randomness in masking language modeling and the
  structure aspect of programming language. We then enhance the representations
  via contrastive learning with hard negative and hard positive constructed in an
  unsupervised manner. We establish an off-the-shelf encoder model that
  persistently outperforms the existing models on a wide variety of downstream
  tasks by large margins. To comprehend the factors contributing to successful
  code representation learning, we conduct detailed ablations and share our
  findings on (i) a customized and effective token-level denoising scheme for
  source code; (ii) the importance of hard negatives and hard positives; (iii)
  how the proposed bimodal contrastive learning boost the cross-lingual semantic
  search performance; and (iv) how the pretraining schemes decide the downstream
  task performance scales with the model size.
paper_authors._id: 65c1b72e35cfc292c0cad02b
paper_authors.name: Dejiao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T09:37:04.263Z
paper_authors._id: 61be0f906196e18d07adbc84
paper_authors.avatarUrl: /avatars/257cc25509e97c6a62e322e876a66c04.svg
paper_authors.fullname: Dejiao Zhang
paper_authors.type: user
paper_authors.user: dzxl1946
paper_authors._id: 65c1b72e35cfc292c0cad02c
paper_authors.name: Wasi Ahmad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:12:27.747Z
paper_authors._id: 67ec45ba92673a326c067b52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PcD60kFRczOblxgc1bWcl.png
paper_authors.fullname: Wasi Uddin Ahmad
paper_authors.type: user
paper_authors.user: wasiuddina
paper_authors._id: 65c1b72e35cfc292c0cad02d
paper_authors.name: Ming Tan
paper_authors._id: 65c1b72e35cfc292c0cad02e
paper_authors.name: Hantian Ding
paper_authors._id: 65c1b72e35cfc292c0cad02f
paper_authors.name: Ramesh Nallapati
paper_authors._id: 65c1b72e35cfc292c0cad030
paper_authors.name: Dan Roth
paper_authors._id: 65c1b72e35cfc292c0cad031
paper_authors.name: Xiaofei Ma
paper_authors._id: 65c1b72e35cfc292c0cad032
paper_authors.name: Bing Xiang
paper_title: Code Representation Learning At Scale
paper_summary: Recent studies have shown that code language models at scale demonstrate
  significant performance gains on downstream tasks, i.e., code generation.
  However, most of the existing works on code representation learning train
  models at a hundred million parameter scale using very limited pretraining
  corpora. In this work, we fuel code representation learning with a vast amount
  of code data via a two-stage pretraining scheme. We first train the encoders
  via a mix that leverages both randomness in masking language modeling and the
  structure aspect of programming language. We then enhance the representations
  via contrastive learning with hard negative and hard positive constructed in an
  unsupervised manner. We establish an off-the-shelf encoder model that
  persistently outperforms the existing models on a wide variety of downstream
  tasks by large margins. To comprehend the factors contributing to successful
  code representation learning, we conduct detailed ablations and share our
  findings on (i) a customized and effective token-level denoising scheme for
  source code; (ii) the importance of hard negatives and hard positives; (iii)
  how the proposed bimodal contrastive learning boost the cross-lingual semantic
  search performance; and (iv) how the pretraining schemes decide the downstream
  task performance scales with the model size.
paper_ai_summary: A two-stage pretraining scheme using a combination of masking language modeling and contrastive learning significantly improves code representation learning and performance on various downstream tasks.
paper_ai_keywords: masking language modeling contrastive learning hard negative hard positive bimodal contrastive learning cross-lingual semantic search token-level denoising