Code completion, which aims to predict the following code token(s) according to the code context, can improve the productivity of software development. Recent work has proved that statistical language modeling with transformers can greatly improve the performance in the code completion task via learning from large-scale source code datasets. However, current approaches focus only on code context within the file or project, i.e. internal context. Our distinction is utilizing external context, inspired by human behaviors of copying from the related code snippets when writing code. Specifically, we propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We adopt a stagewise training approach that combines a source code retriever and an auto-regressive language model for programming language. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark. With the growth of software engineering field, large-scale source code corpus gives a chance to train language models in code domain (Hindle et al., 2016; Tu et al., 2014). And benefiting from the large transformer models (Vaswani et al., 2017) and pre-training techniques (Devlin et al., 2018; Radford et al., 2018), a rapid progress has been made in many code-related tasks like code search (Feng et al., 2020; Guo et al., 2020), code summarization (Clement et al., 2020; Ahmad et al., 2020), bug fixing (Mashhadi and Hemmati, 2021; Drain et al., 2021) and code completion (Svyatkovskiy et al., 2020; Liu et al., 2020; Kim et al., 2021; Clement et al., 2020). Code completion is considered as an essential feature towards efficient software development in modern Integrated Development Environments (IDEs). The task is formulated by predicting the following code token(s) based on the code context. Traditionally, code completion requires real-time program analysis and recommends type-correct code tokens (Tu et al., 2014). Recently, statistical language models trained on large-scale source code data have shown high accuracy in the code completion task. Primitive approaches take the given context only (Liu et al., 2016; Karampatsis et al., 2020), some methods use richer information, e.g., adding code token types (Liu et al., 2020), abstract syntax tree (AST) structures (Li et al., 2018; Kim et al., 2021), or extended hierarchical context (Clement et al., 2021). However, one key limitation of existing methods is the scope of information they utilize; all the information is bounded in the given input file. This is unnatural from human perspective, as studies demonstrate that programmers tend to reuse an existing code snippet by copying part of code with or without minor modifications to accelerate software development (Roy and Cordy, 2008; Baker, 2007), leading a software repository usually containing 7-23% cloned codes (Svajlenko and Roy, 2015). Motivated by this phenomenon, in this paper, we argue the utility of extending the information scope beyond the input file, i.e., into a large code-base. We conjecture that using codes with similar semantics as auxiliary information are beneficial to predict the following code tokens. Therefore, we propose ReACC  a Re trievalA ugmented C ode C ompletion framework (See Figure 1). The code completion task under our framework can be re-formulated by, given a source code corpus for search and an unfinished code snippet to complete, using the unfinished code as a query to retrieve similar code snippets from search corpus, and predicting the following code tokens by reusing the 6227  Retriever def read_as_jsonl(self, json_file): lines = retrieved code. ReACC consists of two core components: (1) a dual-encoder model served as the code-to-code search retriever (2) an auto-regressive language model served as the code completion generator . ReACC adopts the stage-wise training strategy which is widely used in other tasks like open-domain question answering (Karpukhin et al., 2020; Izacard and Grave, 2021), natural language to code generation (Hashimoto et al., 2018; Parvez et al., 2021), etc. The simplest technique for retrieving code is to build a sparse vector retriever like TF-IDF or BM25 (Robertson and Zaragoza, 2009) which are both based on keyword matching algorithms. The sparse retriever can capture lexical information and is sensitive to the names of code identifiers. The dense retriever, on the contrary, can capture syntactic and semantic information by mapping a code snippet to a dense vector. In the code completion task, the code retriever is expected to comprehend the source code's intent in order to retrieve the semantically similar codes. On the other hand, considering programmers are prone to copy-and-paste existing code, the retriever should evaluate lexical similarity as well. To that end, we adopt the hybrid retriever (Karpukhin et al., 2020; Ma et al., 2021), which combines results of dense and sparse retriever. We employ a dual-encoder model architecture as the dense retriever since the cross-encoder model has a high computational complexity. To achieve a better understanding ability, we initialize our dense retriever with GraphCodeBERT (Guo et al., 2020), which is a pre-trained BERT-based programming language understanding model. Then we continue pre-training the retriever by contrastive learning to enhance sentence embedding. As the labeled data containing similar code pairs is rare, we utilize various transformations to generate programs with similar functionality for data augmentation. We implement the generator with a decoder-only transformer model. To incorporate the external information from retrieved similar code, we concatenate the obtained code and code context as input. The generator is initialized by CodeGPT-adapted (Lu et al., 2021) which is a domain-adaptation model from GPT-2 (Radford et al., 2018) pre-trained on code corpus. We evaluate our ReACC framework on two benchmark datasets  CodeXGLUE (Lu et al., 2021) and CodeNet (Puri et al., 2021), in Python and Java programming languages. ReACC achieves a state-of-the-art performance on both datasets. The experimental results demonstrate that external source code retrieved by our retriever is useful for auto-completing the partial code. To summarize, our main contributions are:  We propose a retrieval-augmented method to assist the code auto-completion task. 1  To adapt to the code completion scenario, where the retrieval query is an unfinished code snippet, we propose the partial code-to-code search task and create datasets for evaluation. 1 Our codes are available at https://github.com/ celbree/ReACC 6228 2 Related Work 2.1 Code completion Code completion is an essential task for code intelligence. Hindle et al. (2016) are the first to use language model for code completion by N-gram technique. Deep neural networks (Liu et al., 2016; Alon et al., 2020; Karampatsis et al., 2020) and pretraining approaches (Liu et al., 2020; Svyatkovskiy et al., 2020) are later frequently utilized to accomplish this. Besides considering source code as code token sequences, some research focuses on completing an abstract syntax tree (AST) by anticipating the next node in the flattened tree (Li et al., 2018; Kim et al., 2021). Guo et al. (2021) complete codes by generating sketches, i.e. code snippets with holes. Svyatkovskiy et al. (2021) and Clement et al. (2021), on the other hand, investigate ways to improve the efficiency and long-range modeling in the code completion task, respectively. All of these works employ previously written code context as inputs, along with AST structural information or token types. But none of them has attempted to leverage existing external code as auxiliary information. Contrastive learning on code Inspired by the great success of contrastive learning in other domains (Wu et al., 2018; Reimers and Gurevych, 2019; Fang et al., 2020; Chen et al., 2020; He et al., 2020; Radford et al., 2021; Gao et al., 2021), researchers have deployed this technique to source code for better code fragment understanding. Jain et al. (2020) and Bui et al. (2021) propose Contra-Code and Corder, respectively. Both models use the self-supervised contrastive learning framework and generate code snippets as data augmentations via compiler-based semantic-preserving transformations. Their models have shown the effectiveness of contrastive learning in code clone detection, code search and code summarization tasks. SYNCOBERT (Wang et al., 2022) and UniXcoder (Guo et al., 2022) are both pre-training models that utilize multi-modal data, including code, comment, and AST, for better code fragment representation through contrastive learning. Retrieval for code-related tasks Many code intelligence tasks benefit from information retrieval (Xia et al., 2017). A common scenario for information retrieval in code domain is code search with natural language description as a query (Arwan et al., 2015; Gu et al., 2018; Cambronero et al., 2019). As for other code intelligence tasks, Hayati et al. (2018) propose an action subtree retrieval method called ReCode for generating general-purpose code. Hashimoto et al. (2018) propose a retrieve-and-edit framework for code autocom-pletion and code generation. Luan et al. (2019) propose Aroma, which utilizes code-to-code structural search and intersecting candidate code snippets to recommend relevant code given another code snippet as a query. Both Wei et al. (2020) and Li et al. (2021) leverage the retrieve-and-edit/refine framework to improve model's performance in code summarization. Parvez et al. (2021) propose RED-CODER, using a dense retriever trained on paired NL-code pairs to retrieve relevant comments or codes as a supplement for code summarization or code generation tasks. In most circumstances where a dense retriever is utilized, a natural language comment is treated as a query to retrieve code. In the code completion scenario, however, we focus on using code as query, particularly partial code, which is a more difficult task since there are few labeled data with semantically similar code pairs and in partial code search, semantics in query is incomplete. We first introduce the formulation of retrieval-augmented code completion task. Then we give detailed descriptions on the retriever and generator in ReACC. We show how we continue pretraining GraphCodeBERT (Guo et al., 2020) with contrastive learning on code and how we address the problem that there is no labeled data for positive instances of similar programs in section 3.2. In section 3.3 we talk about the way to aggregate retrieved code and code context in the generator. Assume that we have a source code database containing a large collection of software repositories, which consist of D source code files, f 1 , f 2 , ..., f D . Following the Dense Passage Retriever (DPR) model (Karpukhin et al., 2020), we split each of the files into code fragments of equal lengths as the basic retrieval units. Such splitting not only leads to a better retrieval results as stated by Karpukhin et al. (2020), but also supports extreme long code files where each part of a file represents differ-6229 def normalize(a):ma= np.mean(a) sa = np.std(a) return (a-ma)/sa def standardization(arr):mu  np.mean np.std def normalize(a):ma= np.mean(a) sa np.mean def sort(a1, a2): tmp  sorted Original code Transformation Positive example Query In-batch negatives E n c o d e r Minimize Maximize Truncate Partial code API seq Figure 2: Illustration on the training process of the retriever in our proposed framework ReACC. ent semantics. Thus we get M code fragments as the retrieval database C = { c 1 , c 2 , ..., c M } . Let X = { x 1 , x 2 , ..., x k } be the unfinished code written previously, a retriever R : ( X, C )  C retrieves the most similar code fragment c s in C . The generator G predicts the following code token(s) Y = { x k +1 , ..., x k + n } , where n = 1 in the token-level code completion task, based on context and retrieved code. Formally, P ( Y ) = (cid:81) ni =1 P ( x k + i | c s , x 1: k + i  1 ) . The retrieval module in ReACC is expected to retrieve semantically equivalent code given an incomplete code. We adopt the hybrid retriever (Karpukhin et al., 2020; Ma et al., 2021) framework by combining scores of sparse and dense retriever. The sparse retriever we use is BM25 (Robertson and Zaragoza, 2009) based on the implementation of ElasticSearch 2 . As a term-based retrieval method, BM25 considers each code fragment as a code token sequence and employs bag-of-words representations. The matching score computed by BM25 indicts lexical similarity between the query and document. As for the dense retriever, it maps each code fragment to a d -dimension dense vector. We construct it in this paper based on the DPR model (Karpukhin et al., 2020). Figure 2 illustrates the training process of the dense retriever of ReACC. In the following, we will walk through it in detail. Dense Retriever Our dense retriever consists of two bidirectional transformer-based encoders EC and EQ . EC encodes each code fragment in the retrieval database C and builds indexes for them. The query is encoded by EQ . We take the representation of [CLS] token as output and the similarity is computed by sim ( q, c ) = EC ( c ) TEQ ( q ) . Since both EC and EQ take source code as inputs with the only difference being whether they are partial or not, the dual encoders share weights in ReACC. At the training stage, following DPR (Karpukhin et al., 2020), we adopt in-batch negatives to calculate the contrastive loss by InfoNCE (Oord et al., 2018): L ( q, c + , c  1 , c  2 , ..., c  m ) =  log e sim ( q,c + ) e sim ( q,c + ) + (cid:80) mi =1 e sim ( q,c  i ) (1) However, unlike DPR, we don't employ "hard" negatives which are retrieved from BM25. Because programmers tend to copy tokens directly, a code with distinct semantics but substantial lexical similarity can help with code completion. Data Augmentation The purpose of contrastive learning of the dense retriever in ReACC is to learn a representation of code fragments that keeps codes with similar or equivalent semantics close and dissimilar codes far apart. It requires numerous positive and negative code pairs. However, it is difficult to identify similar programs based on an unlabeled code corpus, e.g., certain widely used datasets (Al-lamanis and Sutton, 2013; Raychev et al., 2016; Husain et al., 2019) mined from GitHub repositories. Searching semantically equivalent code requires extra code compilation and execution costs (Mas-salin, 1987; Churchill et al., 2019), which is unrealistic in a large database. Instead of searching, an alternative way is to create code snippets with same functionalities for data augmentation. To do so, we apply several semantic-preserving transformations to the original source code to construct a set of variants. There exists several attempts to apply such transformation to code (Jain et al., 2020; Rabin et al., 2021; Bui et al., 2021). In this paper, we mainly adopt identifier renaming and dead code (unreachable or unused code) insertion. Figure 3 shows an example of performing such transformations to a Python code. Identifier renaming is a method of renaming an identifier with another. We only rename variable and method names as other identifiers cannot be changed arbitrarily like built-in types or API calls. Different from previous works, we preserve 6230 import socket def echo_server(client, timeout, bufsize): try:if timeout > 0: client.settimeout(timeout) get_buf = client.recv(bufsize) client.send(get_buf) except socket.timeout: pass client.close() import socket def get_mean(c, doc, local): try:if doc > 0: c.settimeout(doc) _user_id = c.recv(local) c.send(_user_id) except socket.timeout: pass c.close() import socket def echo_server(client, timeout, bufsize): try:if timeout > 0: client.settimeout(timeout) get_buf = client.recv(bufsize) if True:tmp= [x**2 for x in range(10)] client.send(get_buf) except socket.timeout: pass client.close() original python code After renaming all variables After inserting dead code Figure 3: An example of applying semantic-preserving transformations to Python code. part of the lexical information while modifying the names at the same time based on the consideration that identifier names typically convey the meanings for humans and lexical similarity contributes a lot for retrieving (It is verified in section 4.4). To do so, we mask all the identifiers in a program and leverage GraphCodeBERT (Guo et al., 2020) to predict each identifier like in the masked language model task. The top-10 predictions (excluding the original identifier) are selected as the candidate set for renaming. Dead code insertion is to insert a dead code into a code fragment at a proper location. Dead code is a code snippet which can never be reached (Xi, 1999) or is reachable but whose result can never be used in any other computation (Debray et al., 2000). In software engineering, dead code insertion is one of the most common techniques for code obfuscation (You and Yim, 2010), whose goal is to modify a code to make it hard to understand but remain its functionality, which is similar to our goal. We first randomly select variable names which don't appear in this program and then use them to form a statement from a predefined set of dead code (See Appendix A for details), such as assignment, method invocations, looping statement, conditional statement and so on. We traverse the AST and identify all the statements. Then we choose a statement at random and insert the dead code after it, leading a new subtree in the AST. Input Format We integrate both the code token sequence and the API usage sequence as inputs. API usage sequence is highly related to the functionality of a code snippet (Gu et al., 2016; Hu et al., 2018). To improve the code representation, we extract the API sequence and append it to the source code token sequence. Finally, we use a random truncation of the original code as the query and the entire created program as the positive example during training to address the problem on how to retrieve based on incomplete semantics. The output of retriever is the retrieved code c s . Considering c s is queried by code context x while our target is the following code of x , so we propose fragment alignment  using the next fragment c (cid:48) s of c s in the same file (we have split each file into code fragments for retrieval as discussed in Section 3.1) for completing the next fragment of x . Thus, the input sequence for the generator is the concatenation of c (cid:48) s and x : x (cid:48) = c (cid:48) s  x . The generator module in ReACC supports any model architecture that can perform code completion task. In our experiments, we adopt CodeGPT-adapted (Lu et al., 2021), which is a decoder-only transformer model pre-trained on Python and Java datasets from CodeSearchNet (Husain et al., 2019) via casual language model. CodeGPT-adapted has shown promising results in the code completion task in CodeXGLUE benchmark (Lu et al., 2021) on two widely used code completion datasets. In order to evaluate the effectiveness of the code-to-code retrieval module in ReACC, we perform code clone detection task which aims to retrieve semantic equivalent programs. In this section, we describe how we create the test dataset for this task and how we evaluate the performance of ReACC's retriever. CodeNet (Puri et al., 2021) dataset consists of a large collection of programs which are derived from online judge websites. We respectively create a code clone detection evaluation dataset from CodeNet in Python and Java with zero-shot setting. We collect code solutions for thousands problems and solutions for the same problem are considered as semantically equivalence. The data statistics are shown in Table 1. Retrieval Training Set The dense retriever in ReACC is pre-trained on CodeSearchNet dataset (Husain et al., 2019), a large-scale source code corpus extracted from GitHub repositories. We employ 1.6M Java methods and 1.2M Python functions from it. CodeBERT (Feng et al., 2020) is a pre-trained model for programming language, which is trained on NL-PL pairs from CodeSearchNet dataset in six programming languages. GraphCodeBERT (Guo et al., 2020) is also pre-trained on CodeSearchNet NL-PL pairs and considers the inherent structure of code i.e. data flow. The retrieval encoder is initialized with GraphCodeBERT. It is continual pre-trained with both masked language model objective and contrastive learning. We use in-batch negatives with a batch size of 256. With a learning rate of 5e-5, We train the retriever for Python and Java for 30 epochs each. We implement the code clone detection experiment in the partial search way, which is ideally adapted to code completion scenarios as it accepts a partial program as a query while maintaining the same goal. Table 2 shows the results in the zero-shot code clone detection task on CodeNet dataset, with the partial search setting. Models are measured by MAP@K (Mean Average Precision at K), which is the evaluation metric in the CodeXGLUE clone detection task, and precision at 1, as we only care about the most similar code for code completion. From the comparison with other transformer-based encoders, we can see CodeBERT and GraphCodeBERT can hardly retrieve equivalent code. While our model significantly outperforms them, which indicts our model is capable of retrieving the semantically equivalent code even when the query's semantics is incomplete. We also find that BM25 performs splendidly in this task, which is quite different from the performance on other tasks like open-domian QA (Karpukhin et al., 2020), code summarization (Parvez et al., 2021), etc. The findings suggest that semantically related codes are likely to be lexically similar, which leads lexical similar to contribute more for retrieval, making code-to-code search easier than text-to-code or question-to-passage search using the term-based retrieval method. In this section, we evaluate ReACC on end-to-end code completion. CodeXGLUE (Lu et al., 2021) is a benchmark dataset containing 14 datasets for 10 diversified code intelligence tasks. We use PY150 dataset (Raychev et al., 2016) in Python and GitHub Java Corpus dataset (Allamanis and Sutton, 2013) in Java from it for code completion task. Table 1 shows the data statistics. CodeGPT/CodeGPT-adapted (Lu et al., 2021) are both pre-trained on Python and Java datasets from CodeSearchNet. CodeGPT is trained from scratch while CodeGPT-adapted is a domain adaptation model which is initialized by GPT-2 (Rad-ford et al., 2019). PLBART (Ahmad et al., 2021) is based on BART (Lewis et al., 2020) architecture which employs denoising sequence-to-sequence (Seq2Seq) 6232 Model Python Java MAP@100 Precision MAP@100 Precision CodeBERT 1.47 4.75 1.15 4.58 GraphCodeBERT 5.31 15.68 4.54 16.05 BM25 10.32 23.17 8.67 25.85 ReACC-retriever 9.60 27.04 9.31 27.55 Table 2: Results on zero-shot code clone detection dataset created from CodeNet. CodeT5 (Wang et al., 2021) is also an encoder-decoder pre-trained model which adapts T5 (Raf-fel et al., 2019) architecture and considers the identifier-aware token type information in code. X-CodeGPT is a variant of CodeGPT which adapts eWASH (Clement et al., 2021) to CodeGPT. Clement et al. (2021) propose eWASH, a method for leveraging the syntax hierarchy of source code to give the model wider field of vision in a file and achieving a new SOTA performance on the CodeXGLUE code completion task. We reproduce their method and develop X-CodeGPT by adapting eWASH to CodeGPT-adapted. Fine-tune We fine-tune CodeGPT-adapted on PY150 and GitHub Java Corpus datasets, respectively, and use it as the generator in ReACC. The number of epochs for training PY150 is 30 and Java Corpus is 10, with a batch size of 96 and a learning rate of 2e-5. Except for X-CodeGPT, all other baseline models are fine-tuned with the same settings. As for X-CodeGPT, we pre-train it with a training set extracted from CodeSearchNet in eWASH format, where each example is a function body with its corresponding extended context, as described by Clement et al. (2021). Since eWASH requires codes parsed into ASTs but codes in CodeXGLUE have been tokenized and cannot be parsed, we build a new dataset from PY150 to fine-tune X-CodeGPT on CodeXGLUE. As a result, we download the origin files in PY150 and create a new dataset that retains the train/valid/test split, as seen in Table 1. Evaluation Following Lu et al. (2021), we conduct two code completion scenarios, token-level and line-level completion, to measure models' ability of predicting one and more tokens. Perplexity is the evaluation metric for token-level completion, whereas exact match accuracy (EM) and edit similarity are used for line-level completion. For token-level completion, based on the consideration of efficiency, instead of applying retrieval at each step, we retrieve similar codes based on current context after predicting the first 100 tokens, and leverage it for further prediction. Retrieval Database We use the training set of PY150 and Java Corpus as retrieval database for test. We don't use the contrastive pre-training corpus (i.e., CodeSearchNet) in order to avoid the duplication between CodeXGLUE and CodeSearchNet as they are both extracted from GitHub. Hybrid Retriever A linear combination of scores from BM25 and our dense retriever forms a hybrid retriever. Specifically, we calculate the score by sim ( q, c ) +   BM 25( q, c ) and let  = 0 . 9 based on the results on dev set for both PY150 and Java Corpus datasets. Table 3 and Table 4 compare different baseline models on code completion task in the CodeXGLUE Python and Java datasets. ReACC framework with the hybrid retriever outperforms consistently than other baselines on all datasets, which proves our conjection that the external context is beneficial to the code completion task. The comparison with X-CodeGPT in Table 4 demonstrates that utilizing external context could be more useful than making the most of the current code file. Among three configurations of the retriever in ReACC, hybrid retriever performs best on almost all metrics except the exact match score in the new test set of PY150. From Table 3, we can observe that comparing the two datasets, the improvement in the PY150 dataset is greater than that in the Java Corpus dataset. The reason for this is that the retrieval database for Java (i.e., the training set) is much smaller. The CodeXGLUE Java Corpus dataset contains only 12,934 files for training so that it's more difficult to retrieve similar code from them. Another finding is that BM25 shows comparable results with dense retriever and even performs better in perplexity and exact match metrics. The findings indict that the code completion task can benefit from both semantically and lexically similar codes. ReACC in specific domain Both PY150 and Java Corpus datasets are extracted from GitHub repositories which are distributed in a wide domain. As some people frequently write codes in a more specific domain, e.g., data mining/pattern recogni-EM Edit Sim ReACC-dense 45.32 73.95 Retriever-identifier renaming 44.91 73.14 dead code insertion 45.11 73.57 API sequence 44.77 73.01 query truncation 43.93 72.65 Generator-fragment alignment 45.08 73.56 Table 6: Ablation study for both retriever and generator module. tion domain for Kaggle 3 users, algorithm domain for ACM community, etc. To evaluate ReACC in a specific code domain, we construct a code completion Python dataset from CodeNet, which can be considered in algorithm domain. Table 5 reveals that ReACC significantly outperforms CodeGPT-adapted in CodeNet by 10% and 18% absolute improvement in edit similarity and exact match, respectively. According to the findings, ReACC is more effective in a specific domain. We also notice that ReACC with dense retriever outperforms BM25 significantly in CodeNet. It can be explained by the fact that in algorithm domain, semantically similar code may be more valuable than for code completion lexically similar code. Ablation study To further understand how our training options affect model performance, we conduct ablation experiments. As seen in Table 6, when data argumentation and training strategies in retriever or generator are eliminated, the metrics degrade. The most essential factor among them is query truncation. Comparing the two semantic-preserving transformations, identifier renaming contributes more than dead code insertion.When fragment alignment is removed from generator, i.e. using the retrieved code snippet itself for generator, performance suffers slightly. ReACC vs GitHub Copilot GitHub Copilot 4 is a powerful technique for code completion which uses OpenAI Codex (Chen et al., 2021) as the model backend. We run some qualitative examples with its extension in VSCode, which are shown in the Appendix B. It worth noting that Codex is more powerful than CodeGPT since it is a large-scale pre-trained model that is trained on all source codes in GitHub based on GPT-3 (Brown et al., 2020). However, in some cases, ReACC with CodeGPT as 3 https://www.kaggle.com/ 4 https://copilot.github.com/ 6234 the generator outperforms Copilot. And in 6 Copilot itself can benefit from ReACC when it takes advantage of ReACC's retriever, which indicates the effectiveness of retrieval-augmented method for strong generative models. We propose ReACC, a retrieval-augmented code completion framework that utilizes external context for the code completion task by retrieving semantically and lexically similar codes from existing codebase. We pre-train a dual-encoder as a retriever for partial code search, which retrieves code fragments given a partial code. Our method can adopt any architecture that can perform code completion as the generator. On the CodeXGLUE benchmark, ReACC achieves a state-of-the-art performance in the code completion task. This work is supported by Microsoft and IITP grants (2021-0-01696, High Individuals Global Training Program) Finding clones with dup: Analysis of an experiment. IEEE Transactions on Software Engineering , 33(9):608621. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2021. Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 511521. Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. 2019. When deep learning met code search. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 964974. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 15971607. PMLR. Saumya K Debray, William Evans, Robert Muth, and Bjorn De Sutter. 2000. Compiler techniques for code compaction. ACM Transactions on Programming languages and Systems (TOPLAS) , 22(2):378 415. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . 6235 Dawn Drain, Chen Wu, Alexey Svyatkovskiy, and Neel Sundaresan.
abstain abstain abstain method objective method result abstain other abstain abstain abstain abstain other abstain abstain objective method objective method abstain abstain abstain abstain abstain abstain abstain abstain result method result method method method result abstain method abstain objective objective objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method method method abstain method abstain abstain abstain abstain method method abstain abstain abstain method abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method method abstain method abstain abstain abstain method method method method method abstain method method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method method method abstain method method method method method abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result other abstain objective method method abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other