Pre-trained models for programming languages have recently demonstrated great success on code intelligence. To support both code-related understanding and generation tasks, recent works attempt to pre-train unified encoder-decoder models. However, such encoder-decoder framework is sub-optimal for auto-regressive tasks, especially code completion that requires a decoder-only manner for efficient inference. In this paper, we present UniXcoder, a unified cross-modal pre-trained model for programming language. The model utilizes mask attention matrices with prefix adapters to control the behavior of the model and leverages cross-modal contents like AST and code comment to enhance code representation. To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task. We evaluate UniXcoder on five code-related tasks over nine datasets. To further evaluate the performance of code fragment representation, we also construct a dataset for a new task, called zero-shot code-to-code search. Results show that our model achieves state-of-the-art performance on most tasks and analysis reveals that comment and AST can both enhance UniXcoder. Pre-trained models such as GPT (Radford et al.) and BERT (Devlin et al., 2018) have substantially advanced the state of the art across numerous natural language processing (NLP) tasks. These pre-trained models are pre-trained on large amounts  Work done while this author was an intern at Microsoft Research. Contact: Daya Guo (guody5@mail2.sysu.edu.cn). of text data with self-supervised objectives, and can be fine-tuned to adapt to downstream tasks. Inspired by the success of pre-trained models in NLP, pre-trained models for programming languages (PL) (Kanade et al., 2019; Feng et al., 2020; Svyatkovskiy et al., 2020) have been proposed to promote the development of code intelligence. Svyatkovskiy et al. (2020) proposes GPT-C that employs a left-to-right Transformer (Vaswani et al., 2017) to support generation tasks such as code completion, but the unidirectional framework is sub-optimal for understanding tasks. In contrast, other works (Kanade et al., 2019; Feng et al., 2020) pre-train a bidirectional Transformer encoder on source code, which significantly improves the performance of code-related understanding tasks. However, its bidirectionality nature requires an additional decoder when applied to generation tasks, where this decoder initializes from scratch and cannot benefit from the pre-training. In this work, we present UniXcoder, a unified cross-modal pre-trained model for programming languages to support both code-related understanding and generation tasks. UniXcoder is based on a multi-layer Transformer and follows Dong et al. (2019) to utilize mask attention matrices with prefix adapters to control the access to context for each token. Compared with current unified encoder-decoder models (Ahmad et al., 2021; Wang et al., 2021) on code intelligence, UniXcoder can be better applied to auto-regressive tasks such as code completion that requires a decoder-only manner to perform efficient inference in practice. Instead of taking code as the only input, we also consider multi-modal contents like code comment and abstract syntax tree (AST) to enhance code representation. Generally, user-written code comments provide crucial semantic information about source code like Sort a given list and AST contains rich syntax information like types of statements and nested relationship among them, which helps the 7212 model better understand source code. To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all information of the tree and then the sequence can be used as the input to enhance code representation. We pre-train UniXcoder using three types of language modeling tasks: masked language modeling (Devlin et al., 2018), unidirectional language modeling (Radford et al.) and denoising objective (Raffel et al., 2019), which can enable the model to support various types of downstream tasks. Furthermore, we introduce two pre-training tasks to learn a embedding that can represent semantics of a code fragment. One is multi-modal contrastive learning that leverages AST to enhance semantics of code fragment embeddings, and the other is cross-modal generation that utilizes code comment to align embeddings among programming languages. We evaluate UniXcoder on five tasks over nine public datasets, including two understanding tasks: clone detection and code search, two generation tasks: code summarization and code generation, and an auto-regressive task: code completion. To further test code fragment embeddings, we propose a new task, called zero-shot code-to-code search, and construct a new dataset from CodeNet corpus (Puri et al., 2021) for this task. Experimental results show that our model achieves state-of-the-art performance on most tasks. Further analysis reveals that AST and code comment can both enhance UniXcoder to better capture code semantics. In summary, the contributions of this paper are: (1) We propose a unified cross-modal pre-trained model that leverages multi-modal contents, i.e. code comment and AST, to support code-related understanding, generation tasks and auto-regressive tasks. (2) We propose a one-to-one mapping function that converts AST into a sequence that retains all information of AST and can be encoded with source code and comment in parallel. (3) We further propose to utilize code comment to learn code fragment representation and construct a new dataset for zero-shot code-code search to evaluate the quality of code fragment representation. (4) Experimental results show that UniXcoder provides significant improvement on most downstream tasks. 1 1 All the codes and data are available at https:// github.com/microsoft/CodeBERT . With the great success of pre-training in natural language (NL) processing (Devlin et al., 2018; Lewis et al., 2019; Raffel et al., 2019; Brown et al., 2020), pre-trained models for programming languages have been proposed to promote the development of code intelligence. These pre-trained models can be generally divided into three categories: encoder-only, decoder-only, and encoder-decoder models. Encode-only models (Kanade et al., 2019; Bu-ratti et al., 2020; Feng et al., 2020; Guo et al., 2020; Wang et al., 2022) pre-train a bidirectional Transformer in which each token can attend to each other. Kanade et al. (2019) pre-train CuBERT on a corpus of Python source codes by masked language modeling and next sentence prediction objectives. CodeBERT(Feng et al., 2020) is pre-trained on NL-PL pairs in six programming languages with a new pre-training task, namely replace token detection. GraphCodeBERT (Guo et al., 2020) leverages data flow to enhance code representation, while SYNCOBERT (Wang et al., 2022) incorporates abstract syntax tree by AST edge prediction and contrastive learning. However, encoder-only models require an additional decoder for generation tasks, where this decoder initializes from scratch and cannot benefit from the pre-training. As for decoder-only pre-trained models, Svyatkovskiy et al. (2020) and Lu et al. (2021) respectively propose GPT-C and CodeGPT, which are both pre-trained using unidirectional language modeling that only allows tokens to attend the previous tokens and itself to predict the next token. Decoder-only models are good at auto-regressive tasks like code completion, but the unidirectional framework is sub-optimal for understanding tasks. Some recent works explore encoder-decoder models to support both understanding and generation tasks. PLBART (Ahmad et al., 2021) is based on the BART (Lewis et al., 2019) architecture and pre-trained on NL and PL corpus using denoising objectives. CodeT5 (Wang et al., 2021) adapts the T5 (Raffel et al., 2019) model that considers the crucial token type information from identifiers and allow for multi-task learning on downstream tasks. TreeBERT (Jiang et al., 2021) follows the encoder-decoder transformer framework but utilizes the tree structural information by modeling AST paths. Different from current unified models, UniXcoder is based on a multi-layer Transformer and utilizes mask attention matrices with prefix adapters 7213 # Return the sample arithmetic mean of data def mean(data):n=len(data)returnsum(data) / n Python code with a comment AST Parser module function_definition def mean parameters : block ( data ) expression_statement assignment call argument_list n = len ( data ) return_statement binary_operator return call argument_list ( data ) sum / n Non-terminal symbols (nodes) Terminal symbols (leaves) Parent-to-childrelation Figure 1: A Python code with its comment and AST. to control the behavior of the model for supporting both understanding and generation tasks. Compared with the encoder-decoder architecture, UniXcoder can be better applied to auto-regressive tasks like code completion that is widely used in IDEs, since the task requires a decoder-only manner to perform efficient inference in practice. Liu et al. (2020) also pre-train a similar model CugLM with multi-task learning, but they only focus on code completion rather than various tasks. Besides, we incorporate syntax information from AST by a one-to-one mapping function that converts an AST into a sequence to enhance code representation. Different from previous pre-trained models that utilize AST, the mapping function retains all structural information from AST and does not require additional pre-training tasks (such as edge prediction) to implicitly learn the AST structure. In this section, we describe UniXcoder, a unified cross-modal pre-trained model that leverages multimodal data (i.e. code comment and AST) to pretrain code representation. The model is based on Transformer and utilizes mask attention matrices (Dong et al., 2019) with prefix adapters to control the behavior of the model. In the following, we first introduce how to unify multi-modal data as the input of UniXcoder (3.1), and then the model architecture (3.2) and pre-training tasks (3.3). We give an example of a python code with its comment and AST in Figure 1. From the figure, we can see that the comment Return the sample arithmetic mean of data highly describes the function of the source code, which provides crucial semantic information about the source code. Besides, AST provides rich syntax information, for example, the subtree parameters  (data) indicates the type (i.e., parameters ) of the term (data) in the function definition. Both of them can be used as additional knowledge to enhance code representation in pre-trained models. However, AST is usually expressed as a tree and cannot be used directly as input to Transformer. In order to encode AST in parallel with code comments, we propose a one-to-one mapping function F , described in Algorithm 1, to transform an AST into a sequence that retains all structural information. Specially, given a root node root of AST, the algorithm recursively applies the same function F to its children and then add its name with two special suffixes (i.e. left and right , respectively) on both sides (line 6-11 of Algorithm 1). If the root node is a leaf, we directly produce its name (line 4-5). Taking parameters  (data) as an example, the mapping function F transforms the subtree to <parameters,left> ( data ) <parameters,right> . There can be various ways to transform a tree to a sequence of tokens, e.g. pre-order traversal. However, a particular transformation should be a one-to-one mapping function. Otherwise, the mapping may confuse a tree with another structure. Our mapping function F satisfies this requirement (see Appendix A for a proof). Finally, given a source code C , we take its comment W = { w 0 , w 1 , ..., w m  1 } and the flattened AST 7214 token sequence F ( T ( C )) = { c 0 , c 1 , ..., c k  1 } as input, where T ( C ) is the root of the AST of the code. For input format, we concatenate them with a prefix as an input sequence, as shown at the bottom of Figure 2, where the prefix represents the work mode of the model and will be discussed next. Figure 2 shows the model architecture of UniXcoder. The model applies N transformer layers over code comment and flattened AST with a prefix to produce hidden states HN = { h N 0 , h N 1 , ..., h Nn  1 } , where the prefix p  { [ Enc ] , [ Dec ] , [ E 2 D ] } indicates the behavior of the model, e.g. [ E 2 D ] means that UniXcoder works as a encoder-decoder model. Each transformer layer contains an architecturally identical transformer that uses a multi-headed self-attention operation (Vaswani et al., 2017) followed by a feed forward layer over the output of the previous layer. For the l -th transformer layer, the output of the multi-headed self-attention is computed via: Q = H l  1 WQ , K = H l  1 WK , V = H l  1 WV (1) head = softmax(QK T  d k + M)V (2) where previous layer's output H l  1  R n  d h is linearly mapped to a triplet of queries, keys and values respectively. d k is the dimension of a head, and M  R n  n is a mask matrix to control the context a token can attend to when computing its contextual representation, as shown in the middle of Figure 2. If the i -th token is allowed to attend to the j -th token, then M ij is set to 0 otherwise  . For encoder-only mode, we add a special token [ Enc ] as the prefix in front of the input and set all elements of the mask matrix as 0 to allow all tokens attend to each other. For decoder-only mode, a prefix [ Dec ] is used and the upper triangular part of the mask is set to  to indicate that each token can only attend to itself and previous tokens. For encoder-decoder mode, tokens in the source input are allowed to attend to each other, while tokens in the target input only attend to itself and previous tokens in both source and target inputs. We use the [ E 2 D ] prefix to indicate that UniXcoder works as an encoder-decoder model. During the pre-training phase, model parameters are shared in different modes and optimized with several objectives to support various types of downstream tasks. We describe the pre-training tasks used in UniXcoder in this section. As shown on the right side of Figure 2, we first pre-train UniXcoder using three tasks, including masked language modeling (De-vlin et al., 2018), unidirectional language modeling (Radford et al.) and denoising objective (Raffel et al., 2019). These tasks are designed for different modes, enabling UniXcoder to support various types of code-related downstream tasks. We then propose to utilize multi-modal data to learn code fragment embeddings through contrastive learning with cross-modal generation, as shown in Figure 3. Masked Language Modeling For encoder-only mode, we follow Devlin et al. (2018) to apply masked language modeling (MLM) pre-training task. Specially, we sample 15% of the tokens S m from the input sequence, and then replace 80% (10%) of them with a [MASK] (random) token and leave another 10% of them unchanged. The task is to predict original tokens of masked tokens based on their bidirectional contextual tokens, as illustrated in Figure 2 (a). In particular, the model can leverage semantic information from comment and syntax information from AST to infer masked code tokens, which encourages the model to learn code representations from different knowledge resources. The objective is calculated as Equation 3, where X mask is the masked input sequence. Unidirectional Language Modeling We use unidirectional language modeling (ULM) pretraining task to pre-train decoder-only mode for supporting auto-regressive tasks like code completion, as shown in Figure 2 (b). The task predicts the next token x i one by one conditioned on previous tokens and itself { x 0 , x 1 , .., x i  1 } , which can be done using a triangular matrix for attention mask. Denoising Objective D e N oi S ing (DNS) pretraining objective has been shown to be quite effective for encoder-decoder models like BART (Lewis et al., 2019) and T5 (Raffel et al., 2019) in NLP. The task randomly masks spans with arbitrary lengths and then generates these masked spans in encoder-decoder mode. To better support generation tasks 7215 Encoder-only Decoder-only Encoder-Decoder Prevent from Attending Self-attention Masks Transformer Layer 1 Transformer Layer 2  Transformer Layer N Position Embeddings Token Embeddings Prefix Comment Flattened AST  !"  #"   $%&"  $%#" UniXcoder with Shared Parameters Pre-training Tasks like code summarization, we utilize similar denoising objective as T5 for encoder-decoder mode, as illustrated in Figure 2 (c). Specially, we first split the input sequence into max ( (cid:98) n  r l (cid:99) , 1) chunks and then randomly mask a span of from 1 to 2 l 1 tokens for each chunk, where n is the length of the input, r is corruption rate and l is the average length of masked spans. We set corruption rate as 15% and the average length as 5, respectively. The concatenation { y 0 , y 1 , ..., y n  1 } of all masked spans with special tokens [ MASK k ] in front of the k -th span will be used as the output: loss DNS =  n  1 (cid:88) i =0 logp ( y i | X mask , y t<i ) (5) Code Fragment Representation Learning In addition to the above three pre-training tasks designed for different modes, we propose to utilize multi-modal data to learn semantic embedding (cid:101) h i of a code fragment C i . As shown in Figure 3, we first use UniXcoder to encode a mapped AST sequence and then apply a mean pooling layer over the hidden states of the source input to obtain semantic embedding (cid:101) h i . In order to learn the semantic embedding, we propose two pre-training tasks. One is multi-modal contrastive learning (MCL), and another is cross-modal generation (CMG). For multi-modal contrastive learning, we follow Gao et al. (2021) to forward the same input using different hidden dropout mask as a positive example (cid:101) h + i and use other representations in the same batch as negative examples. The loss is calculated as Equation 6, where b is batch size,  is a temperature hyperparameter, and cos (  ,  ) is the cosine similarity between two vectors. For cross-modal generation, we ask the model to generate its comment W = { w 0 , w 1 , ..., w m  1 } . The comment describes the function of the code, which can help the model not only understand the code semantics but align representations among different programming languages by a unified natural language description as a fulcrum. Since the generation of the comment is conditioned on the 7216 code, it will force the model to fuse semantic information from the comment into the hidden states of the code. The loss is calculated as Equation 7, where X is the flattened AST token sequence. In order to learn the semantic embedding of natural language, we randomly exchange the source input and the target input with a probability of 50%. Considering that explicitly adding AST in downstream tasks will introduce extra costs like parsing time and increasing input length (70% longer input length after tokenization), we implicitly learn knowledge from AST by pre-training and only keep leaves of AST (i.e. source code) in the fine-tuning phase. This gap can be alleviated by randomly drop all non-terminal symbols of AST with a probability of 50% in the pre-training phase. More details about pre-training dataset and settings can be found in the Appendix B. 4 Experiments We evaluate UniXcoder on five tasks over nine public datasets, including two understanding tasks (4.2), two generation tasks (4.3) and an autoregressive task (4.4). To further evaluate the performance of code fragment embeddings, we also propose a new task called zero-shot code-to-code search (4.5). More details about datasets and fine-tuning can be found in the Appendix C. 4.1 Baselines We compare UniXcoder with state-of-the-art pre-trained models, including encoder-only , decoder-only and encoder-decoder models. For encoder-only models, we consider Roberta (Liu et al., 2019) pre-trained on text corpus with MLM, CodeBERT (Feng et al., 2020) pre-trained on NL-PL pairs using both MLM and replaced token detection, GraphCodeBERT (Guo et al., 2020) that leverages data flow to enhance code representation, and SYNCOBERT that incorporates AST by edge prediction and contrastive learning. For decoder-only models, we consider GPT-2 (Radford et al., 2019) and CodeGPT (Lu et al., 2021), where the former one is pre-trained on text corpus and the latter one is pre-trained on CodeSearchNet dataset. Both use ULM as the objective. For encoder-decoder models, we mainly compare the current unified models PLBART (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021). PLBART is based on BART and pre-trained on 470M Python and 210M Java functions, and 47M NL posts from StackOverflow using denoising objective. CodeT5, adapted from T5, considers the crucial token type information from identifiers and allows multi-task learning on downstream tasks. Clone Detection The task is to measure the similarity between two code fragments. We conduct experiments on POJ-104 (Mou et al., 2016) and BigCloneBench (Svajlenko et al., 2014) datasets. The first dataset is to predict whether two codes have the same semantics and uses F1-score as the evaluation metric, while the second aims to retrieve semantically similar codes given a code as the query with the Mean Average Precision (MAP) as the metric. Code Search The task aims to find the most relevant code from a collection of candidates given a natural language query. We conduct experiments on three datasets, namely CSN (Guo et al., 2020), AdvTest (Lu et al., 2021) and CosQA (Huang et al., 2021). CSN dataset is constructed from CodeSearchNet dataset of six programming languages, and low-quality queries are filtered by handcrafted rules. AdvTest normalizes python function and variable names to better test the understanding and generalization capabilities of models. The code base of CosQA is also from CodeSearchNet corpus but queries come from the search logs of Microsoft Bing search engine. We use Mean Reciprocal Rank (MRR) evaluation metric for the task. Results The results are shown in Table 1. Compared with encoder-only pre-trained models (i.e. the first group) and encoder-decoder models (i.e. the second group), UniXcoder outperforms them and achieves state-of-the-art performance on two tasks on all five datasets. By comparing with the results of ablation studies in the last six rows, we can see that the improvement mainly comes from contrastive learning and the use of multi-modality. Code Summarization The task aims to generate an NL summary of a code snippet. We use the dataset provided by the CodeXGLUE team (Lu et al., 2021) for this task. We use the smoothed BLEU-4 (Lin and Och, 2004) as the evaluation metric and report overall score of six PLs, including Ruby, JavaScript, Go, Python, Java, and PHP. Code Generation The task is to generate a code snippet based on an NL description. we use CONCODE (Iyer et al., 2018) dataset, where the input consists of an NL description and code environments. For this task, we use exact match (EM) and BLEU-4 as evaluation metrics. Results From Table 2, UniXcoder achieves comparable performance on generation tasks compared with CodeT5-base and brings a 0.3% improvement in code generation accuracy. However, UniXcoder has slightly worse BLEU-4 scores on both code summarization and generation tasks. The main reasons may come from two aspects. One is the amount of NL-PL pairs in the pre-training data. As shown in the ablation study (see w/o comment ) in the table, NL-PL pairs bring significant improvement on two tasks. Wang et al. (2021) collect 50% more NL-PL pairs from Github to pre-train CodeT5. Since the collected data is not public, we cannot use it to pre-train UniXcoder for fair comparison. Anothor reason is the model size. CodeT5-base uses a 12-layer encoder and a 12-layer decoder, which is twice larger than other baselines and UniXcoder. Therefore, we also list the results of CodeT5-small using a 6-layer encoder and a 6-layer decoder. We can see that UniXcoder outperforms CodeT5-small. We use PY150 (Raychev et al., 2016) and Github Java Corpus (Allamanis and Sutton, 2013) datasets in CodeXGLUE (Lu et al., 2021) for line-level code completion tasks. The task entails the completion of a whole-line of code, and is evaluated using exact match accuracy and Levenshtein edit similarity (Svyatkovskiy et al., 2020). In practice, the task requires a decoder-only manner to perform efficient inference. Therefore, we first compare our UniXcoder with decoder-only models (the first group) in Table 3. As we can see, UniXcoder achieves comparable performance on 7218 Model Ruby Python Java Overall Ruby Python Java Ruby Python Java Ruby Python Java CodeBERT 13.55 3.18 0.71 3.12 14.39 0.96 0.55 0.42 7.62 4.94 GraphCodeBERT 17.01 9.29 6.38 5.01 19.34 6.92 1.77 3.50 13.31 9.17 PLBART 18.60 10.76 1.90 8.27 19.55 1.98 1.47 1.27 10.41 8.25 CodeT5-base 18.22 10.02 1.81 8.74 17.83 1.58 1.13 0.81 10.18 7.81 UniXcoder 29.05 26.36 15.16 23.96 30.15 15.07 13.61 14.53 16.12 20.45 -w/o contras 24.03 17.35 7.12 15.80 22.52 7.31 7.55 7.98 13.92 13.73 -w/o cross-gen 28.73 24.16 12.92 21.52 26.66 12.60 11.14 10.82 13.75 18.03 -w/o comment 22.24 15.90 7.50 15.09 19.88 6.54 7.84 7.12 13.20 12.81 -w/o AST 27.54 23.37 10.17 21.75 27.75 9.94 9.79 9.21 14.06 17.06 -using BFS 26.67 23.69 13.56 21.31 27.28 13.63 11.90 12.55 14.92 18.39 -using DFS 27.13 22.65 11.62 20.21 25.92 11.85 9.59 10.19 13.30 16.94 Table 4: MAP score (%) of zero-shot setting on code-to-code search task. both datasets and brings absolute 2.3% gain of accuracy on java corpus, which demonstrates the effectiveness of our model for code completion. Besides, we also compare with current unified models (the second group). Since they are based the encoder-decoder framework, we fine-tune their decoders by feeding a placeholder into the encoder. Results show that UniXcoder outperforms PLBART and CodeT5, which demonstrates our model framework is better applied to code completion tasks. To further evaluate the performance of code fragment embeddings, we also propose a new task called zero-shot code-to-code search. Given a source code as the query, the task aims to retrieve codes with the same semantics from a collection of candidates in zero-shot setting. The task can help users translate from one PL to another by retrieving source codes with the same semantics. We collect 11,744/15,594/23,530 functions from the CodeNet corpus (Puri et al., 2021) in Ruby/Python/Java PL. Each function solves one of 4,053 problems. We take each function as a query and retrieve all functions that solve the same problem from each PL. We use average MAP score as the evaluation metric. More details about the dataset and an example can be found in Appendix C.6. We re-implement the publicly released pre-trained models on this task using the mean vector or CLS vector of last hidden states and report the results in Table 4. The first row is the query PL and the second row is the target PL. From the table, we can see that UniXcoder achieves state-of-the-art performance and about 11 points improvement on the overall score compared with GraphCodeBERT. Ablation studies further show that both multi-modal data and code fragment representation pre-training tasks can enhance UniXcoder. The Effect of Representation Pre-training We conduct ablation study to analyze the effect of code fragment representation pre-training tasks by removing contrastive learning task ( w/o constras ) and cross-modal generation task ( w/o cross-gen ). As we can see in Table 1 and 4, two pre-training tasks significantly improve understanding tasks. Taking zero-shot code-code search task as an example, after removing contrastive learning, the performance drops from 20.45% to 13.73%. Besides, the two pre-training tasks also bring a small improvement on generation tasks, as shown in Table 2 and 3. Overall, the ablation study demonstrates the effectiveness of the two pre-training tasks. The Effect of Multi-modal Data We also study the effect of multi-modal data. By removing comment ( w/o comment ), the results from Tables indicate that code comment plays an important role in both understanding and generation tasks. For AST ( w/o AST ), we observe that injecting AST can boost the performance on all code understanding tasks. However, AST does not bring improvements on generation tasks, which may require a better way to incorporate AST for generation tasks. Overall, AST and comment can both improve UniXcoder. Comparison of Traversal Algorithms We compare our mapping function with other mapping functions used to map a tree into a sequence, namely BFS and DFS algorithms. As we can see, after replacing our mapping function by BFS or DFS algorithms, the performance of UniXcoder drops on both understanding and generation tasks, which demonstrates the effectiveness of our mapping function. In particular, using BFS or DFS algorithms even hurt the performance of UniXcoder on some tasks by comparing w/o BFS (DFS) with 7219 w/o AST . The main reason may be that BFS and DFS algorithms are not one-to-one mapping functions and can confuse a tree with another structure. Case Study We also conduct a case study to intuitively demonstrate the effectiveness of UniXcoder, as shown in Figure 4. We give an example for code search task on CosQA dataset and output predictions from different models. The input query from the search logs of Microsoft Bing search engine is python dict rank by value . We know that the intent of the user is to sort a dictionary by its value in Python language. Although the prediction from PLBART has higher lexical overlap than UniXcoder like rank and value , the function is incorrect since the input of the ground truth should be a dictionary. We can see that UniXcoder retrieves a correct function whose input is a dictionary. Besides, although the value in the query is expressed as the statement key=lambda t: t[1] in the function definition, UniXcoder can understand the code semantics and successfully retrieves the ground truth, which demonstrates the effectiveness of UniXcoder. To support both code-related understanding and generation tasks, we present UniXcoder, a unified pre-trained model that incorporates semantic and syntax information from code comment and AST. We propose a one-to-one mapping method to transform AST to a sequence structure and two new pre-training tasks to learn code fragment representation. To further investigate the performance of code representation, we propose a new downstream task of zero-shot code-to-code search and create a dataset for this task. Experiments show that UniXcoder significantly outperforms previous works on most tasks. Further ablation studies also show that both AST and code comment can enhance UniXcoder and reveal the effectiveness of our proposed mapping function and pre-training tasks. Yanlin Wang is the corresponding author. Daya Guo and Jian Yin are supported by the National Natural Science Foundation of China (U1811264, U1811262, U1811261, U1911203, U2001211), Guangdong Basic and Applied Basic Research Foundation (2019B1515130001), Key-Area Research and Development Program of Guangdong Province (2018B010107005, 2020B0101100001).
abstain abstain abstain method abstain objective objective method objective result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain objective abstain abstain abstain method abstain result abstain objective objective objective abstain other other other other other other other other other other other other other other other other other other method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain other abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective abstain objective other other