Understanding the functional (dis)-similarity of source code is significant for code modeling tasks such as software vulnerability and code clone detection. We present DISCO ( DISsimilarity of COde ), a novel self-supervised model focusing on identifying (dis)similar functionalities of source code. Different from existing works, our approach does not require a huge amount of randomly collected datasets. Rather, we design structure-guided code transformation algorithms to generate synthetic code clones and inject real-world security bugs, augmenting the collected datasets in a targeted way. We propose to pre-train the Transformer model with such automatically generated program contrasts to better identify similar code in the wild and differentiate vulnerable programs from benign ones. To better capture the structural features of source code, we propose a new cloze objective to encode the local tree-based context ( e.g., parents or sibling nodes). We pre-train our model with a much smaller dataset, the size of which is only 5% of the state-of-the-art models' training datasets, to illustrate the effectiveness of our data augmentation and the pre-training approach. The evaluation shows that, even with much less data, DISCO can still outperform the state-of-the-art models in vulnerability and code clone detection tasks. Understanding the functional similar-ity/dissimilarity of source code is at the core of several code modeling tasks such as software vulnerability and code clone detection, which are important for software maintenance (Kim et al., 2017; Li et al., 2016). Existing pre-trained Transformer models (Guo et al., 2021; Feng et al., 2020; Ahmad et al., 2021) show promises for understanding code syntax ( i.e., tokens and structures). However, they still get confused when trying to identify functional (dis)-similarities. For instance, syntax-based models can embed two code fragments with identical functionality but very different tokens and structures as distinct vectors and fail to identify them as semantically similar. Likewise, these models cannot distinguish between two code fragments that differ in functionalities but share a close syntactic resemblance. For example, consider an if statement if(len(buf) < N) checking buffer length before accessing the buffer. Keeping the rest of the program the same, if we simply replace the token  < ' with   ,' the modification can potentially trigger security vulnerability, e.g., buffer overflow bug 1 . It is challenging for existing pre-training techniques to tell apart such subtle differences in the functionalities. In addition, existing pre-training techniques rely on a huge volume of training corpus that is randomly selected. For fine-tuning tasks like code clone detection or vulnerability detection, such random selection of training data is never tailored to teach the model about code functionalities. To address these limitations, we present DISCO, a self-supervised pre-trained model that jointly learns the general representations of source code and specific functional features for identifying source code similarity/dis-similarity. Similar to state-of-the-art pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019), we apply the standard masked language model (MLM) to capture the token features of source code. To learn about the structural code properties, we propose a new auxiliary pre-training task that consumes additional inputs of local tree-based contexts ( e.g., parent or sibling nodes in abstract syntax trees) and embeds such structural context, together with the token-based contexts, into each token representation. On top of such well-learned general code representations, we further incorporate prior knowledge of code clones and vulnerable programs into the pre-training to help the model learn the functional (dis)-similarity. We design structure-guided 1 https://en.wikipedia.org/wiki/Buffer_overflow 6300 code transformation heuristics to automatically augment each training sample with one synthetic code clone ( i.e., positive samples) that is structurally different yet functionally identical and one vulnerable contrast ( i.e., hard negative samples) that is syntactically similar but injected with security bugs. During the pre-training, DISCO learns to bring similar programs closer in the vector space and differentiate the benign code from its vulnerable contrast, using a contrastive learning objective. Since we augment the dataset in a more targeted way than existing works and the model explicitly learns to reason about a code w.r.t. its functional equivalent and different counterparts during pre-training, DISCO can learn sufficient knowledge for downstream applications from a limited amount of data, consequently saving computing resources. In particular, we evaluate DISCO for clone detection and vulnerability detection, as the knowledge of similar/dissimilar code fragments is at the core of these tasks. To this end, we pre-train DISCO on a small dataset, with only 865 MB of C code and 992 MB Java code from 100 most popular GitHub repositories, and evaluate the model on four different datasets for vulnerability and code clone detection. Experiments show that our small models outperform baselines that are pre-trained on 20  larger datasets. The ablation study (5.4) also reveals that pre-training our model with 10  larger datasets further improves the performance up to 8.2%, outperforming state-of-the-art models by 1% for identifying code clones and up to 9.6% for vulnerability detection, even if our dataset is still smaller. In summary, our contributions are: 1) We design structure-guided code transformation heuristics to automatically augment training data to integrate prior knowledge of vulnerability and clone detection without human labels. 2) We propose a new pre-training task to embed structural context to each token embedding. 3) We develop DISCO, a self-supervised pre-training technique that jointly and efficiently learns the textual, structural, and functional properties of code. Even though pre-trained with significantly less data, DISCO matches or outperforms the state-of-the-art models on code clone and vulnerability detection. models (Vaswani et al., 2017) for source code with two categories: encoder-only and encoder-decoder (Ahmad et al., 2021; Wang et al., 2021; Rozire et al., 2021; Phan et al., 2021). Our work focuses on pre-training encoder-only Transformer models to understand code. Existing models are pre-trained with different token level objectives, such as masked language model (MLM) (Kanade et al., 2020; Buratti et al., 2020), next sentence prediction (NSP) (Kanade et al., 2020), replaced token detection, and bi-modal learning between source code and natural languages (Feng et al., 2020). However, these approaches ignore the underlying structural information to fully understand the syntax and semantics of programming languages. Recently, more works aimed to understand the strict-defined structure of source code leveraging abstract syntax tree (AST) (Zgner et al., 2021; Jiang et al., 2021), control/data flow graphs (CFG/DFG) (Guo et al., 2021). DISCO leverages code structures differently from existing works in two ways: (a.) with AST/CFG/DFG, we automatically generate program contrasts to augment the datasets targeting specific downstream tasks. (b.) DISCO takes an additional input of local AST context, and we propose a new cloze task to embed local structural information into each token representation. Self-supervised Contrastive Learning. Self-supervised contrastive learning, originally proposed for computer vision (Chen et al., 2020), has gained much interest in language processing (Giorgi et al., 2021; Wu et al., 2020; Gao et al., 2021). The common practice of self-supervised contrastive learning is building similar counterparts, without human interference, for the original samples and forcing the model to recognize such similarity from a batch of randomly selected samples. Corder (Bui et al., 2021) leverages contrastive learning to understand the similarity between a program and its functionally equivalent code. While Corder approach will help code similarity detection type of applications, their pre-training does not learn to differentiate syntactically very close, but functionally different programs. Such differentiation is crucial for models to work well for bug detection (Ding et al., 2020). ContraCode (Jain et al., 2020) also leverages contrastive learning. However, they generate negative contrast for a program from unrelated code examples, not from variants of the same code. They also do not encode the structural information into the code as we do. Inspired by the empirical findings that hard negative image and text 6301 samples are beneficial for contrastive learning (Gao et al., 2021; Robinson et al., 2021), DISCO learns both from equivalent code as the positive contrast, and functionally different yet syntactically close code as the hard-negative contrast. We generate hard-negative samples by injecting small but crucial bugs in the original code (3.1). Our pre-training aims to identify similar programs that can be structurally different (positive sample) and differentiate the buggy programs (negative sample) that share structural resemblances with the benign ones. Thus, we need a labeled positive and a negative example for each original sample. Manually collecting them is expensive, especially at the scale of pre-training. To this end, we design code transformation heuristics to automatically generate such positive and negative samples so that the transformation can be applied to any amount of programs without human efforts. We first represent a code sample as Abstract Syntax Tree (AST), and build a control/data flow graph from the AST. The code transformation heuristics are then applied to this graph. For every original code sample ( x ), we apply semantic preserving transformation heuristics (3.2) to generate a positive sample ( x + ) and a bug injection heuristics (3.1) to generate a hard-negative code example ( x  ). We design the heuristics in a way that makes x + be the functional equivalent or semantic clone of x and x  be the buggy/noisy version of x . Noted that not all heuristics are applicable to all code samples; we decide on applicable heuristics based on the flow graph of the original code. Figure 1 shows an example of the code transformation. To generate a hard negative sample ( x  ) from a given code ( x ), we define six categories of bug injection heuristics. Here our goal is to maintain maximum token-level similarity to the original code, so that the model can learn to analyze source code beyond token-level similarity. These heuristics are inspired by the buggy code patterns from a wide range of Common Weakness Enumeration (CWE) types (Appendix A.1). While it is challenging to guarantee that x  will exhibit vulnerability or security bug, our heuristics will force x  to exhibit different functionality than x . Compared with a concurrent work from Allamanis et al. (2021), our methods are significantly different. First, we focus on concrete types of security bugs that have been identified by the security experts, while they mainly target regular bugs. Second, our scope is not only bug detection but clone detection as well, and we apply contrastive learning to differentiate the code functionalities of code clones and vulnerabilities. Misuse of Data Type. Usage of the wrong data type can trigger several security flaws. For instance, using a smaller data type ( e.g., short ) to replace a larger one ( e.g., long ) may result in an overflow bug ( e.g., CVE-2021-38094 (2021)). Such errors are complicated to track since they are usually exhibited in input extremities ( i.e., very large or very small values). For languages allowing implicit typecasting, such an incorrect type may even cause imprecision, resulting in the unpredictable behavior of the code. We intentionally change the data types in x to inject potential bugs, while ensuring the code can still be compiled ( e.g., we will not replace int with char ). Misuse of Pointer. Incorrect pointer usage is a ma-jor security concern. Accessing uninitialized pointers may lead to unpredictable behavior. ANULL pointer or freed pointer could lead to Null Pointer Dereferencing vulnerability ( e.g., CVE-2021-3449 (2021)). To inject such bugs, we randomly remove the initialization expression during pointer declaration, or set some pointers to NULL . Change of Conditional Statements. Programmers usually check necessary preconditions using if-statement before doing any safety-critical operation. For instance, before accessing an array with an index, a programmer may add a condition checking the validity of the index. Lack of such checks can lead to buffer-overflow bugs in code ( e.g., CVE-2020-24020 (2020)). We introduce bugs in the code by removing such small if-statement s. In addition, we also inject bugs by modifying randomly selected arithmetic conditions replace the comparison operator ( < , > ,  ,  , == , ! = ) with another operator, to inject potential out-of-bound access, forcing the program to deviate from its original behavior. Misuse of Variables. When there are multiple variables present in a code scope, incorrect use of variables may lead to erroneous behavior of the program. Such errors are known as VARMISUSE bug (Allamanis et al., 2018). We induce code with such bugs by replacing one variable with another. To keep the resultant code compilable, we perform scope analysis on the AST and replace a variable with another variable reachable in the same scope. Misuse of Values. Uninitialized variables or variables with wrong values may alter the program behaviors and consequently cause security flaws ( e.g., CVE-2019-12730 (2019)). We modify the original code by removing the initializer expression of some variables. In addition, to induce the code with divide-by-zero vulnerability, we identify the potential divisor variables from the flow graph and forcefully assign zero values to them immediately before the division. Change of Function Calls. We induce bugs in the code by randomly changing arguments of function calls. For a randomly selected function call, we add, remove, swap, or assign NULL value to arguments, forcing the code to behave unexpectedly. To generate positive samples ( x + ) from a given code, we use three different heuristics. In this case, our goal is to generate functionally equivalent code while inducing maximum textual difference. These heuristics are inspired by code clone literature (Fu-naro et al., 2010; Sheneamer et al., 2018). Variable Renaming. Variable renaming is a typical code cloning strategy and frequently happens during software development (Ain et al., 2019). To generate such a variant of the original code, we either (a.) rename a variable in the code with a random identifier name or (b.) with an abstract name such as VAR_i (Rozire et al., 2021). While choosing random identifier names, we only select available identifiers in the dataset. We ensure that both the definition of the variable and subsequent usage(s) are renamed for any variable renaming. We also ensure that a name is not used to rename more than one variable. we make more tokens different compared with the original code but keep the same syntax and semantics. We do not rename library calls for the code ( e.g., memcpy() in C). Noted that even if tokens like VAR_i and FUNC_i are rare in normal code, the model will not bias towards identifying samples with these tokens as positive samples. The reason is that, as shown in Figure 2, x + , y + and z + all potentially have these abstract tokens, but the model learns to move EMB x closer to EMB x + and further from EMB y + and EMB z + , regardless of the existence of abstract tokens. Statement Permutation. The relative order among the program statements that are independent of each other can be changed without altering the code functionality. More specifically, we focus on the variable declaration or initialization statements. We first conduct the dependency analysis to identify a set of local variables that do not depend on other values for initialization. Then we move their declaration statements to the beginning of the function and permute them. This section presents the model architecture, input representation, and pre-training tasks. DISCO uses a 12-layered Transformer encoder model similar to BERT. We feed the model with both source code text and structure (AST) information (4.1). We pre-train DISCO using three different pre-training tasks (4.2). Figure 2 depicts an example workflow of DISCO. We randomly select tokens in the original sample, mask them and their node types, and then use the embedding of these masks to predict them back. We further extract the sequence embed-dings within a minibatch and contrast them based on the code functionality. Source Code. Given a program ( x ), we apply a lexical analyzer to tokenize it based on the language grammar and flatten the program as a token sequence ( x 1 x 2 ...x m , where x i is i th token in the code). We further train a sentencepiece (Kudo and Richardson, 2018) tokenizer based on such flattened code token sequences with vocabulary size 20,000. We use this tokenizer to divide the source code tokens into subtokens. We prepend the subtoken sequence with a special token [CLS] and append with a special token [SEP] . Finally, DISCO converts the pre-processed code sequence C = { [ CLS ] , c 1 , c 2 , ..., c k , [ SEP ] } to vectors V src = { v src [ CLS ] , v src 1 , v src 2 , ..., v srck , v src [ SEP ] } with a token embedding layer. Local AST Types. For every token in the input code, we extract the node type ( tt ) from the syntax tree. Since such types are all terminal node types ( e.g., keyword, identifier, punctuation), we do not get enough information about the structure only with these types. In order to add more information about the tree, we also extract its parent type ( pt ) for each token. Such parent type provides us with information about the structural context of a token. For instance, when the parent type of an identifier is Function-Declarator , we know that the identifier is a function name. In contrast, when the identifier 's parent is a Binary Expression , it should be a variable. Consequently, we annotate each code subtoken c i with a local AST-type token t = tt # pt . It is worth noting that sub-tokens coming from the same code token will all have the same type. Therefore, we have the AST-type sequence for the code T = { [ CLS ] , t 1 , t 2 , ..., t k , [ SEP ] } , and DISCO converts it as vectors V type = { v type [ CLS ] , v type 1 , v type 2 , ..., v typek , v type [ SEP ] } with a type embedding layer. Appendix Table 7 shows an example of code tokens and their AST types. DISCO generates token representation v i of subtoken c i as a sum of token embedding v srci and type embedding v typei . Thus, V = V src + V type . We aim to pre-train the DISCO to learn the representation of source code based on (a.) token-based context, (b.) AST-based context, and (c.) code functionality. In that spirit, we pre-train DISCO to optimize on three different objectives, i.e., masked language model (MLM), local AST node type-MLM (NT-MLM), and Contrastive Learning (CLR). For a given program x , we first embed the tokens and node-types to vectors V = { v [ CLS ] , v 1 , ..., v [ SEP ] } . We optimize MLM loss ( LMLM ) (4.2.1) and NT-MLM loss ( LNT  MLM ) (4.2.2) based on x . These two loss functions learn about the textual and syntactic context of source code. For every code sample x in a minibatch of input, we generate a positive example x + and a hard-negative example x  using the heuristics described in Section 3. We optimize CLR loss ( LCLR ) (4.2.3) considering the original code and its positive and hard-negative counterparts. The final loss function to optimize for pre-training DISCO is L (  ) = LMLM (  ) + LNT  MLM (  ) + LCLR (  ) 6304 4.2.1 Encoding Token-based Context We apply the standard masked language model to the original code ( x ). Given a source code sequence C , we randomly choose 15% of tokens and replace them with a special token [ MASK ] for 80% of the time and a random token for 10% of the time and leave the rest 10% unchanged. We record the indices of masked token as loc m , replaced token as loc r and unchanged tokens as loc u for node-type MLM. We define the union of these indices as M = loc m  loc r  loc u . MLM will learn to recover the masked source code { c i | i  M } given the Transformer encoder's output h i . We present the loss for MLM as LMLM = (cid:80) i  M  logP ( c i | h i ) 4.2.2 Encoding AST-based Context Token-based MLM re-builds the token using its surrounding tokens and successfully encodes the contextual information into each token representation. Motivated by MLM, we propose the tree-based context-aware pre-training task, to encode the structural context, such as parent, sibling, and children nodes. As we have shown in Figure 2, we flatten the ASTs as sequences and we expect the flattened trees can preserve the local structure information (i.e., sub-trees containing terminal nodes), and existing work (Chakraborty et al., 2020; Hellen-doorn et al., 2020) has empirically shown such potentials. To this end, we introduce AST node-type masked language model (NT-MLM). Given the corresponding AST-type sequence T of source code C , we mask the AST types { t p | p  loc m } with the special token [ MASK ] , and replace the AST types { t q | q  loc r } with random tokens. Specifically, by doing this, we make sure that if a source code token is chosen to be masked or replaced, its corresponding AST type will perform the same operation. NT-MLM will learn to recover the masked AST type { t i | i  M } given the Transformer encoder's output h i . We present the loss for NT-MLM as LNT  MLM = (cid:80) i  M  logP ( t i | h i ) A recent work, CodeT5 (Wang et al., 2021), proposes to predict token type as well. However, our new objective is different from them in both high-level designs and the detailed implementation. First, their objective only predicts one single token type: identifiers, while our approach predicts all possible AST types. Also, we do not only consider the AST node type of tokens, but also include their AST parents to embed the local sub-tree context (4.1). Second, CodeT5 implements the identifier tagging task as a binary classification (0/1) for each token, while our NT-MLM reconstructs the local ASTs out of hundreds of distinct types. We adopt contrastive learning to focus on the functional characteristics of code. With the structure-guided code transformation algorithms in Section 3, we are able to generate a positive sample ( x + in Figure 2) and a hard negative sample ( x  in Figure 2) for each program in the dataset. More specifically, we have a minibatch of N programs, and for each program, we extract the sequence representation from the Transformer outputs h = h [ CLS ] . We will augment every sequence in the minibatch with positive and negative samples, and then the minibatch is extended to N triplets of ( h , h + , h  ) . We refer to the contrastive loss with hard negative samples from Gao et al. (2021) and we adapt it to our scope as follows. We use cosine similarity as the sim () function and  is the temperature parameter to scale the loss, and we use  = 0 . 05 . We also consider to pre-train the model with only positive counterparts as a variation. In such a case, the minibatch will contain N pairs of ( h , h + ) and the loss is computed as LCLR =  log e sim ( h , h + ) / (cid:80) Nn =1 (cid:16) e sim ( h , h + n ) / (cid:17) 5 Experiments In this section, we will explain our experimental settings and report the results. We evaluate our model on vulnerability and code clone detection. Data. We collect our pre-training corpus from open-source C and Java projects. We rank Github repositories by the number of stars and focus on the most popular ones. After filtering out forks from existing repositories, we collect the dataset for each language from top-100 repositories. We only consider the .java and .c files for Java and C repositories respectively, and we further remove comments and empty lines from these files. The corresponding datasets for Java and C are of size of 992MB and 865MB, respectively. Our datasets are significantly smaller than existing pre-training models (Feng et al., 2020; Ahmad et al., 2021; Guo 6305 et al., 2021). For example, while CodeBERT and GraphCodeBERT are trained on 20GB data, we used an order of magnitude less data. Details of our datasets and the comparison can be found in Appendix Table 5. Models. To study the different design choices, we train four variations of DISCO. (i) MLM+CLR  +NT-MLM is trained by all three tasks with hard negative samples. (ii) MLM+CLR  . The input of this model only considers the source code sequence and ignores the AST-type sequence. This model helps us understand the impact of NT-MLM. (iii) MLM+CLR + . This variant evaluates the effectiveness of hard negative code samples, by contrasting its performance with MLM+CLR  . (iv) MLM . This is the baseline trained with only MLM objective. We provide detailed model configuration in Appendix A.4 to ensure the reproducibility. Baselines. We consider two types of baselines: encoder-only pre-trained Transformers and existing deep-learning tools designed for code clone and vulnerability detection. We do not consider encoder-decoder pre-trained Transformers as baselines, since such generative models always need much more pre-training data and training steps to converge, so it is unfair to compare our model with them. For example, PLBART uses 576G source code for pre-training, while we only use less than 1G. Based on the data size. As future work, we plan to pre-train the model on much larger datasets. VD is the task to identify security bugs: given source code function, the model predicts 0 (benign) or 1 (vulnerable) as binary classification. Dataset and Metrics. We consider two datasets for VD task: REVEAL (Chakraborty et al., 2021) and CodeXGLUE (Lu et al., 2021; Zhou et al., 2019). In the real-world scenario, vulnerable programs are always rare compared to the normal ones, and Chakraborty et al. (2021) have shown such imbalanced ratio brings challenges for deep-learning models to pinpoint the bugs. To imitate the real-world scenario, they collect REVEAL dataset from Chromium (open-source project of Chrome) and Linux Debian Kernel, which keeps the ratio of vulnerable to benign programs to be roughly 1:10. Following Chakraborty et al. (2021), we consider precision, recall and F1 as the metrics. security vulnerabilities. It is less real-world than REVEAL, since it a balanced dataset, but it has been frequently used by existing Transformer-based models to evaluate their tools for VD task. To compare with these baselines, we use CodeXGLUE train/valid/test splits for training and testing. We use accuracy as the metric, following the design of the benchmark. REVEAL. Table 1 shows the results. We compare with four deep-learning-based VD tools. VulDeePecker (Li et al., 2018b) and SySeVR (Li et al., 2018a) apply program slices and sequence-based RNN/CNN to learn the vulnerable patterns. Devign (Zhou et al., 2019) uses graph-based neural networks (GNN) to learn the data dependencies of program. REVEAL (Chakraborty et al., 2021) applies GNN + SMOTE (Chawla et al., 2002) + triplet loss during training to handle the imbalanced distribution. We also consider pre-trained RoBERTa, CodeBERT and GraphCodeBERT, and a 12-Layer Transformer model trained from scratch. *We take this result from Buratti et al. (2020). They did not use CodeXGLUE splits, so the test data can be different with other baselines. In our case, the best DISCO variation with contrastive learning and NT-MLM objective outperforms all the baselines, including the graph-based approaches and models pre-trained with larger datasets. This empirically proves that DISCO can 6306 efficiently understand the code semantics and data dependencies from limited amount of data, helping the identification of the vulnerable patterns. We notice that hard negative samples ( i.e., buggy code contrasts) helps DISCO improve the performance. The reason is that REVEAL contains thousands of (buggy version, fixed version) pairs for the same function. Two functions in such a pair are different by only one or a few tokens. Such real-world challenges align well with our automatically generated buggy code, and pre-training with these examples teaches the model better distinguish the buggy code from the benign ones. We provide an example in Appendix Figure 3 to illustrate this. CodeXGLUE. We consider four pre-trained models: RoBERTa, CodeBERT, GraphCodeBERT and C-BERT. The first three are pre-trained on much larger datasets than ours. However, even trained with small dataset, three variations of DISCO outperforms the baselines. Unlike REVEAL, CodeXGLUE does not have those challenging pairs of functions' buggy and patched version; thus the hard negative contrast in DISCO does not help the model much. Clone detection aims to identify the programs with similar functionality. It also can help detecting security vulnerabilitiesgiven a known vulnerability, we can scan the code base with clone detector and check for similar code snippets. Dataset and Metrics. We consider POJ-104 (Mou et al., 2016) and BigCloneBench (Svajlenko et al., 2014) as the evaluation datasets. We again strictly follow the CodeXGLUE train/dev/test splits for experiments. Following CodeXGLUE's design, we use MAP@R as the metric for POJ-104 and preci-sion/recall/F1 as the metric for BigCloneBench. 12-layer Transformer model trained from scratch as baselines. Table 3 shows that, with hard negative contrast and NT-MLM, DISCO outperforms all baselines including CodeBERT, which is pre-trained on much larger datasets. This highlights the significance of learning the code contrasts together with syntactical information to better capture the functional similarities. Interestingly, we notice that DISCO-MLM performs the best among all variations. This indicates that our current positive heuristics might not align with all the clone patterns in this benchmark. As future work, we will propose more code transformation rules to imitate more real-world clone patterns. BigCloneBench. Our best model achieves slightly better precision than the baselines indicating that our designs with contrastive learning and structure information can compensate the loss brought by less data. However, our recall is slightly worse than GraphCodeBERT, since they are pre-trained on large datasets with code graph. We conclude that enlarging our Java pre-training dataset is necessary for code clone detection and we regard this as future work. As shown in Section 5, DISCO trained on a small dataset achieves comparable or even better performance than models pre-trained on large datasets in vulnerability and clone detection (Let's call this version DISCO small ). We further explore the benefits of pre-training using larger data. We pre-train a MEDIUM model, DISCO medium , on our extended datasets with more C-language Github repositories (13G). Note that our medium dataset is still smaller than the large dataset of the baseline models (13G vs. 20G). We evaluate DISCO medium on C-language tasks. The results are shown in Table 4. Increasing the pre-training dataset improves the performance of downstream tasks, outperforming the best baselines' results. In this work, we present DISCO, a self-supervised contrastive learning framework to both learn the general representations of source code and specific characteristics of vulnerability and code clone detections. Our evaluation reveals that DISCO pre-trained with smaller dataset can still outperform the large models' performance and thus prove the effectiveness of our design. We would appreciate the insightful feedback and comments from the anonymous reviewers. This work was partially done when Yangruibo Ding was an intern at IBM Research. This work is also supported in part by NSF grants CCF-2107405, CCF-1845893, IIS-2040961, and IBM. The main goal of DISCO is to generate functionality-aware code embeddings, producing similar representations for code clones and differentiating security bugs from the benign programs. Our data is collected from either the open-source projects, respecting corresponding licences' restrictions, or publicly available benchmarks. Meanwhile, throughout the paper we make sure to summarize the paper's main claims. We also discussed DISCO's limitation and potential future work for clone detection in Section 5.3. We report our model configurations and experiment details in Appendix A.4.
abstain objective method objective objective objective method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method objective abstain other abstain objective method method result result objective objective objective abstain other method other other other objective other objective other other other other other other other other method other method objective abstain other method objective other method method abstain other method abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain method method abstain abstain method method abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method method method abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method method method method method abstain method abstain abstain abstain method method abstain abstain abstain abstain method method method method method abstain method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain method result other other other abstain method abstain abstain abstain