Abstract Automatic code summarization, which aims to describe the source code in natural language, has become an essential task in software maintenance. Our fellow researchers have attempted to achieve such a purpose through various machine learning-based approaches. One key challenge keeping these approaches from being practical lies in the lacking of retaining the semantic structure of source code, which has unfortunately been overlooked by the state-of-the-art methods. Existing approaches resort to representing the syntax structure of code by modeling the Abstract Syntax Trees (ASTs). However, the hierarchical structures of ASTs have not been well explored. In this paper, we propose CODESCRIBE to model the hierarchical syntax structure of code by introducing a novel triplet position for code summarization. Specifically, CODESCRIBE leverages the graph neural network and Transformer to preserve the structural and sequential information of code, respectively. In addition, we propose a pointer-generator network that pays attention to both the structure and sequential tokens of code for a better summary generation. Experiments on two real-world datasets in Java and Python demonstrate the effectiveness of our proposed approach when compared with several state-of-the-art baselines 1 . Code documentation in the form of code comments has been an integral component of software development, benefiting software maintenance (Iyer et al., 2016), code categorization (Nguyen and Nguyen, 2017) and retrieval (Gu et al., 2018). However, few real-world software projects are well-documented with high-quality comments. Many projects are either inadequately documented due to missing important code comments or inconsistently documented due to different naming conven-1 The source code of CODESCRIBE is available at https: //github.com/GJCEXP/CODESCRIBE tions by developers, e.g., when programming in legacy code bases, resulting in high maintenance costs (de Souza et al., 2005; Kajko-Mattsson, 2005). Therefore, automatic code summarization, which aims to generate natural language texts (i.e., a short paragraph) to describe a code fragment by extracting its semantics, becomes critically important for program understanding and software maintenance. Recently, various works have been proposed for code summarization based on the encoder-decoder paradigm, which first encodes the code into a distributed vector, and then decodes it into natural-language summary. Similarly, several works (Iyer et al., 2016; Allamanis et al., 2016) proposed to tokenize the source code into sequential tokens, and design RNN and CNN to represent them. One limitation of these approaches is that they only consider the sequential lexical information of code. To represent the syntax of code, several structural neural networks are designed to represent the Abstract Syntax Trees (AST) of code, e.g., TreeLSTM (Wan et al., 2018), TBCNN (Mou et al., 2016), and Graph Neural Networks (GNNs) (LeClair et al., 2020). To further improve the efficiency on AST representation, various works (Hu et al., 2018a; Alon et al., 2019) proposed to linearize the ASTs into a sequence of nodes or paths. Despite much progress on code summarization, there are still some limitations in code comprehension for generating high-quality comments. Particularly, when linearizing the ASTs of code into sequential nodes or paths, the relationships between connected nodes are generally discarded. Although the GNN-based approaches can well preserve the syntax structure of code, they are insensitive to the order of nodes in AST. For example, given the expressions a=b/c and a=c/b , current approaches cannot capture the orders of variables b and c . However, these orders are critical to accurately preserve the semantics of code. paper proposes to model the hierarchical syntax structure of code using triplet position, inspired by the positional encoding used in sequence modeling (Gehring et al., 2017; Vaswani et al., 2017), and incorporates the triplet position into current GNNs for better code summarization. The triplet position records the depth, width position of its parent, and width position among its siblings for each node. To utilize the triplet position in AST, this paper proposes CODESCRIBE , an encoder-decoder-based neural network for source code summarization. Specially, we initialize the embedding of each AST node by incorporating the triplet positional embeddings, and then feed them into an improved GNN, i.e., GraphSAGE (Hamilton et al., 2017) to represent the syntax of code. In addition, we also account for the sequential information of code by using a Transformer encoder (Vaswani et al., 2017). In such a case, the decoding process is performed over the learned structural features of AST and sequential features of code tokens with two multi-head attention modules. To generate summaries with higher quality, we further design a pointer-generator network based on multi-head attention (Vaswani et al., 2017), which allows the summary tokens to be generated from the vocabulary or copied from the input source code tokens and ASTs. To validate the effectiveness of our proposed CODESCRIBE , we conduct experiments on two real-world datasets in Java and Python. Overall, the primary contributions of this paper are as follows. It is the first time that we put forward a simple yet effective approach of triplet position to preserve the hierarchical syntax structure of source code accurately. We also incorporate the triplet position into an adapted GNN (i.e., GraphSAGE) for source code summarization. We conduct comprehensive experiments on two real-world datasets in Java and Python to evaluate the effectiveness of our proposed CODESCRIBE . Experimental results on both datasets demonstrate the superiority of CODESCRIBE when comparing with several state-of-the-art baselines. For example, we get 3.70/5.10/4.77% absolute gain on BLEU/METEOR/ROUGE-L metrics on the Java dataset, when comparing with the most recent mAST+GCN (Choi et al., 2021). Recent studies have showed promising results by using AST context for tasks based on code representation learning (Yao et al., 2019; Zhang et al., 2019; Choi et al., 2021). Therefore, our work also relies on AST information besides source code tokens. As a type of intermediate representation, AST represents the hierarchical syntactic structure for source code, which is an ordered tree with labeled nodes (cf. Figure 1). In this work, we divide the nodes into two categories: (1) function node that controls the structure of AST and function realization, e.g., Module and Assign in Figure 1, and (2) attribute node that provides the value or name of its parent function node, which is always visualized as leaf node, such as  a ' and  b ' in dotted boxes of Figure 1. Due to the strict construction rules of AST, positions are crucial for AST nodes. For example in Figure 1, the node BinOp has two children with the same label Name . If the positions of the two siblings are swapped, the source code will become a=c/b , which is totally different from the intent of the code a=b/c . However, GNNs are insensitive to the positions of neighbouring nodes when encoding such tree structures. Based on this obser-487 Probabilities of Next Summary Tokens 6  Multi-Head Att (cid:17) with Res. vation, we specify triplet positions for AST nodes to retain accurate structural information in AST learning. The triplet position of a node includes: (1) the depth of the node in the AST, (2) the width position of its parent node in the layer, and (3) the node's width position among its siblings, which can also distinguish function node from attribute node. That is, the width position of a function node is a non-negative integer starting from 0, while the width position of an attribute node is a negative integer counting from -1. Note that, width positions are estimated in a breadth traversal from left to right. With such triplet indices specified, all nodes can be marked with unique positions in a given AST. Taking a Python code snippet a=b/c as an example, Figure 1 illustrates its AST structure with triplet positions of nodes. Specifically, by traversing the tree, we can represent the function node (Name,{2,0,0}) as the first child node of node (Assign,{1,0,0}) : the depth position 2 means the third level (counting from the top to bottom starting with 0); the second width position 0 means that the parent node Assign is the first function node at this level (counting from the left to right); and the third position 0 indicates that the node is the first (counting from left to right) among its siblings (i.e., all children nodes of node Assign ). Another example is the node (  a ' ,{3,0,-1}) . The difference lies in the third position that represents it is an attribute node and it is the first among the siblings. In particular, we set the position of root node Module to {0,0,0} as it has no parent node. This triplet positioning is very precise and unique, allowing to track and discriminate among the Name nodes which also include (Name,{3,1,0}) and (Name,{3,1,2}) . Given a code snippet with l c tokens T c = ( c 1 , c 2 , . . . , c l c ) and sequential positions P c = (1 , 2 , . . . , l c ) , and its AST with l n nodes T n = ( n 1 , n 2 , . . . , n l n ) and triplet positions P n = ( { x 1 , y 1 , z 1 } , { x 2 , y 2 , z 2 } , . . . , { x l n , y l n , z l n } ) , CODESCRIBE predicts the next summary token s m based on the existing tokens T s = ( </s> , s 1 , s 2 , . . . , s m  1 , . . . ) with the sequential positions P s = (1 , 2 , . . . , l s ) , where </s> is a special starting tag for summary input. Note that T s is padded to a maximum length of l s with special padding tags (e.g., <pad> s). Figure 2 illustrates the architecture of CODESCRIBE model, which is mainly composed of four modules: source code encoder, AST encoder, summary decoder and multi-source pointer-generator network (MPG) for output. As shown in Figure 2, the source code, AST, and summary tokens are firstly mapped into embedding vectors E 0 c  R l c  d , E 0 n  R l n  d , and E 0 s  R l s  d where d is the embedding size. In the encoding process, the embedded code and AST are fed into Transformer encoder (Vaswani et al., 2017) and GNN layers respectively for learning the source code representation E (cid:48) c  R l c  d and the AST representation E (cid:48) n  R l n  d . Then, the decoding process is performed to yield the decoded vector e (cid:48) s  R d for the predicted summary token by fusing the learned source code and AST features (i.e., E (cid:48) c and E (cid:48) n ) as an initial state for decoding E 0 s . At the decoding stage, we build MPG stacked on the decoder and encoders to predict the next summary token s m by selecting from summary vocabulary or copying from the input source code and AST tokens. The detailed process will be further described in the following sub-sections. Before feeding code tokens, AST nodes, and summary tokens into neural networks, it is essential to embed them into dense numerical vectors. In this work, the source code tokens T c , AST nodes T n , and summary tokens T s are all embedded into numeric vectors with their related positions P c , P n , and P s incorporated through learnable positional embeddings (Gehring et al., 2017). In particular for AST, we take each triplet position { x i , y i , z i } in P n as an individual tuple, and directly map it into a positional embedding vector e i  R d . The embedded triplet positional information is then added to the node embeddings for initializing the AST representation. The embedding processes are formulated as follows: E 0 c = CNEmb ( T c )   d + CPEmb ( P c ) , E 0 n = CNEmb ( T n )   d + NPEmb ( P n ) , E 0 s = SEmb ( T s )   d + SPEmb ( P s ) , (1) where CNEmb denotes the shared embedding operation for source code tokens and AST nodes; SEmb means the token embedding operation for summary text; CPEmb , NPEmb , and SPEmb are the corresponding positional embedding operations. Afterwards, the initialized representations E 0 c , E 0 n , and E 0 s are fed into the encoders and decoder of CODESCRIBE for in-depth processing. Source Code Encoder. As shown in Figure 2, the code encoder is composed of two identical layers. And each layer consists of two sub-layers: multi-head attention mechanism and fully connected position-wise feed-forward network (FFN). In addition, residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) are performed in the two sub-layers for the sake of vanishing gradient problem in multi-layer processing and high offset of vectors in residual connection. For the k -th layer, the process can be formulated as: H kc = LayerNorm ( E k  1 c + Att ( E k  1 c , E k  1 c , E k  1 c )) , E kc = LayerNorm ( H kc + FFN ( H kc )) , (2) where E k  1 c  R l c  d is the output vectors from the ( k  1 )-th layer ; LayerNorm denotes layer normalization; and Att means the multi-head attention (Vaswani et al., 2017) that takes query, key, and value vectors as inputs. AST Encoder. Considering that AST is a kind of graph, it can be learned by GNNs. Since GraphSAGE (Hamilton et al., 2017) shows high efficiency and performance dealing with graphs, we introduce the idea of GraphSAGE and improve it by adding residual connection for AST encoding, as shown in Figure 2. The encoding layer processes the AST by firstly aggregating the neighbors of the nodes with edge information and then updating the nodes with their aggregated neighborhood information. For a node i and its neighbors in the k -th layer, the process can be formulated as follows: h ki = W 1  e k  1 i + W 2  Aggr ( { e k  1 j ,  j  N ( i ) } ) , (3) where e k  1 i  R d means the vector representation of i -th node from the ( k  1 )-th layer; N ( i ) is the neighbors of the node i ; e k  1 j  R d denotes the j -th neighbor vector for node i ; W 1 , W 2  R d  d are learnable weight matrices; Aggr represents aggregation function. With the increase of the number of layers, a node aggregates the neighborhood information from a deeper depth. In order to achieve strong capability of aggregation, the AST encoder is composed of six layers. And to mitigate gradient vanishing and high offset caused by multi-layer processing, we adopt residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) in each layer for improvement, which is formulated as follows: E kn = LayerNorm ( H kn + E k  1 n ) . Note that, E k  1 n  R l n  d in this formula denotes the output vectors of nodes from the ( k  1 )-th layer. The decoder of CODESCRIBE is designed with six stacks of modified Transformer decoding blocks. Given the existing summary tokens, the k -th decoding block firstly encodes them by masked multihead attention with residual connection and layer normalization, which is formalized as: H ks = LayerNorm ( E k  1 s + MaskAtt ( E k  1 s , E k  1 s , E k  1 s )) , (6) where E k  1 s  R l s  d is the output vectors from the ( k  1 )-th layer and MaskAtt denotes the masked multi-head attention (Vaswani et al., 2017). After that, we expand the Transformer block by leveraging two multi-head attention modules to interact with the two encoders for summary decoding. One multi-head attention module is performed over the AST features to get the first-stage decoded information, which will then be fed into the other over the learned source code for the second-stage decoding. Then the decoded summary vectors are put into FFN for non-linear transformation. The process can be formalized as follows: H ks,n = LayerNorm ( H ks + Att ( H ks , E (cid:48) n , E (cid:48) n )) , H ks,c = LayerNorm ( H ks,n + Att ( H ks,n , E (cid:48) c , E (cid:48) c )) , E ks = LayerNorm ( H ks,c + FFN ( H ks,c )) , (7) where E (cid:48) n and E (cid:48) c are the learned features of AST nodes and code tokens, respectively. We present a multi-source pointer-generator network (MPG) on top of the decoder and encoders to yield the final probability of the next summary token. Considering that tokens such as function names and variable names appear both in code and summary text (Ahmad et al., 2020), MPG is designed to allow CODESCRIBE to generate summary tokens both from the summary vocabulary and from the AST and source code. Taking the m -th output token as an example, three probability distributions p v , p c , and p n will be calculated from decoded summary, code, and AST and determine the probabilities for the token. To get the first probability distribution p v , a Linear sub-layer with Softmax is applied over the decoded summary token vector e (cid:48) s  R d , as follows: p v = Softmax ( Linear ( e (cid:48) s )) . For a token w , p v ( w ) = 0 if w is an out-of-vocabulary word to the summary vocabulary. As for the distributions p c and p n , we only describe p c since the two have the similar calculation process. In detail, our model applies an additional multi-head attention layer stacked on the last code encoding block and summary decoding block. It takes the decoded summary token vector e (cid:48) s  R d as query and the encoded code information E (cid:48) c  R l c  d as key and value:  c = Att ( e (cid:48) s , E (cid:48) c , E (cid:48) c ) ,  c = Softmax ( Mean ( a 1 , a 2 , . . . , a i , . . . )) , a i = Softmax (cid:32) e (cid:48) s W Qi ( E (cid:48) c W Ki ) T  d (cid:33) ( E (cid:48) c W Vi ) , (9) where W Qi , W Ki , and W Vi are learnable parameters. The context vector  c  R d will be used for the final distribution. Through the function Mean and Softmax , the attention vectors ( a 1 , a 2 , . . . , a i , . . . ) of all heads are averaged as  c  R l c . For the token w , its probability p c ( w ) is formulated as follows: p c ( w ) = (cid:80) i : w i = w  ci , (10) where w i means the i -th token in the source code. Similarly, we can get  n and p n corresponding to the AST. After that, the final probability p s ( w ) of the token w is defined as a mixture of the three probabilities: p s ( w ) =  v  p v ( w ) +  c  p c ( w ) +  n  p n ( w ) , [  v ,  c ,  n ] = Softmax ( Linear ([ e (cid:48) s ,  c ,  n ])) , (11) where  v ,  c , and  n are the weight values for p v ( w ) , p c ( w ) , and p n ( w ) . The higher the probability p s ( w ) is, the more likely the token w is considered as the next summary token. We conduct experiments to answer the following research questions: (1) How effective is CODESCRIBE compared with the state-of-the-art baselines? (2) How effective is the structure design of CODESCRIBE ? (3) What is the impact of model size on the performance of CODESCRIBE ? We also perform a qualitative analysis of two detailed examples. The experiments are conducted based on two benchmarks: (1) Java dataset (Hu et al., 2018b) and (2) Python dataset (Wan et al., 2018). The two datasets are split into train/valid/test sets with 69,708/8,714/8,714 and 55,538/18,505/18,502 , respectively. In the experiments, we follow the divisions for the fairness of the results. In the data preprocessing, NLTK package (Bird, 2006) is utilized for the tokenization of source code and summary text. And we apply javalang 2 and ast 3 packages to parsing Java and Python code into ASTs. In addition, the tokens in forms of  Cammel-Case ,  snake_case , and  concatenatecase  are split into sub-tokens as  Cammel Case ,  snake case , and  concatenate case . We leverage PyTorch 1.9 for CODESCRIBE implementation. The model runs under the development environment of Python 3.9 with NVIDIA 2080 Ti GPUs and CUDA 10.2 supported. We follow the previous works (Ahmad et al., 2020; Choi et al., 2021) and set all the embedding sizes of code tokens, AST nodes, and summary tokens to 512, and the number of attention headers to 8. As described in Section 3, the numbers of layers of code encoder, AST encoder, and summary decoder are 2, 6, and 6, respectively. The model is trained with Adam optimizer (Kingma and Ba, 2015). We initialize the learning rate as 5 e  4 that will be decreased by 5% after each training epoch until to 2 . 5 e  5 . The dropout rate is set to 0 . 2 . We set the batch size to 96 and 160 for the Java and Python datasets, respectively. The training process will terminate after 100 epochs or stop early if the performance does not improve for 10 epochs. In addition, we leverage beam search (Koehn, 2004) during the model inference and set the beam width to 5. We introduce eight state-of-the-art works as baselines for comparison, including six RNN-based models and two Transformer-based models. RNN-based Models. Among these baselines, CODE-NN (Iyer et al., 2016), API+CODE (Hu et al., 2018b), and Dual Model (Wei et al., 2019) learn source code for summarization. Tree2Seq (Eriguchi et al., 2016) and DeepCom (Hu et al., 2018a) generate summaries from AST features. RL+Hybrid2Seq (Wan et al., 2018) combines source code and AST based on LSTM. baselines include CopyTrans (Ahmad et al., 2020) and mAST+GCN (Choi et al., 2021), both of which leverage Transformer for code summary generation. The main difference is that CopyTrans learns sequential source code, and mAST+GCN is built based on AST. For the model evaluation, three metrics are introduced: BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004). All the scores are presented in percentage. We first evaluate the performance of CODESCRIBE by comparing it with eight state-of-the-art baselines. The results of baselines are all from Choi et al. (2021) and are shown in Table 1. The overall results in Table 1 illustrate that the recent Transformer-based models (Ahmad et al., 2020; Choi et al., 2021) are superior to the previous works based on RNNs (Iyer et al., 2016; Eriguchi et al., 2016; Wan et al., 2018; Hu et al., 2018a,b; Wei et al., 2019). Although the two models CopyTrans and mAST+GCN have high performance in code summarization, our approach CODESCRIBE performs much better than them both on the two datasets. Intuitively, CODESCRIBE improves the performance (i.e., BLEU/METEOR/ROUGE-L) by 4.46/5.84/4.83% on the Java dataset and 2.59/3.71/3.73% on the Python dataset compared to CopyTrans. In comparison with mAST+GCN, the performance of CODESCRIBE improves by 3.70/5.10/4.77% on the Java dataset and 2.29/3.36/3.65% on the Python dataset. RNN-based models in code summarization task; (2) AST information contributes significantly to code comprehension; and (3) by incorporating both AST and source code into CODESCRIBE based on GraphSAGE and Transformer, the performance can be greatly improved due to its more comprehensive learning capacity for code and better decoding for summary generation. This section validates the effectiveness of CODESCRIBE 's structure to by performing an ablation study on the Java dataset. We firstly design five models for comparison that remove one of important components in CODESCRIBE including: (1) the AST encoder (R-AST), (2) the source code encoder (R-Code), (3) the triplet positions (R-ASTPos), (4) the MPG (R-Copy), and (5) the residual connection in the AST encoder (R-ASTRes). We further investigate the rationality of CODESCRIBE 's structure by comparison with five variants: (1) V-Copy that replaces MPG with the copying mechanism (See et al., 2017) used in Ahmad et al. (2020), (2) V-GCN that replaces GraphSAGE with GCN (Kipf and Welling, 2017), (3) V-GAT that replaces GraphSAGE with GAT (Kipf and Welling, 2017), (4) V-Emb that replaces the shared embedding layer for code tokens and AST nodes with two independent embedding layers, and (5) V-Dec that reverses the decoding order for the source code and AST features. As shown in Table 2, the performance of CODESCRIBE is affected if the components are removed. The results of R-AST and R-Code show that the two encoders are the most significant learning components to CODESCRIBE . Moreover, the AST encoder is more important than the code encoder as R-Code performs better than R-AST. The performances of R-ASTPos and R-Copy indicate that the triplet positions for nodes and copying mechanism (MPG) we proposed are effective for CODESCRIBE in code summarization. In addition, we find that R-ASTRes suffers from under-fitting on the Java dataset, which indicates that the residual connection in AST encoder has a powerful influence on CODESCRIBE . As illustrated in Table 2, CODESCRIBE improves the performance by 0.26/0.22/0.30% on the Java dataset compared with V-Copy. It indicates that our proposed MPG is more effective than the copying mechanism in Ahmad et al. (2020). As for the GNN module in AST encoding, it can be observed that CODESCRIBE still has the higher performance than V-GCN and V-GAT. This demonstrates the superiority of GrahpSAGE for the architecture of CODESCRIBE compared to GCN and GAT. Compared with V-Emb, it shows that the shared embedding layer works better than two separated embedding layers for AST and source code. The result of V-Dec turns out that the performance will not be affected sinificantly if the order of decoding over AST and code features is reversed. The results on the Python dataset are presented in Table 7 in Appendix A. 4.6 Study on the Model Size (RQ3) This section studies the performance of CODESCRIBE with the change of model size 4 on the Java dataset. To that end, we modify the number of layers of the encoders and the decoder respectively for performance observation and comparison. Table 3 presents the performance of CODESCRIBE when the number of AST encoding layers 4 This work considers the number of trainable parameters in the encoders and decoder of CODESCRIBE as the model size to facilitate observation. varies from 2 to 12. The results show that the performance improves as the number of AST encoding layers increases from 2 to 6. With the increase of the number from 6 to 12, the performance does not improve any more and is even impacted slightly. As illustrated in Table 4, CODESCRIBE has the best performance with 2 code encoding layers. With the number of code layers growing from 4 to 12, there is a trend of gradual decrease of the performance. For the model size concerned with summary decoding layers, as shown in Table 5, the performance is getting better when the number of layers ranges from 2 to 6, and can not be improved as the number continues to increase. The overall results show that it the performance of CODESCRIBE will not be improved if the encoders and the decoder become too deep (i.e. with more layers), especially for the source code encoder. More experimental results are provided in Table 8 11 in Appendix B. 4.7 Case Study Table 6 shows the qualitative examples of R-AST, R-Copy, V-GCN, V-Dec, and CODESCRIBE on the two datasets. From the table, it can be observed that CODESCRIBE with the whole architecture generates better code summaries compared with the four variants. In the case on the Java dataset, only R-Copy and CODESCRIBE get the right intent of the code. The other variants miss out the key word  history . In the case on the Python dataset, CODE Code Model BLEU METEOR ROUGE-L Layers Size (  10 6 ) 2 40.99 49.19 32.27 59.59 4 47.30 48.80 32.15 59.32 6 53.60 48.92 32.10 59.30 8 59.91 48.73 31.95 58.95 10 66.21 49.11 31.97 59.09 12 72.52 48.36 31.59 58.59 Table 4: Performance of CODESCRIBE with different numbers of code encoding layers on the Java dataset. SCRIBE generates the most accurate summary compared to the other variants. In contrast, although the four variants output the first half of the summary (i.e.,  create an image ), the rest information  from the value dictionary .  can not be generated correctly. More qualitative examples are referred to Table 12 and 13 in Appendix C. 5 Related Work With the development of deep learning, most works have considered code summarization as a sequence generation task. In many of the recent approaches, source code snippets are modeled as plain texts based on RNNs (Iyer et al., 2016; Hu et al., 2018b; Wei et al., 2019; Ye et al., 2020). For example, Hu et al. (2018b) proposed an RNN-based model that learns API knowledge from a different but related task and incorporates the knowledge into code summarization. Wei et al. (2019) presented a dual learning framework based on LSTMs to train code generation and code summarization and improve the performances of both tasks. Ye et al. (2020) combined code summarization and code generation to train the code retrieval task via multi-task learning, which achieved competitive performance for the code summarization task. Most recently, Ahmad et al. (2020) applied Transformer to encoding the source code sequence to improve the summarization performance. Since considering source code as plain text ignores the structural information in code, recent works have explored the AST of code and modeled the tree-based structure for code summarization. Typically, Hu et al. (2018a) proposed a structure-based traversal (SBT) method to traverse ASTs into node sequences and used a sequence-to-sequence model based on LSTMs to generate code comments. Alon et al. (2019) represented a code snippet as a set of compositional paths in its AST and used LSTMs to encode these paths. Shido et al. (2019) extended Tree-LSTM (Tai et al., 2015) to Multi-way Tree-LSTM to learn the representation of AST for code summary generation. Liu et al. (2020) built code property graph (CPG) (Ya-maguchi et al., 2014) based on AST and combined retrieval method and GNNs for describing C programming language. The latest work (Choi et al., 2021) performed graph convolutional networks (GCNs) (Kipf and Welling, 2017) before Transformer framework to learn AST representation for summary generation. To represent the code comprehensively, more and more works have paid attention to both the source code and the AST for code summarization. For example, Hu et al. (2020) integrated both AST node sequence and source code into a hybrid learning framework based on GRUs. Wei et al. (2020) and Zhang et al. (2020) both utilized the information retrieval techniques to improve the quality of code summaries that are generated from the code snippets and ASTs. Wan et al. (2018) incorporated AST as well as sequential content of code snippet into a deep reinforcement learning framework based on LSTM and AST-based LSTM. LeClair et al. (2020) proposed a graph-based neural architecture for code summarization, which uses GRUs and GCN to encode AST and GRUs to learn source code sequence. Wang et al. (2022) presented the first hierarchical-attention based learning approach for code summarization by integrating source code, type-augmented AST, and control-flow graphs. Recently, several pre-trained models, e.g., CodeBERT (Feng et al., 2020), CodeT5 (Wang et al., 2021), PLBART (Ahmad et al., 2021) and Co-TexT (Phan et al., 2021), have been proposed to better represent the source code, and verified on code summarization. For example, CodeBERT (Feng et al., 2020) is a pre-trained model based on ELEC-TRA (Clark et al., 2020), which has achieved promising performance on downstream tasks including code summarization. CodeT5 (Wang et al., 2021) considers the token type information in code and builds on the T5 architecture (Raffel et al., 2020) that utilizes denoising sequence-to-sequence pre-training. PLBART (Ahmad et al., 2021) is another start-of-the-art pre-trained model on an extensive collection of Python and Java functions, as well as their natural language summaries via denoising auto-encoding. Note that, our work is aim to introduce an encoder network with a novel triplet position to better represent the hierarchical structure of programs, rather than pre-training a language model for source code. We think that our introduced encoder can be easily incorporated into the pre-training models through masking and predicting code tokens or code graphs. We leave the comparison between our model and those mentioned pre-trained code models to future work. This paper has presented CODESCRIBE , an encoder-decoder-based neural network for source code summarization. CODESCRIBE designs a triplet position to model the hierarchical syntax structure of code, which is then incorporated into the Transformer and GNN based framework for better representation of lexical and syntax information of code, respectively. The performance of CODESCRIBE is further enhanced by the introduced multi-source pointer generator in decoding. Experiments on two benchmarks reveal that the summaries generated by CODESCRIBE are of higher quality when compared with several recent state-of-the-art works. This work is supported by the National Natural Science Foundation of China under Grants 61972290. Yao Wan is partially supported by the National Natural Science Foundation of China under Grant No. 62102157. We would like to thank all the anonymous reviewers for their constructive comments on improving this paper.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain method objective objective objective abstain objective objective method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method method abstain other abstain abstain abstain abstain abstain other abstain abstain method abstain other abstain abstain method abstain abstain other method abstain abstain other method abstain abstain abstain abstain method method other abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other other other