Program comprehension is a fundamental task in software development and maintenance processes. Software developers often need to understand a large amount of existing code before they can develop new features or fix bugs in existing programs. Being able to process programming language code automatically and provide summaries of code functionality accurately can significantly help developers to reduce time spent in code navigation and understanding, and thus increase productivity. Different from natural language articles, source code in programming languages often follows rigid syntactical structures and there can exist dependencies among code elements that are located far away from each other through complex control flows and data flows. Existing studies on tree-based convolutional neural networks (TBCNN) and gated graph neural networks (GGNN) are not able to capture essential semantic dependencies among code elements accurately. In this paper, we propose novel tree-based capsule networks (TreeCaps) and relevant techniques for processing program code in an automated way that encodes code syntactical structures and captures code dependencies more accurately. Based on evaluation on programs written in different programming languages, we show that our TreeCaps-based approach can outperform other approaches in classifying the functionalities of many programs.
The paper proposes a neural architecture for summarizing trees inspired by capsule networks from computer vision. The authors re-use a tree convolution from previous work for the bottommost layer, and then propose adaptations to the dynamic routing from capsule networks so that it can be applied to variable-sized trees. The paper applies the proposed architecture to three different program classification datasets, which are in three different languages. The paper reports empirical gains compared to two architectures proposed by previous work.
I think that it's interesting to apply the capsule network architecture to tree classification, but unfortunately it doesn't appear that some of the motivation for capsule networks on images didn't seem to transfer neatly to this setting; for example, there is no equivalent of inverse graphics as there is no reconstruction loss (as pointed out by the authors in Section 6.4).
Also, the variable-to-static capsule routing indeed appears novel, but I was a bit confused by its internal details. It appears that the outputs of the previous layer which occur most often will get routed (considering lines 6-8 of Algorithm 1 which up-weights each of the $\hat{u}_i$ based on its similarity to $v_j$; the $v_j$ are initially a re-numbered subset of $\hat{u}_i$), without any prior transformation of the previous layer first. It seems to me that this doesn't allow for the prior layer to predict more complex features about the input that the subsequent layer is expected to capture. In fact, for certain code classification tasks, it may be that rare capsule outputs from the initial layer are the most important to preserve.
My biggest concern has to do with the empirical results. The source of Dataset C (Mou et al 2016, https://arxiv.org/pdf/1409.5718.pdf) reports 94.0% accuracy in Table 3 on their TBCNN method on the same dataset, whereas this paper reports 79.40% accuracy for TBCNN. I understand that the later result comes from a reimplementation, but it seems fairer to compare against (or additionally report) the results from the original authors of the method.
Also, the paper cites ASTNN (Zhang et al 2019, https://dl.acm.org/citation.cfm?id=3339604) in the introduction, and even though that paper reports (in table 2) 98.2% accuracy on Dataset C, the results table of the paper under review does not mention this in the evaluation section. I don't think that a paper necessarily has to achieve empirical results beating all previous ones in order to merit acceptance, but the way that the comparison is currently set up doesn't seem to facilitate a clear comparison of the pros and cons of this method versus other ones in the literature.
For the above reasons, I vote to reject the paper. For future submissions, it would be good to see a more comprehensive empirical comparison of the proposed method compared to others, and also to have more explanations about the design of the network.