CODE2SEQ: GENERATING SEQUENCES FROM STRUCTURED REPRESENTATIONS OF CODE
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present CODE2SEQ: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding. We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to 16M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as state-of-the-art NMT models. An online demo of our model is available at http://code2seq.org. Our code, data and trained models are available at
This paper introduces an AST-based encoding for programming code and
shows the effectivness of the encoding in two different task of code
summarization:
1. Extreme code summarization - predicting (generating) function name from function body (Java)
2. Code captioning - generating a natural language sentence for a (short) snippet of code (C#)
Pros:
- Simple idea of encoding syntactic structure of the program through random paths in ASTs
- Thorough evaluation of the technique on multiple datasets and using multiple baselines
- Better results than previously published baselines
- Two new datasets (based on Java code present in github) that will be made available
- The encoding is used in two different tasks which also involve two different languages
Cons:
- Some of the details of the implementation/design are not clear (see some clarifying questions below)
- More stats on the collected datasets would have been nice
- Personally, I'm not convinced "extreme code summarization"
is a great task for code understanding (see more comments below)
Overall, I enjoyed reading this paper and I think the authors did a
great job explaining the technique, comparing it with other baselines,
building new datasets, etc.
I have several clarifying questions/points (in no particular order):
* Can you provide some intuition on why random paths in the AST encode
  the "meaning" of the code? And perhaps qualitatively compare it with
  recording some other properties from the tree that preserve its
  structure more?
* When you perform the encoding of the function body, one sample in a
  training step contains all the k (k = 200) paths and all the 2*k
  terminals (end of Section 2)? Or one path at a time (Section 3.2)?
  I'm guessing is the latter, but not entirely sure. Figure 3 could
  improve to make it clear.
* Can you explain how you came up with k = 200? I think providing some
  stats on the dataset could be helpful to understand this number.
* The results for the baselines - do you train across all projects?
  (As you point out, ConvAttention trained separately, curious whether
  it makes a difference for the 2 datasets med and large not present
  in the original paper).
* I'm not sure I understand parts of the ablation study. In particular
  for point 1., it seems that instead of the AST, only the terminal
  nodes are used. Do you still use 200 random pairs of terminal? Is
  this equivalent to a (randomly shuffled) subset of the tokens in the
  program? Also could you explain why you do the ablation study on the
  validation set of the medium dataset? In fact, the caption of Table
  3 says it's done on the dev set. This part was a bit confusing.
* I would have liked to see more details on the datasets introduced,
  in particular wrt metrics that are relevant for training the model
  you describe (e.g., stats on the ASTs, stats on the number of random
  paths in ASTs, code length in tokens, etc.)
* I'm not convinced that the task of "extreme code summarization" is a
  meaningful task. My main problem with it is that the performance of
  a human on this task would not be that great. On one hand humans
  (I'm referring to "programming humans" :) ) have no problem in
  coming up with a name for a function body; however, I'm not
  convinced they could predict the "gold" standard. Or, another way of
  thinking about this, if you have 3 humans who provided names for the
  same function, my guess it that there will be a wide degree of
  (dis)agreement. Some of the examples provided in the supplementary
  material can serve as confirmation bias to my thought :): Fig 7. I
  claim "choose random prime" and "generate prime number" are
  semantically close, however, the precision and recall for this
  example are both low. All this being said, I understand that it's a
  task for which data can be generated fairly quickly to feed the
  (beast) NN and that helps pushing the needle in understanding code
  semantics.
* It would be nice to see "exact match" as one of the metrics (it is
  probably low, judging by F1 scores, but good to be reported).
* Most likely the following paper could be cited in the related work:
Neural Code Comprehension: A Learnable Representation of Code Semantics
https://arxiv.org/abs/1806.07336
https://nips.cc/Conferences/2018/Schedule?showEvent=11359
Page 5 first phrase at the top, perhaps zi is a typo and it is
supposed to be z1?
----
Update: after all the discussion, I'm lowering my score a bit while still hoping the paper will get published. I'm satisfied with the results and the improvement of the paper. I still find it a bit surprising that the pairs of literals/leaves in the tree are a good approximation for the program itself (as shown in one of the ablation study).