Neural Code Comprehension: A Learnable Representation of Code Semantics
With the recent success of embeddings in natural language processing, research has been conducted into applying similar methods to code analysis. Most works attempt to process the code directly or use a syntactic tree representation, treating it like sentences written in a natural language. However, none of the existing methods are sufficient to comprehend program semantics robustly, due to structural features such as function calls, branching, and interchangeable order of statements. In this paper, we propose a novel processing technique to learn code semantics, and apply it to a variety of program analysis tasks. In particular, we stipulate that a robust distributional hypothesis of code applies to both human-and machine-generated programs. Following this hypothesis, we define an embedding space, inst2vec, based on an Intermediate Representation (IR) of the code that is independent of the source programming language. We provide a novel definition of contextual flow for this IR, leveraging both the underlying data-and control-flow of the program. We then analyze the embeddings qualitatively using analogies and clustering, and evaluate the learned representation on three different high-level tasks. We show that even without fine-tuning, a single RNN architecture and fixed inst2vec embeddings outperform specialized approaches for performance prediction (compute device mapping, optimal thread coarsening); and algorithm classification from raw code (104 classes), where we set a new state-of-the-art.
This paper proposes to use programming code embeddings for individual statements to be used in all kinds of program analysis tasks, such as classification of type of algorithms, placing of code on a heterogeneous cpu-gpu architecture and scheduling of programs on gpus.  The novelty of the paper consists of the way the code embeddings are computed/trained. Instead of looking at code statements in high-level languages and following either data or control flow, the paper proposes to use statements at an intermediate-representation level (which are independent of the high-level language used) and take both data and control flow into account. As such, the proposed technique builds "contextual flow graphs" where the nodes are connected either through data or control flow edges. The nodes are variable or label identifiers.  The statements embeddings are computed using the skip-gram model on paths of the contextual flow graphs.  The embeddings are trained on a sizeable dataset that includes the code of tensorflow, gpu benchmarks, linux kernel and synthetic code.  The trained embeddings are used for three different predictive tasks: classification of algorithm type, placing code on a cpu or a gpu, and optimizations for gpu scheduling. In general, the embeddings provide competitive results when compared to either hand-tuned versions or state of the art baselines.  I think this paper is a nice bridge from traditional program analysis and machine learning. For a researcher with experience in program analysis the contextual graphs are somewhat intuitive. The authors choose to represent variables as nodes instead of whole sentences. It would be nice to provide some intuition why that performs better or is a better choice than having nodes as statements (while keeping the edges to represent both data and control flow).  I would have liked to see some simple experiments with code similarity. One idea for such an experiment: take two implementations for the same algorithm and try to show that in terms of code embeddings they appear similar (it's common to find the implementation of the same neural network in two different frameworks, such as tensorflow and pytorch).  The authors could include a discussion on the limitation of the analysis. In particular, a discussion about how library code is treated. For the suggestion above with two different frameworks, would the embeddings work? Or would the information be lost due to library calls and the reliance of frameworks? For which type of tasks are code embeddings expected to work and for which ones they don't provide sufficient information?  I'm not sure "semantics" is the right term to use for characterizing similarity wrt to consuming/producing similar resources. In general, "semantics" makes me think of a deeper meaning (potentially expressed at a higher granularity than one IR statement).