Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pretrained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5% improvement in test loss, 4% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.
THE EFFECTIVENESS OF PRE-TRAINED CODE EMBEDDINGS
Summary:
This work shows how pre-training word vectors using corpuses of code leads to representations that are more suitable than randomly initialized (and trained) representations for function/method name prediction (here called extreme summarization). This paper applies a standard language model to several collected corpuses of code written in different programming languages in order to pretrain the embeddings. It then uses a standard model for the extreme summarization task that takes pre-trained language model embeddings. This leads to speedups in training, improvements in validation loss, and less overfitting. I worry that these results didn't really need much proving given that we have already seen the exact same methods work with natural languages. It would actually be more surprising if they didn't work for programming languages, which suggests that the real question is whether code embeddings are actually more effective than natural language embeddings for this problem given that the authors show syntax of the code is far less important than the semantics of the words in the vocabulary. It is not clear that embeddings pre-trained with much more data on natural languages wouldn't work just as well.
Pros:
This paper takes the time to clearly explain the effectiveness of pre-training words vectors in a new setting. It is easy to follow and understand thanks to the clear organization and exposition.
Speedup and validation loss improvements are demonstrated for a variety of programming languages despite the final evaluation being only in Java, which is quite surprising. 
The authors discover that overlap in actual programming language syntax is less important than overlap in semantic representation. 
Because the models are out-of-the-box, it is easy to focus on the actual contributions of this paper related to pre-training.
Does seem to clearly demonstrate that pre-trained embeddings of some form should be used for this extreme summarization task.
Cons:
It is not clear how big the collected corpus is. This is important because the work that this paper cites on pre-trained embeddings (Mikolov et al 2013, Pennington et al 2014, McCann et al 2017, Peters et al 2018) typically use fairly large datasets for pre-training. All of these results might be watered down by insufficient pre-training data for the language model when in fact the results could be much stronger with more data. It would be nice to show the effects of pre-training dataset size as is done in the aforementioned previous works. Without this comparison, it is hard to tell whether the paper sufficiently explores this idea.
The models are both standard, out-of-the-box models. There is no novelty on the modeling side of this paper.
The pre-training methods are also not novel. They are methods that have already been shown to work applied in a slightly different setting.
It is not clear that the setting is actually different enough to require this pre-training. Comparing to randomly initialized embeddings is fine, but I would also like to see a comparison to other pre-trained embeddings like GloVe, GloVe+CoVe, or ELMo (Pennington et al 2014, McCann et al 2017, Peters et al 2018). Since the authors find that it is the semantics of the words that matter more than the syntax of any particular programming language, then perhaps it would actually be better to use pre-trained embeddings that tap into much larger amounts of data. At the very least, it seems it would make sense to perhaps supplement a standard pre-trained embedding with those suggested by the authors since so many of the words in the code must be English words. If this is too farfetch'd, then I would suggest that the authors provide some statistics showing why GloVe, GloVe+CoVe, and ELMo are not appropriate starting points for comparison, but the overlap from the pre-training corpuses is already so low that it seems supplementing with standard pre-trained embeddings should only help.
The evaluation dataset detailed in Allamanis et al 2016 uses two metrics: an F1 metric and an exact match metric. This paper only compares on validation loss. What's more, it reports everything in relative terms so that the raw improvement is masked until Figure 1 makes it somewhat possible to deduce. The problem here is that we don't know how a 0.0-0.5 raw improvement in validation loss translates to the metrics established for the dataset by Allamanis et al 2016. If those are no longer the standard metrics, the authors should explain how validation loss came to supplant the original metrics proposed by Allamanis et al 2016.
What's more, there is no context for how well models typically do on this evaluation task. Without any comparisons it is impossible to tell whether any of the experiments are using models in a reasonable realm of performance on this task.
Overall:
All these effects have already been shown for pre-trained embeddings in the past, and the experiments involve running standard methods on newly collected datasets. This means there is no novelty in the pre-training method or the extreme summarization method. Little is known about the newly collected datasets, it is not clear how to interpret the relative improvements in validation loss compared to the original metrics of Allamanis et al 2016, and the paper lacks necessary comparisons to othef pre-trained embeddings, so though the overall claim that pre-trained embeddings should be used for this task seems to hold up, it is not clear that this is a complete argument for the method chosen by the authors.