The Effectiveness of Pre-Trained Code Embeddings
Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pre-trained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5\% improvement in test loss, 4\% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.
 This paper sets to understand whether pretraining word embeddings for programming language code by using NLP-like language models has an impact on extreme code summarization task (i.e., generate/predict the name of a function based on its body). I think the paper asks some important questions, however the execution of the research and the results presented are not convincing. I think the area is relevant and the research questions are worth pursuing; however the work as it is presented in the paper needs improvement to be accepted for publication. Pros: * The study of language models for programming language code * Pretraining is performed for 3 different languages (C, Java, Python) - target task is in Python Cons: * Strange claims of speedup and performance improvement * Inconclusive results Some suggestions for improvement: * The section on language models pretraining is very sparse, more details are needed. * The claims of speedup and improvement are strange. Speedup refers to the training speed, I suppose. The performance of the downstream task is never discussed. Only the validation loss is shown and all the performance "improvement" is discussed on these graphs, which I found strange. Also, the graphs have their y-axes starting at non-zero values. I personally prefer graphs that start at zero and if there is a need to "zoom-in" find a way to "zoom-in" to the part of the graph that is important. * In general the paper writing and reporting on the experiments sounds ad-hoc and not well thought-out. * I don*t agree with many of the explanations in the paper. For example (page 6), it*s not true that the extreme summarization task does not require much of the syntactic information (there are submission at the current ICLR*19 that show exactly the opposite, encoding based on syntactic information is useful). The model studied in the paper does NOT use any syntactic information, it treats the code like a sequence of tokens. * The last question in Section 6 is not a Yes/No question, the answer is phrased as a Yes/No question. I encourage the authors to pursue the research questions, however in a more systematic and with better methodology.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Effectiveness of Pre-Trained Code Embeddings
Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pre-trained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5\% improvement in test loss, 4\% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper sets to understand whether pretraining word embeddings for programming language code by using NLP-like language models has an impact on extreme code summarization task (i.e., generate/predict the name of a function based on its body). I think the paper asks some important questions, however the execution of the research and the results presented are not convincing. I think the area is relevant and the research questions are worth pursuing; however the work as it is presented in the paper needs improvement to be accepted for publication. Pros: * The study of language models for programming language code * Pretraining is performed for 3 different languages (C, Java, Python) - target task is in Python Cons: * Strange claims of speedup and performance improvement * Inconclusive results Some suggestions for improvement: * The section on language models pretraining is very sparse, more details are needed. * The claims of speedup and improvement are strange. Speedup refers to the training speed, I suppose. The performance of the downstream task is never discussed. Only the validation loss is shown and all the performance "improvement" is discussed on these graphs, which I found strange. Also, the graphs have their y-axes starting at non-zero values. I personally prefer graphs that start at zero and if there is a need to "zoom-in" find a way to "zoom-in" to the part of the graph that is important. * In general the paper writing and reporting on the experiments sounds ad-hoc and not well thought-out. * I don*t agree with many of the explanations in the paper. For example (page 6), it*s not true that the extreme summarization task does not require much of the syntactic information (there are submission at the current ICLR*19 that show exactly the opposite, encoding based on syntactic information is useful). The model studied in the paper does NOT use any syntactic information, it treats the code like a sequence of tokens. * The last question in Section 6 is not a Yes/No question, the answer is phrased as a Yes/No question. I encourage the authors to pursue the research questions, however in a more systematic and with better methodology.
assistant
 This paper sets to understand whether pretraining word embeddings for programming language code by using NLP-like language models has an impact on extreme code summarization task (i.e., generate/predict the name of a function based on its body). I think the paper asks some important questions, however the execution of the research and the results presented are not convincing. I think the area is relevant and the research questions are worth pursuing; however the work as it is presented in the paper needs improvement to be accepted for publication. Pros: * The study of language models for programming language code * Pretraining is performed for 3 different languages (C, Java, Python) - target task is in Python Cons: * Strange claims of speedup and performance improvement * Inconclusive results Some suggestions for improvement: * The section on language models pretraining is very sparse, more details are needed. * The claims of speedup and improvement are strange. Speedup refers to the training speed, I suppose. The performance of the downstream task is never discussed. Only the validation loss is shown and all the performance "improvement" is discussed on these graphs, which I found strange. Also, the graphs have their y-axes starting at non-zero values. I personally prefer graphs that start at zero and if there is a need to "zoom-in" find a way to "zoom-in" to the part of the graph that is important. * In general the paper writing and reporting on the experiments sounds ad-hoc and not well thought-out. * I don*t agree with many of the explanations in the paper. For example (page 6), it*s not true that the extreme summarization task does not require much of the syntactic information (there are submission at the current ICLR*19 that show exactly the opposite, encoding based on syntactic information is useful). The model studied in the paper does NOT use any syntactic information, it treats the code like a sequence of tokens. * The last question in Section 6 is not a Yes/No question, the answer is phrased as a Yes/No question. I encourage the authors to pursue the research questions, however in a more systematic and with better methodology.