Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is the large number of out-of-vocabulary words. In this paper we proposed a method which combines the general pre-trained word embedding vectors with those generated on the taskspecific training set to address this issue. We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.
The main contributions in this paper are:
1) New variants of a recent LSTM-based model ("ESIM") are applied to the task of response-selection in dialogue modeling -- ESIM was originally introduced and evaluated for natural language inference. In this new setting, the ESIM model (vanilla and extended) outperform previous models when trained and evaluated on two distinct conversational datasets.
2) A fairly trivial method is proposed to extend the coverage of pre-trained word embeddings to deal with the OOV problem that arises when applying them to these conversational datasets.
The method itself is to combine d1-dimensional word embeddings that were pretrained on a large unannotated corpus (vocabulary S) with distinct d2-dimensional word embeddings that are trained on the task-specific training data (vocabulary T). The enhanced (d1+d2)-dimensional representation for a word is constructed by concatenating its vectors from the two embeddings, setting either the d1- or d2-dimensional subvector to zeros when the word is absent from either S or T, respectively. This method is incorporated as an extension into ESIM and evaluated on the two conversation datasets.
The main results can be characterized as showing that this vocabulary extension method leads to performance gains on two datasets, on top of an ESIM-model extended with character-based word embeddings, which itself outperforms the vanilla ESIM model.
These empirical results are potentially meaningful and could justify reporting, but the paper's organization is very confusing, and too many details are too unclear, leading to low confidence in reproducibility. 
There is basic novelty in applying the base model to a new task, and the analysis of the role of the special conversational boundary tokens is interesting and can help to inform future modeling choices. The embedding-enhancing method has low originality but is effective on this particular combination of model architecture, task and datasets. I am left wondering how well it might generalize to other models or tasks, since the problem it addresses shows up in many other places too...
Overall, the presentation switches back and forth between the Douban corpus and the Ubuntu corpus, and between word2vec and Glove embeddings, and this makes it very challenging to understand the details fully.
S3.1 - Word representation layer: This paragraph should probably mention that the character-composed embeddings are newly introduced here, and were not part of the original formulation of ESIM. That statement is currently hidden in the figure caption.
Algorithm 1:
- What set does P denote, and what is the set-theoretic relation between P and T?
- Under one possible interpretation, there may be items in P that are in neither T nor S, yet the algorithm does not define embeddings for those items even though its output is described as "a dictionary with word embeddings ... for P". This does not seem consistent? I think the sentence in S4.2 about initializing remaining OOV words as zeros is relevant and wonder if it should form part of the algorithm description?
S4.1 - What do the authors mean by the statement that response candidates for the Douban corpus were "collected by Lucene retrieval model"?
S4.2 - Paragraph two is very unclear. In particular, I don't understand the role of the Glove vectors here when Algorithm 1 is used, since the authors refer to word2vec vectors later in this paragraph and also in the Algorithm description.
S4.3 - It's insufficiently clear what the model definitions are for the Douban corpus. Is there still a character-based LSTM involved, or does FastText make it unnecessary?
S4.3 - "It can be seen from table 3 that the original ESIM did not perform well without character embedding." This is a curious way to describe the result, when, in fact, the ESIM model in table 3 already outperforms all the previous models listed.
S4.4 - gensim package -- for the benefit of readers unfamiliar with gensim, the text should ideally state explicitly that it is used to create the *word2vec* embeddings, instead of the ambiguous "word embeddings".