CAN AI GENERATE LOVE ADVICE?: TOWARD NEURAL ANSWER GENERATION FOR NON-FACTOID QUESTIONS
Deep learning methods that extract answers for non-factoid questions from QA sites are seen as critical since they can assist users in reaching their next decisions through conversations with AI systems. The current methods, however, have the following two problems: (1) They can not understand the ambiguous use of words in the questions as word usage can strongly depend on the context (e.g. the word “relationship” has quite different meanings in the categories of Love advice and other categories). As a result, the accuracies of their answer selections are not good enough. (2) The current methods can only select from among the answers held by QA sites and can not generate new ones. Thus, they can not answer the questions that are somewhat different with those stored in QA sites. Our solution, Neural Answer Construction Model, tackles these problems as it: (1) Incorporates the biases of semantics behind questions (e.g. categories assigned to questions) into word embeddings while also computing them regardless of the semantics. As a result, it can extract answers that suit the contexts of words used in the question as well as following the common usage of words across semantics. This improves the accuracy of answer selection. (2) Uses biLSTM to compute the embeddings of questions as well as those of the sentences often used to form answers (e.g. sentences representing conclusions or those supplementing the conclusions). It then simultaneously learns the optimum combination of those sentences as well as the closeness between the question and those sentences. As a result, our model can construct an answer that corresponds to the situation that underlies the question; it fills the gap between answer selection and generation and is the first model to move beyond the current simple answer selection model for non-factoid QAs. Evaluations using datasets created for love advice stored in the Japanese QA site, Oshiete goo, indicate that our model achieves 20 % higher accuracy in answer creation than the strong baselines. Our model is practical and has already been applied to the love advice service in Oshiete goo.
1 INTRODUCTION
Recently, dialog-based natural language understanding systems such as Apple’s Siri, IBM’s Watson, Amazon’s Echo, and Wolfram Alpha have spread through the market. In those systems, Question Answering (QA) modules are particularly important since people want to know many things in their daily lives. Technically, there are two types of questions in QA systems: factoid questions and non-factoid ones. The former are asking, for instance, for the name of a person or a location such that “What/Who is X?”. The latter are more diverse questions which cannot be answered by a short fact. They range from advice on making long distance relationships work well, to requests for opinions on some public issues. Significant progress has been made at answering factoid questions (Wang et al. (2007); Yu et al. (2014)), however, retrieving answers for non-factoid questions from the Web remains a critical challenge in improving QA modules. The QA community sites such as Yahoo! Answers and Quora can be sources of training data for the non-factoid questions where the goal is to automatically select the best of the stored candidate answers.
Recent deep learning methods have been applied to this non-factoid answer selection task using datasets stored in the QA sites resulting in state-of-the-art performance (Yu et al. (2014); Tan et al. (2015); Qiu & Huang (2015); Feng et al. (2015); Wang & Nyberg (2015); Tan et al. (2016)). They usually compute closeness between questions and answers by the individual embeddings obtained using a convolutional model. For example, Tan et al. (2016) builds the embeddings of questions and those of answers based on bidirectional long short-term memory (biLSTM) models, and measures their closeness by cosine similarity. It also utilizes an efficient attention mechanism to generate the answer representation following the question context. Their results show that their model can achieve much more accurate results than the strong baseline (Feng et al. (2015)). The current methods, however, have the following two problems when applying them to real applications:
(1) They can not understand the ambiguous use of words written in the questions as words are used in quite different ways following the context in which they appear (e.g. the word “relationship” used in a question submitted to “Love advice” category is quite different from the same word submitted to “Business advice” category). This makes words important for a specific context likely to be disregarded in the following answer selection process. As a result, the answer selection accuracies become weak for real applications.
(2) They can only select from among the answers stored in the QA systems and can not generate new ones. Thus, they can not answer the questions that are somewhat different from those stored in the QA systems even though it is important to cope with such differences when answering nonfactoid questions (e.g. questions in the “Love advice” category are often different due to the situation and user even though they share the same topics.). Furthermore, the answers selected from QA datasets often contain a large amount of unrelated information. Some other studies have tried to create short answers to the short questions often seen in chat systems (Vinyals & Le (2015); Serban et al. (2015)). Our target, non-factoid questions in QA systems, are, however, much longer and more complicated than those in chat systems. As described in their papers, the above methods, unfortunately, create unsatisfying answers to such non-factoid questions.
To solve the above problems, this paper proposes a neural answer construction model; it fills the gap between answer selection and generation and is the first model to move beyond the current simple answer selection model for non-factoid QAs. It extends the above mentioned biLSTM model since it is language independent and free from feature engineering, linguistic tools, or external resources. Our model takes the following two ideas:
(1) Before learning answer creation, it incorporates semantic biases behind questions (e.g. titles or categories assigned to questions) into word vectors while computing vectors by using QA documents stored across semantics. This process emphasizes the words that are important for a certain context. As a result, it can select the answers that suit the contexts of words used in the questions as well as the common usage of words seen across semantics. This improves the accuracies of answer selections.
For example, in Fig. 1-(a), there are two questions in category “Family” and “Love advice”. Words marked with rectangles are category specific (i.e. “son” and “homework” are specifically observed in “Family” while “distance”, “relationship”, and “lovers” are found in “Love advice”.) Our method can emphasize those words. As a result, answers that include the topics, “son” and “homework”, or topics, “distance”, “relationship”, and “lovers”, will be scored highly for the above questions in the following answer selection task.
(2) The QA module designer first defines the abstract scenario of answer to be created; types of sentences that should compose the answer and their occurrence order in the answer (e.g. typical answers in “Love advice” are composed in the order of the sentence types “sympathy”, “conclusion”, “supplementary for conclusion”, and “encouragement”). The sentence candidates can be extracted from the whole answers by applying sentence extraction methods or sentence type classifiers (Schmidt et al. (2014); Zhang et al. (2008); Nishikawa et al. (2010); Chen et al. (2010)). It next simultaneously learns the closeness between questions and sentences that may include answers as well as combinational optimization of those sentences. Our method also uses an attention mechanism to generate sentence representations according to the prior sentence; this extracts important topics in the sentence and tracks those topics in subsequent sentences. As a result, it can construct answers that have natural sentence flow whose topics correspond to the questions. Fig. 1-(b) explains the proposed neural-network by using examples. Here, the QA module designer first defines the abstract scenario for the answer as in the order of “conclusion” and “supplement”. Thus, there are three types of inputs “question”, “conclusion”, and “supplement”. It next runs biLSTMs over those inputs separately; it learns the order of word vectors such that “relationships” often appears next to “distance”. It then computes the embedding for the question, that for conclusion, and that for supplement by max-pooling over the hidden vectors output by biLSTMs. Finally, it computes the closeness between question and conclusion, that between question and supplement, and combinational optimization between conclusion and supplement with the attention mechanism, simultaneously (dotted lines in Fig. 1-(b) represent attention from conclusion to supplement).
We evaluated our method using datasets stored in the Japanese QA site Oshiete goo1. In particular, our evaluations focus on questions stored in the “Love advice” category since they are representative non-factoid questions: the questions are often complicated and most questions are very long. The results show that our method outperforms the previous methods including the method by (Tan et al. (2016)); our method accurately constructs answers by naturally combining key sentences that are highly close to the question.
2 RELATED WORK
Previous works on answer selection normally require feature engineering, linguistic tools, or external resources. Recent deep learning methods are attractive since they demonstrate superior performance compared to traditional machine learning methods without the above mentioned tiresome procedures. For example, (Wang & Nyberg (2015); Hu et al. (2014)) construct a joint feature vector on both question and answer and then convert the task into a classification or ranking problem. (Feng et al. (2015); Yu et al. (2014); dos Santos et al. (2015); Qiu & Huang (2015)) learn the question and answer representations and then match them by certain similarity metrics. Recently, Tan et al. (2016) took the latter approach and achieved more accurate results than the current strong baselines (Feng et al. (2015); Bendersky et al. (2011)). They, however, can only select answers and not generate them. Other than the above, recent neural text generation methods (Serban et al. (2015); Vinyals & Le (2015)) can also intrinsically be used for answer generation. Their evaluations showed that they could generate very short answer for factoid questions, but not the longer and more complicated answers demanded by non-factoid questions. Our Neural Answer Construction Model fills the gap between answer selection and generation for non-factoid QAs. It simultaneously learns the closeness between questions and sentences that may include answers as well as combinational optimization of those sentences. Since the sentences themselves in the answer are short, they can be generated by neural conversation models like (Vinyals & Le (2015));
As for word embeddings with semantics, some previous methods use the semantics behind words by using semantic lexicons such as WordNet and Freebase (Xu et al. (2014); Bollegala et al. (2016); Faruqui et al. (2015); Johansson & Nieto Piña (2015)). They, however, do not use the semantics be-
1http://oshiete.goo.ne.jp
hind the question/answer documents; e.g. document categories. Thus, they can not well catch the contexts in which the words appear in the QA documents. They also require external semantic resources other than QA datasets.
3 PRELIMINARY
Here, we explain QA-LSTM (Tan et al. (2015)), the basic discriminative framework for answer selection based on LSTM, since we base our ideas on its framework.
We first explain the LSTM and introduce the terminologies used in this paper. Given input sequence X = {x(1),x(2), · · · ,x(N)}, where x(t) is t-th word vector, t-th hidden vector h(t) is updated as:
it = σ(Wix(t) +Uih(t− 1) + bi) ft = σ(Wfx(t) +Ufh(t− 1) + bf ) ot = σ(Wox(t) +Uoh(t− 1) + bo) c̃t = tanh(Wcx(t) +Uch(t− 1) + bc) ct = it ∗ c̃t + ft ∗ ct−1
h(t) = ot ∗ tanh(ct)
There are three gates (input it, forget ft, and output ot), and a cell memory vector ct. σ is the sigmoid function. W ∈ RH×N , U ∈ RH×H , and b ∈ RH×1 are the network parameters to be learned. Single-direction LSTMs are weak in that they fail to make use of the contextual information from the future tokens. BiLSTMs use both the previous and future context by processing the sequence in two directions, and generate two sequences of output vectors. The output for each token is the concatenation of the two vectors from both directions, i.e. −−→ h(t) = −−→ h(t) ∥ ←−− h(t).
In the QA-LSTM framework, given input pair (q, a) where q is a question and a is a candidate answer, it first retrieves the word embeddings (WEs) of both q and a. Next, it separately applies a biLSTM over the two sequences of WEs. Then, it generates fixed-sized distributed vector representations oq for q (or oa for a) by computing max pooling over all the output vectors and then concatenating the resulting vectors on both directions of the biLSTM. Finally, it uses cosine similarity cos(oq,oa) to score the input (q, a) pair.
It then defines the training objective as the hinge loss of:
L = max{0,M − cos(oq,o+a ) + cos(oq,o−a )} where o+a is an output vector for ground truth answer, o − a is that for an incorrect answer randomly chosen from the entire answer space, and M is a margin. It treats any question with more than one ground truth as multiple training examples. Finally, batch normalization is performed on the representations before computing cosine similarity (Ioffe & Szegedy (2015)).
4 METHOD
We first explain our word embeddings with semantics.
4.1 WORD EMBEDDINGS WITH DOCUMENT SEMANTICS
This process is inspired by paragraph2vec (Le & Mikolov (2014)); an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents.
First, we explain paragraph2vec model. It averages the paragraph vector with several word vectors from a paragraph and predicts the following word in the given context. It trains both word vectors and paragraph vectors by stochastic gradient descent and backpropagation (Rumelhart et al. (1988)). While paragraph vectors are unique among paragraphs, the word vectors are shared.
Next, we introduce our method that incorporates the semantics behind QA documents into word embeddings (WEs) in the training phase. The idea is simple. Please see Fig. 2. It averages the vector
of category token and the vectors of title tokens, which are assigned to the QA documents, with several of the word vectors present in those documents. It then predicts the following word in the given context. Here, title tokens are defined by nouns that are extracted from titles assigned to the question. Multiple title tokens can be extracted from a title while one category token is assigned to a question. Those tokens are shared among datasets in the same category. It trains the category vector and title vectors as well as word vectors in QA documents as per paragraph2vec model. Those additional vectors are used as semantic biases for learning WEs. They are useful in emphasizing the words following the contexts of particular categories or titles. This improves the accuracies of answer selection described later as explained in Introduction.
For example, in Fig. 2, it can incorporate semantic biases from category “Love advice” into the words (e.g. “Will”, “distance”, “relationship”, “ruin”, “love” and so on) in the question in “Love advice”. Thus, it can well apply the biases from category “Love advice” to the words (e.g. “distance” and “relationship”) if they specifically appear in “Love advice”. On the other hand, words that appear in several categories (e.g. “will”) are biased with several categories and thus will not be emphasized.
4.2 NEURAL ANSWER CONSTRUCTION MODEL
Here, we explain our model. We first explain our approach and then the algorithm.
Approach It takes the following three approaches:
• Design the abstract scenario for the answer: The answer is constructed according to the order of the sentence types defined by the designer. For example, there are the sentence types such as sentence that states sympathy with the question, sentence that states a conclusion to the question, sentence that supplements the conclusion, and sentence that states encouragement to the questioner. This is inspired by the automated web service composition framework (Rao & Su (2005)) where the requester should build an abstract process before the web service composition planning starts. In our setting, the process is the scenario of answer and the service is the sentence in the scenario. Thus, our method can construct an answer by binding concrete sentences to fit the scenario. For example, the scenario for love advice can be designed as follows: it begins with a sympathy sentence (e.g. “You are struggling too.”), next it states a conclusion sentence (e.g. “I think you should make a declaration of love to her as soon as possible.”), then it supplements the conclusion by a supplemental sentence (e.g. “If you are too late, she maybe fall in love with someone else.”), and finally it ends with an encouragement sentence (e.g. “Good Luck!”).
• Joint neural network to learn sentence selection and combination: Our model computes the combination optimization among sentences that may include the answer as well as the closeness between question and sentences within a single neural network. This improves answer sentence selection; our model can avoid the cases in which the combination of sentences are not good enough though the scores of closeness between the question and each sentence are high. It also can let the parameter tuning simpler than the model that separates the network for sentence selection and that for sentence combination. The image of this neural-network is depicted in Fig. 1-(b). Here, it learns the closeness between sentence “Will distance relationship ruin love?” and “Distance cannot ruin true love”, the closeness between “Will distance relationship ruin love?” and “Distance certainly tests your love.”, and the combination between “Distance cannot ruin true love’ and “Distance certainly tests your love.”.
Algorithm 1 A neural answer construction model Input: Pairs of question, conclusion, and supplement, {(q, ac, and as)}. Output: Parameters set by the algorithm. 1: for n = 1, n++, while n < N do 2: for each pair (q, ac, as) do 3: Computes ocq and oc by biLSTMs and max pooling. 4: Computes osq by biLSTM and max pooling. 5: for each t-th hidden vector for supplement do 6: Computes h̃s(t) by Eq. (1). 7: end for 8: Computes os by max pooling. 9: Computes L by Eq. (2). 10: end for 11: end for
• Attention mechanism to improve the combination of sentences : Our method extracts important topics in the conclusion sentence and emphasizes those topics in the supplemental sentence in the training phase; this is inspired by (Tan et al. (2016)) who utilizes an attention mechanism to generate the answer representation following the question context. As a result, it can combine conclusions with the supplements following the contexts written in the conclusion sentences. This makes the story in the created answers very natural. In Fig. 1-(b), our attention mechanism extracts important topics (e.g. topic that represents “distance”) in the conclusion sentence “Distance cannot ruin true love” and emphasizes those topics in computing the representation of the supplement sentence “Distance certainly tests your love.”.
Procedure The core part of the answer is usually the conclusion sentence and its supplemental sentence. Thus, for simplicity, we here explain the procedure of our model in selecting and combining the above two types of sentences. As the reader can imagine, it can easily be applied to four sentence types. Actually, our love advice service by AI in oshiete-goo was implemented for four types of sentences, sympathy, conclusion, supplement, and encouragement (see Evaluation section). The model is illustrated in Fig. 1-(b) in which the input pair is (q, ac, as) where q is the question, ac is a candidate conclusion sentence, and as is a candidate supplemental sentence. The word embeddings (WEs) for words in q, ac, and as are extracted in the way described in the previous subsection. The procedure of our model is as follows (please see the Algorithm 1 also.):
(1) It iterates the following procedures (2) to (7) N times (line 1 in the algorithm).
(2) It picks up each pair (q, ac, and as) in the dataset (line 2 in the algorithm).
In the following steps (3) and (4), the same biLSTM is applied to both q and ac to compute the closeness between q and ac. Similarly, the same biLSTM is applied to both q and as. However, the biLSTM for computing closeness between q and ac differs from that between q and as since ac and as have different characteristics.
(3) It separately applies a biLSTM over the two sequences of WEs, q and ac, and computes the max pooling over the t-th hidden vector for question hcq(t) and that for conclusion hc(t). As a result, it acquires the question embedding, ocq and the conclusion embedding, oc (line 3 in the algorithm).
(4) It also separately applies a biLSTM over the two sequences of WEs, q and as, and computes the max pooling over the t-th hidden vector for question hsq(t) to acquire the question embedding, o s q (line 4 in the algorithm). osq is different from o c q since our method does not share the sub-network used for computing closeness between q and ac and that between q and as as described above.
(5) It applies the attention mechanism from conclusion to supplement. Specifically, given the output vector of biLSTM on the supplemental side at time step t, hs(t), and the conclusion embedding, oc, the updated vector h̃s(t) for each conclusion token is formulated as below (line 6 in the algorithm):
Wsm,Wcm, andwmb are attention parameters. Conceptually, the attention mechanism gives more weights on words that include important topics in the conclusion sentence.
(6) It computes the max pooling over h̃s(t) and acquires the supplemental embedding, os (line 8 in the algorithm).
(7) It computes the closeness between question and conclusion and that between question and supplement as well as the optimization combination between conclusion and supplement. The training objective is given as (line 9 in the algorithm):
L = max{0,M−(cos(oq, [o+c ,o+s ])−cos(oq, [o+c ,o−s ]))} (2) + max{0,M−(cos(oq, [o+c ,o+s ])−cos(oq, [o−c ,o+s ]))} + max{0, (1 + k)M −(cos(oq, [o+c ,o+s ])−cos(oq, [o−c ,o−s ]))} + max{0,M−(cos(oq, [o+c ,o−s ])−cos(oq, [o−c ,o−s ]))} + max{0,M−(cos(oq, [o−c ,o+s ])−cos(oq, [o−c ,o−s ]))}
where [y, z] is the concatenation of two vectors, y and z, oq is [ocq,o s q], o + is an output vector for a ground truth answer, and o− is that for an incorrect answer randomly chosen from the entire answer space. In the above equation, the first (or second) term presents the loss that occurs when both question-conclusion pair (q-c) and question-supplemental pair (q-s) are correct while q-c (or q-s) is correct but q-s (or q-c) is incorrect. The third term computes the loss that occurs when both q-c and q-s are correct while both q-c and q-s are incorrect. The fourth (or fifth) term computes the loss that occurs when q-c (or q-s) is correct but q-s (or q-c) is incorrect while both q-c and q-s are incorrect. M is constant margin and k (0< k< 1) is a parameter controlling the margin. Thus, the resulting margin for the third term is larger than those for other terms. In this way, by considering the case when either conclutions or supplements are incorrect or not, this equation optimizes the combinations among conclusion and supplement. In addition, it can take the closeness between question and conclusion (or supplement) in consideration by cosine similarity.
The parameter sets {Wi,Wf ,Wo,Wc,Ui,Uf ,Uo,Uc,bi,bf ,bo,bc}c for question-conclusion matching, {Wi,Wf ,Wo,Wc,Ui,Uf ,Uo,Uc,bi,bf ,bo,bc}s for question-supplement matching, and {Wsm,Wcm,wmb} for conclusion-supplement attention are trained during the iterations. After the model is trained, our method uses cos(oq, [oc,os]) to score the input (q, ac, as) pair and constructs an answer that has a conclusion and its supplement.
5 EVALUATION
We used our method to select or construct answers to the questions stored in “Love advice” category.
5.1 DATASET
We evaluated our method using a dataset stored in Japanese online QA service Oshiete-goo. First, the word embeddings were built by using 189,511 questions and their 771,956 answers stored in 16 categories including “Love Advice”, “Traveling”, and “Health Care”. 6,250 title tokens were extracted from the titles. Then, we evaluated answer selection and construction tasks by using a corpus containing about 5,000 question-conclusion-supplement sentences. Conclusions and supplement sentences were extracted by human experts from answers. The readers could use sentence extraction methods (Schmidt et al. (2014); Zhang et al. (2008); Nishikawa et al. (2010); Chen et al. (2010)) or neural conversation models like (Vinyals & Le (2015)) to semi-automatically extract/generate those sentences.
5.2 COMPARED METHODS
We compared the accuracy of the following five methods:
• QA-LSTM proposed by (Tan et al. (2015)). • Attentive LSTM: introduces an attention mechanism from question to answer and is evaluated as the current best answer selection method Tan et al. (2016).
• Semantic LSTM: performs answer selection by using our word embeddings biased with semantics.
• Construction: performs our proposed answer construction without attention mechanism. • Our method: performs our answer construction with attention mechanism from conclusion to supplement.
5.3 METHODOLOGY AND PARAMETER SETUP
We randomly divided the dataset into two halves, training dataset and predicted one, and conducted two-fold cross validation. Results shown later are the average values.
Both for answer selection and construction, we used Average Precision (AP) against the top-K ranked answers in the results because we consider that the most highly ranked answers are important for users. If the number of ranked items isK, the number of correct answers among the top-j ranked itemsNj , and the number of all correct answers (paired with the questions)D, AP is defined as follows:
AP = 1
D ∑ 1≤j≤K Nj j
For answer construction, we checked whether each method could recreate the original answers. As the reader easily can understand, this is a much more difficult task than answer selection and thus the values of AP will be smaller than the results for answer selection.
We tried word vectors and qa vectors of different sizes, and finally set the word vector size to 300 and the LSTM output vectors for biLSTMs to 50 × 2. We also tried different margins in the hinge
loss function, and fixed the margin,M , to 0.2 and k to 1.0. The iteration count N was set to 20. For our method, the embeddings for questions, those for conclusions, and those for supplements were pretrained by Semantic LSTM before answer construction since this enhances the overall accuracy.
We did not use attention mechanism from question to answer for Semantic LSTM, Construction and Our method. This is because, as we present in the results subsection, the lengths of questions are much longer than those of answer sentences, and thus the attention mechanism from question to answer became noise for sentence selection.
5.4 RESULTS
We now present the results of the evaluations.
Answer Selection We first compare the accuracy of methods for answer selection. The results are shown in Table 1. QA-LSTM and Attentive LSTM are worse than Semantic-LSTM. This indicates that Semantic-LSTM can incorporate semantic information (titles/categories) into word embeddings; it can emphasize words according to the context they appeared and thus the matching accuracy between question vector and conclusion (supplement) vector was improved. Attentive LSTM is worse than QA-LSTM as described above. Construction and Our method are better than Semantic-LSTM. This is because they can avoid the combinations of sentences that are not good enough even though the scores of closeness between questions and sentences are high. This implies that, if the combination is not good, the selection of answer sentences also tends to be erroneous. Finally, Our method, which provides sophisticated selection/combination strategies, yielded higher accuracy than the other methods. It achieved 4.4% higher accuracy than QA-LSTM (QA-LSTM marked 0.8472 while Our method marked 0.8846.).
Answer Construction We then compared the accuracy of the methods for answer construction. Especially for the answer construction task, the top-1 result is most important since many QA applications show only the top-1 answer. The results are shown in Table 2. There is no answer construction mechanism in QA-LSTM, Attentive-LSTM, and Semantic-LSTM. Thus we simply merge the conclusion and supplement, each of which has the highest similarity with the question by each method. QA-LSTM and Attentive LSTM are much worse than Semantic-LSTM. This is because the sentences output by Semantic-LSTM are selected by utilizing the words that are emphasized for a context for “Love advice” (i.e. category and titles). Construction is better than Semantic-LSTM since it simultaneously learns the optimum combination of sentences as well as the closeness between the question and sentences. Finally, Our method is better than Construction. This is because it well employs the attention mechanism to link conclusion and supplement sentences and thus the combinations of the sentences are more natural than those of Construction. Our method achieved 20% higher accuracy than QA-LSTM (QA-LSTM marked 0.3262 while Our method marked 0.3901.).
The computation time for our method was less than two hours. All experiments were performed on NVIDIA TITAN X/Tesla M40 GPUs, and all methods were implemented by Python in the Chainer framework. Thus, our method well suits real applications. In fact, it is already being used in the love advice service of Oshiete goo 2.
Human evaluation The outputs of QA-LSTM and Our method were judged by two human experts. The experts entered the questions, which were not included in our evaluation datasets, to the AI system and rated the created answers based on the following scale: (1) the conclusion and supplement sentences as well as their combination were good, (2) the sentences were good in isolation but their combination was not good, (3) One of the selections (conclusion or supplement) was good but their combination was not good, and (4) both sentences and their combination were not good. The answers were judged as good if they satisfied the following two points: (A) the contents of answer sentences correspond to the question. (B) the story between conclusion and supplement is natural.
The results are shown in Table 3. Table 4 presents examples of the questions and answers constructed (they were originally Japanese and translated into English for readability. The questions are summarized since the original ones were very long.). The readers can also see Japanese answers from our service URL presented the above. Those results indicate that the experts were much more satisfied with the outputs of Our method than those by QA-LSTM; 58 % of the answers created by Our method were classified as (1). This is because, as can be see in Table 4, Our method can naturally
2http://oshiete.goo.ne.jp/ai
combine the sentences as well as select sentences that match the question. It well coped with the questions that were somewhat different from those stored in the evaluation dataset.
Actually, when the public used our love advice service, it was surprising to find that the 455 answers created by the AI whose name is oshi-el (uses Our method) were judged as Good answers by users from among the 1,492 questions entered from September 6th to November 5th3. The rate of getting Good answers by oshi-el is twice that of the average human user in oshiete-goo when we focus on users who answered more than 100 questions in love advice category. Thus, we think this is a good result.
6 CONCLUSION
This is the first study that create answers for non-factoid questions. Our method incorporates the biases of semantics behind questions into word embeddings to improve the accuracy of answer selection. It then simultaneously learns the optimum combination of answer sentences as well as the closeness between questions and sentences. Our evaluation shows that our method achieves 20 % higher accuracy in answer construction than the method based on the current best answer selection method. Our model presents an important direction for future studies on answer generation. Since the sentences themselves in the answer are short, they can be generated by neural conversation models like (Vinyals & Le (2015)); this means that our model can be extended to generate complete answers once the abstract scenario is made.

Summary: The paper presents an approach – Neural Answer Construction Model for the task of answering non-factoid questions, in particular, love-advice questions. The two main features of the proposed model are the following – 1) it incorporates the biases of semantics behind questions into word embeddings, 2) in addition to optimizing for closeness between questions and answers, it also optimizes for optimum combination of sentences in the predicted answer. The proposed model is evaluated using the dataset from a Japanese online QA service and is shown to outperform the baseline model (Tan et al. 2015) by 20% relatively (6% absolutely). The paper also experiments with few other baseline models (ablations of the proposed model).
Strengths:
1. The two motivations behind the proposed approach – need to understand the ambiguous use of words depending on context, and need to generate new answers rather than just selecting from answers held by QA sites – are reasonable.
2. The novelty in the paper involves the following – 1) incorporating biases of semantics behind questions into word embeddings using paragraph2vec like model, modified to take as inputs - words from questions, question title token and question category token, 2) modelling optimum combination of sentences (conclusion and supplement sentences) in the predicted answer, 3) designing abstract scenario for answers, inspired by automated web-service composition framework (Rao & Su (2005)), and 4) extracting important topics in conclusion sentence and emphasizing them in supplemental sentence using attention mechanism (attention mechanism is similar to Tan et al. 2016).
3. The proposed method is shown to outperform the current best method (Tan et al. 2015) by 20% relatively (6% absolutely) which seems to be significant improvement.
4. The paper presents few ablations studies that provide insights on how much different components of the model (such as incorporating biases into word embeddings, incorporating attention from conclusion to supplement) are helping towards performance improvement.
Weaknesses/Suggestions/Questions:
1. How are the abstract patterns determined, i,e., how did the authors determine that the answers to love-advice questions generally constitute of sympathy, conclusion, supplement for conclusion and encouragement? How much is the improvement in performance when using abstract patterns compared to the case when not using these patters, i.e. when candidate answers are picked from union of all corpuses rather than picking from respective corpuses (corpuses for sympathy, conclusion etc.).
2. It seems that the abstract patterns are specific to the type of questions. So, the abstract patterns for love-advice will be different from those for business advice. Thus, it seems like the abstract patterns need to be hand-coded for different types and hence one model cannot generalize across different types.
3. The paper should present explicit analysis of how much combinational optimization between sentences help – comparison between model performance with and without combinational optimization keeping rest of the model architecture same. The authors could also plot the accuracy of the model as a function of the combinational optimization scores. This will provide insights into how significant are the combinational optimization scores towards overall model accuracy.
4. Paper says that current systems designed for non-factoid QA cannot generalize to questions outside those stored in QA sites and claims that this is one of the contributions of this paper. In order to ground that claim, the paper should show experimentally how well the proposed method generalized to such out-of-domain questions. Although the questions asked by human experts in the human evaluation were not from the evaluation datasets, the paper should analyze how different those questions were compared to the questions present in the evaluation datasets.
5. For human evaluation, were the outputs of the proposed model and that of the QA-LSTM model judged each judged by both the human experts OR one of the human experts judged the outputs of one system and the other human expert judged the outputs of the other system? If both the sets of outputs were each judged by both the human experts, how were the ratings of the two experts combined for every questions?
6. I wonder why the authors did not do a human evaluation where they just ask human workers (not experts) to compare the output of the proposed model with that of the QA-LSTM model – which of the two outputs they would like to hear when asking for advice. Such an evaluation would not get biased by whether each sentence is good or not, whether the combination is good or not. Looking at the qualitative examples in Table 4, I personally like the output of the QA-LSTM more than that of the proposed model because they seem to provide a direct answer to the question (e.g., for the first example the output of the QA-LSTM says “You should wait until you feel excited”, whereas the output of the proposed model says “It is better to concentrate on how to confess your love to her” which seems a bit indirect to the question asked.)
7. Given a question, is the ground-truth answer different in the two tasks -- answer selection and answer construction?
8. The paper mentions that Attentive LSTM (Tan et al. 2016) is evaluated as the current best answer selection method (section 5.2). So, why is its accuracy lower than that of QA-LSTM in table 1. The authors explain this by pointing out the issue of questions being very long compared to answers and hence the attention being noisy. But, did these issues not exist in the dataset used by Tan et al. 2016?
9. The paper says the proposed method achieves 20% gain over current best (in Conclusion section) where they refer to QA-LSTM as the current best method. However, in the description of Attentive LSTM (section 5.2), the paper mentions that Attention LSTM is the current best method. So, could authors please clarify the discrepancy?
10. Minor correction: remove space between 20 and % in abstract.
Review Summary: The problem of non-factoid QA being dealt with in the paper is an interesting and useful problem to solve. The motivations presented in the paper behind the proposed approach are reasonable. The experiments show that the proposed model outperforms the baseline model. However, the use of abstract patterns to determine the answer seems like hand-designing and hence it seems like these abstract patterns need to be designed for every other type of non-factoid question and hence the proposed approach is not generalizable to other types. Also, the paper needs more analysis of the results to provide insights into the contribution of different model components.


ENET: A DEEP NEURAL NETWORK ARCHITECTURE FOR REAL-TIME SEMANTIC SEGMENTATION
The ability to perform pixel-wise semantic segmentation in real-time is of paramount importance in practical mobile applications. Recent deep neural networks aimed at this task have the disadvantage of requiring a large number of floating point operations and have long run-times that hinder their usability. In this paper, we propose a novel deep neural network architecture named ENet (efficient neural network), created specifically for tasks requiring low latency operation. ENet is up to 18× faster, requires 75× less FLOPs, has 79× less parameters, and provides similar or better accuracy to existing models. We have tested it on CamVid, Cityscapes and SUN datasets and report on comparisons with existing state-of-the-art methods, and the trade-offs between accuracy and processing time of a network. We present performance measurements of the proposed architecture on embedded systems and suggest possible software improvements that could make ENet even faster.
1 INTRODUCTION
Recent interest in augmented reality wearables, home-automation devices, and self-driving vehicles has created a strong need for semantic-segmentation (or visual scene-understanding) algorithms that can operate in real-time on low-power mobile devices. These algorithms label each and every pixel in the image with one of the object classes. In recent years, the availability of larger datasets and computationally-powerful machines have helped deep convolutional neural networks (CNNs) (LeCun & Bengio (1998); Krizhevsky et al. (2012); Simonyan & Zisserman (2014a); Szegedy et al. (2015a)) surpass the performance of many conventional computer vision algorithms (Shotton et al. (2009); Perronnin et al. (2010); van de Sande et al. (2011)). Even though CNNs are increasingly successful at classification and categorization tasks, they provide coarse spatial results when applied to pixel-wise labeling of large images. Therefore, they are often cascaded with other algorithms to refine the results, such as color based segmentation (Farabet et al. (2013)) or conditional random fields (Chen et al. (2014)), to name a few.
In order to both spatially classify and finely segment images, several neural network architectures have been proposed, such as SegNet (Badrinarayanan et al. (2015a;b)) or fully convolutional networks (Long et al. (2015)). All these works are based on a VGG16 (Simonyan & Zisserman (2014b)) architecture, which is a very large model designed for multi-class classification. These references use models with a large number of parameters, and slow inference time. In these conditions, they become unusable for many mobile or battery-powered applications, which require processing images at rates higher than 10 fps.
In this paper, we propose a new neural network architecture optimized for high-accuracy and also fast inference. In our work, beside neural network processing, we chose not to use any other post-processing steps, in order to focus on the intrinsic performance of an end-to-end CNN approach.
In Section 3 we propose a fast and compact encoder-decoder architecture named ENet. It has been designed according to rules and ideas that have appeared in the literature recently, all of which we discuss in Section 4. Performance of the proposed network has been tested on Cityscapes (Cordts et al. (2016)) and CamVid (Brostow et al. (2008)) for driving scenario, whereas SUN dataset (Song et al. (2015)) has been used for testing our network in an indoor situation. We benchmark it on NVIDIA Jetson TX1 Embedded Systems Module as well as on an NVIDIA Titan X GPU. The results can be found in Section 5.
2 RELATED WORK
Semantic segmentation is important in fully understanding the content of images, find target objects and segment them. This technique is of utmost importance in applications such as driving and augmented reality. Moreover, real-time operation is a must for these applications, and therefore, designing CNNs carefully is vital. Contemporary computer vision applications extensively use deep neural networks, now one of the most widely used techniques for many different tasks, including semantic segmentation. This work presents a fully trainable neural network architecture, and therefore we aim to compare to other literature that performs the large majority of inference in the same way.
State-of-the-art scen-parsing CNNs use two separate neural network architectures combined together: an encoder and a decoder. Inspired by probabilistic auto-encoders (Ranzato et al. (2007); Ngiam et al. (2011)), encoder-decoder network architecture have been introduced in SegNet-basic (Badrinarayanan et al. (2015a)), and further improved in SegNet (Badrinarayanan et al. (2015b)). The encoder is a vanilla CNN (such as VGG16 from Simonyan & Zisserman (2014b)) which is trained to classify the input, while the decoder is used to upsample the output of the encoder (Long et al. (2015); Noh et al. (2015); Zheng et al. (2015); Eigen & Fergus (2015); Hong et al. (2015)). However, these networks are slow during inference due to their large architectures and numerous parameters. Unlike in Noh et al. (2015), fully connected layers of VGG16 were discarded in the latest incarnation of SegNet, in order to reduce the number of operations and memory footprint, making it the smallest of these networks. Still, none of them can operate in real-time.
Other existing architectures use simpler classifiers and then cascade it with Conditional Random Field (CRF) as a post-processing step (Chen et al. (2014); Sturgess et al. (2009)). As explained in Badrinarayanan et al. (2015b), these techniques use onerous post-processing steps and often fail to label the classes that occupy fewer number of pixels in a frame. CNNs can be combined with recurrent neural networks (Zheng et al. (2015)) for better performance, but suffers from further speed degradation. Also, one has to keep in mind that RNN, used as a post-processing step, can be used in conjunction with any other technique, including the one presented in this work.
3 NETWORK ARCHITECTURE
The architecture of our network is presented in Table 1. It is divided into several stages, as highlighted by horizontal lines in the table and the first digit after each block name. Output sizes are reported for an example input image resolution of 512× 512. We adopt a view of ResNets (He et al. (2015b)) that describes them as having a single main branch and extensions with convolutional filters that separate from it, and then merge back with an element-wise addition, as shown in Figure 1b. Just as in the original paper, we refer to these as bottleneck modules. They consist of three convolutional layers: a 1×1 projection that reduces the dimensionality, a main convolutional layer (conv in Figure 1b), and a 1× 1 expansion. If the bottleneck is downsampling, a max pooling layer is added to the main branch. We zero pad the activations, to match the number of feature maps. Also, the first 1× 1 projection is replaced with a 2× 2 convolution with stride 2 in both dimensions. conv is either a regular, dilated or full convolution (also known as deconvolution or fractionally strided convolution) with 3 × 3 filters. Sometimes we replace it with asymmetric convolution i.e. a sequence of 5 × 1 and 1× 5 convolutions. For the regularizer, we use Spatial Dropout (Tompson et al. (2015)), with p = 0.01 before bottleneck2.0, and p = 0.1 afterwards.
The initial stage contains a single block, that is presented in Figure 1a. Stage 1 consists of 5 bottleneck blocks, while stage 2 and 3 have the same structure, with the exception that stage 3 does not downsample the input at the beginning (we omit the 0th bottleneck). These three first stages are the encoder. Stage 4 and 5 belong to the decoder.
We did not use bias terms in any of the projections, in order to reduce the number of kernel calls and overall memory operations, as cuDNN (Chetlur et al. (2014)) uses separate kernels for convolutions and bias addition. This choice didn’t have any impact on the accuracy. Between each convolutional layer and following non-linearity we use Batch Normalization (Ioffe & Szegedy (2015)). In the decoder max pooling is replaced with max unpooling, and padding is replaced with spatial convolution without bias. We did not use unpooling information in the last upsampling module, because the initial block operated on the 3 channels of the input frame, while the final output has C feature maps (the number of object classes). Also, for performance reasons, we decided to place only a bare full convolution as last module of the network, which alone takes up a sizeable portion of the decoder processing time.
4 DESIGN CHOICES
In this section we will discuss our most important experimental results and intuitions, that have shaped the final architecture of ENet.
Feature map resolution: Downsampling images during semantic segmentation has two main drawbacks. Firstly, reducing feature map resolution implies loss of spatial information like exact edge shape. Secondly, full pixel segmentation requires that the output has the same resolution as the input. This implies that strong downsampling will require equally strong upsampling, which increase model size and computational cost. The first issue has been addressed in Long et al. (2015) by adding the feature maps produced by encoder, and in SegNet (Badrinarayanan et al. (2015a)) by saving the elements index from the corresponding encoder max-pooling module. We followed the SegNet approach, because it allows to reduce memory requirements, but we found that using a strong downsampling still reduces the final accuracy.
However, downsampling has one big advantage. Filters operating on downsampled images have a bigger receptive field, that allows them to gather more context. This is especially important when trying to differentiate between classes occupying a small portion of the overall image, as, for example, rider and pedestrian in a road scene. It is just not enough that the network learns how people look, the context in which they appear is important as well. At the end, we have found that it is better to use dilated convolutions for the purpose of extending context information (Yu & Koltun (2015)).
Early downsampling: One crucial intuition to achieving good performance and real-time operation is realizing that processing large input frames is very expensive. This might sound very obvious, however many popular architectures (Hong et al. (2015); Badrinarayanan et al. (2015b)) do not pay much attention towards optimization of early stages of network, which are often the most expensive.
ENet first two blocks heavily reduce the input size, and use only a small set of feature maps. The idea behind it, is that visual information is highly redundant in space, and thus can be compressed into a more efficient representation. Also, our intuition is that the initial network layers should not be used specifilly only for classification. Instead, they should rather act as good feature extractors and preprocess the input for later portions of the network. This insight worked well in our experiments. Increasing the number of feature maps from 16 to 32 did not improve the accuracy on Cityscapes dataset (Cordts et al. (2016)).
Decoder size: In this work we would like to provide a different view on encoder-decoder architectures than the one presented in Badrinarayanan et al. (2015b). SegNet is a very symmetric architecture, as the encoder is an exact mirror of the encoder. Instead, our architecture consists of a large encoder, and a small decoder. This is motivated by the idea that the encoder should be able to work in a similar fashion to original classification architectures, i.e. to operate on smaller resolution data and provide for information processing and filtering. Instead, the role of the the decoder, is only to upsample the output of the encoder, fine-tuning the details.
Nonlinear operations: He et al. (2016) report that it is beneficial to use ReLU and Batch Normalization layers before convolutions. We tried applying these ideas to ENet, but this had a detrimental effect on accuracy. Investigating its cause we replaced all ReLUs in the network with PReLUs (He et al. (2015a)), which use an additional parameter per feature map, with the goal of learning the negative slope of non-linearities. We expected that in layers where identity is a preferable transfer function, PReLU weights will have values around 1, and conversely, values around 0 if ReLU is preferable. Results of this experiment can be seen in Figure 2.
The first layers weights exhibit a large variance and are slightly biased towards positive values, while in the later portions of the encoder they settle to recurring pattern. All layers in the main branch behave nearly exactly like regular ReLUs, while the weights inside bottleneck modules are negative i.e. the function inverts and scales down negative values. We hypothesize that identity did not work out well in our architecture because of its limited depth. We hypothesize that the reason why such lossy functions are learned is that He et al. (2016) uses networks that are hundreds of layers deep, while our network uses fewer layers, and it needs to quickly filter out information. It is notable that the decoder weights become much more positive and learn functions closer to identity. This confirms our intuitions that the decoder is used only to fine-tune the upsampled output.
Information-preserving dimensionality changes: As stated earlier, it is necessary to downsample the input early, but aggressive dimensionality reduction can also hinder the information flow. A very good approach to this problem has been presented in Szegedy et al. (2015b). However, pooling after a convolution, in case of increasing feature map depth, is computationally expensive. Therefore, we prefer to perform pooling operation in parallel with convolution of stride 2, and concatenate resulting feature maps. This technique allowed us to speed up inference time of the initial block 10 times.
Additionally, we have found one problem in the original ResNet architecture. When downsampling, the first 1×1 projection of the convolutional branch is performed with a stride of 2 in both dimensions, which effectively discards 75% of the input. Increasing the filter size to 2× 2 allows to take the full input into consideration, and thus improves the information flow and accuracy. Of course, it makes these layers 4× more computationally expensive, however there are so few of these in ENet, that the overhead is not noticeable.
Factorizing filters: It has been shown that convolutional weights have a fair amount of redundancy, and each n× n convolution can be decomposed into two smaller asymmetric convolutions: one with a n × 1 filter followed by a 1 × n filter (Jin et al. (2014); Szegedy et al. (2015b)). We have used asymmetric convolutions with n = 5 in our network, so cost of these two operations is similar to a single 3× 3 convolution. This allowed to increase the variety of functions learned by each block and increase the receptive field.
Sequence of operations used in the bottleneck module (projection, convolution, projection) can be seen as decomposing one large convolutional layer into a series of smaller and simpler low-rank approximation operations. Such factorization allows for large speedups and reduction in number of parameters, making them less redundant (Jin et al. (2014)).
Dilated convolutions: As argued above, it is very important for the network to have a wide receptive field, so it can perform classification by taking a bigger portion of the image (context) into account. We wanted to avoid overly downsampling the feature maps, and decided to use dilated convolutions (Yu & Koltun (2015)) to improve our model. We have used them inside several bottleneck modules, in particular the ones that operate on the smallest resolutions. These gave a significant accuracy boost, by raising IoU on Cityscapes by around 4 percentage points, with no additional cost. We obtained the best accuracy when we interleaved them with other bottleneck modules (both regular and asymmetric), instead of arranging them in sequence, as has been done in Yu & Koltun (2015).
Regularization: Most pixel-wise segmentation datasets are relatively small (on order of 103 images), so such expressive models as neural networks quickly begin to overfit them. In initial experiments, we used L2 weight decay with little success. Then, inspired by Huang et al. (2016), we have tried stochastic depth, which increased accuracy. However it became apparent that dropping branches (i.e. setting their output to 0) is in fact a special case of applying Spatial Dropout (Tompson et al. (2015)), where either all of the channels, or none of them are ignored, instead of selecting a random subset. We placed Spatial Dropout at the end of convolutional branches, right before the addition, and it turned out to work much better than stochastic depth.
5 RESULTS
We benchmarked the performance of ENet on three different datasets to demonstrate real-time and accurate for practical applications. We tested on CamVid and Cityscapes datasets of road scenes, and SUN RGB-D dataset of indoor scenes. We set SegNet (Badrinarayanan et al. (2015b)) as a baseline since it is one of the fastest segmentation model available, that also requires less memory to operate than plain CNNs. All our models, training, testing and performance evaluation scripts were written using the Torch7 machine-learning library. To compare results, we use class average accuracy and intersection-over-union (IoU) metrics.
5.1 PERFORMANCE ANALYSIS
We report results on inference speed on widely used NVIDIA Titan X GPU as well as on NVIDIA TX1 embedded system module. ENet was designed to achieve more than 10 fps on the NVIDIA TX1 board with an input image size 640× 360 (W,H), which is adequate for practical road scene parsing applications. For inference we merge batch normalization and dropout layers into the convolutional filters, to speed up all networks.
Inference time: Table 2 compares inference time for a single input frame of varying resolution. We also report the number of frames per second that can be processed. Dashes indicate that we could not obtain a measurement, due to lack of memory. ENet is significantly faster than competing architectures, providing high frame rates for real-time applications and allowing for practical use of very deep neural network models with encoder-decoder architecture.
Hardware requirements: Table 3 reports a comparison of number of operations and parameters used by different models. ENet efficiency is evident in the much low number of operations per frame and overall parameters. Please note that we report storage required to store the models in half precision floating point format. ENet has so few parameters that it can be saved into a file of just 0.7MB, which makes it possible to fit the whole network in an extremely fast on-chip memory in embedded processors. This alleviates the need for model compression (Han et al. (2015)), making it possible to use general purpose code for neural network computation. However, if one needs to operate under incredibly strict memory constraints, these techniques can still be applied to ENet as well.
Software limitations: One of the most important techniques that has allowed us to reach these levels of performance is convolutional layer factorization. However, we have found one surprising drawback. Although applying this method allowed us to greatly reduce the number of floating point operations and parameters, it also increased the number of individual kernels calls, making each of them smaller.
We have found that some of these operations become so cheap, that the cost of GPU kernel launch starts to outweigh the cost of the actual computation. Also, because kernels do not have access to values that have been kept in registers by previous ones, they have to load all data from global memory at launch, and save it when their work is finished. This means that using a higher number of kernels, increases the number of memory operations, because feature maps have to be constantly saved and reloaded. This becomes especially apparent in case of non-linear operations. In ENet, PReLUs consume more than a quarter of inference time. Since they are only simple point-wise operations and very easy to parallelize, we hypothesize it is caused by the aforementioned data movement.
These are serious limitations, however they could be resolved by performing kernel fusion in existing software i.e. create kernels that apply non-linearities to results of convolutions directly, or perform a
number of smaller convolutions in one call. This improvement in GPU libraries, such as CuDNN, could increase the speed and efficiency of our network even further.
5.2 BENCHMARKS
During training we have used the Adam optimization algorithm (Kingma & Ba (2014)). It allowed ENet to converge very quickly and on every dataset we haver used training took only 3-4 hours on Titan X. Training of ENet was performed in two stages: first we train only the encoder to categorize downsampled regions of the input image, then we appended the decoder and train the network to perform upsampling and pixel-wise categorization. in this work, a learning rate of 5e−4 and L2 weight decay of 2e−4, along with batch size of 10 consistently provided the best results. For categorization, we have used a custom class weighing scheme defined as wclass = 1ln(c+pclass) . In contrast to the inverse class probability weighing, the weights are bounded as the probability approaches 0. c is an additional hyper-parameter, which we set to 1.02 (i.e. we restrict the class weights to be in the interval of [1, 50]).
Cityscapes: This dataset consists of 5000 fine-annotated images, out of which 2975 are available for training, 500 for validation, and the remaining 1525 have been selected as test set (Cordts et al. (2016)). Cityscapes was the most important benchmark for us, because of its outstanding quality and highly varying road scenarios, often featuring many pedestrians and cyclists. We trained on 19 classes that have been selected in the official evaluation scripts (Cordts et al. (2016)). It makes use of an additional metric called instance-level intersection over union metric (iIoU), which is IoU weighed by the average object size. As reported in Table 4, ENet outperforms SegNet in class IoU and iIoU, as well as in category IoU. ENet is currently the fastest model in the Cityscapes benchmark.
CamVid: Another automotive dataset, on which we have tested ENet, was CamVid. It contains 367 training and 233 testing images (Brostow et al. (2008)). There are eleven different classes such as building, tree, sky, car, road, etc. while the twelfth class contains unlabeled data, which we ignore while training. The original frame resolution for this dataset is 960×720 (W,H) but we downsampled the images to 480×360 before training. In Table 5 we compare the performance of ENet with existing state-of-the-art algorithms. ENet outperforms other models in six classes, which are difficult to learn because they correspond to smaller objects.
SUN RGB-D: The SUN dataset consists of 5285 training images and 5050 testing images with 37 indoor object classes. We did not make any use of depth information in this work and trained the network only on RGB data. In Table 6 we compare the performance of ENet with SegNet (Badrinarayanan et al. (2015b)), which is the only neural network model that reports accuracy on this dataset. Our results, though inferior in global average accuracy and IoU, are comparable in class average accuracy. Since global average accuracy and IoU are metrics that favor correct classification of classes occupying large image patches, researchers generally emphasize the importance of other metrics in case of semantic segmentation. One notable example is introduction of iIoU metric (Cordts et al. (2016)). Comparable result in class average accuracy indicates, that our network is capable of differentiating smaller objects nearly as well as SegNet. Moreover, the difference in accuracy should not overshadow the huge performance gap between these two networks. ENet can process the images in real-time, and is nearly 20× faster than SegNet on embedded platforms.
6 CONCLUSION
We have proposed a novel neural network architecture designed from the ground up specifically for semantic segmentation. Our main aim is to make efficient use of scarce resources available on embedded platforms, compared to fully fledged deep learning workstations. Our work provides large gains in this task, while matching and at times exceeding existing baseline models, that have an order of magnitude larger computational and memory requirements. The application of ENet on the NVIDIA TX1 hardware exemplifies real-time portable embedded solutions.
Even though the main goal was to run the network on mobile devices, we have found that it is also very efficient on high end GPUs like NVIDIA Titan X. This may prove useful in data-center applications, where there is a need of processing large numbers of high resolution images. ENet allows to perform large-scale computations in a much faster and more efficient manner, which might lead to significant savings.
ACKNOWLEDGMENT
This work is partly supported by the Office of Naval Research (ONR) grants N00014-12-1-0167, N00014-15-1-2791 and MURI N00014-10-1-0278. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the TX1, Titan X, K40 GPUs used for this research.

Paper summary: this work presents ENet, a new convnet architecture for semantic labeling which obtains comparable performance to the previously existing SegNet while being ~10x faster and using ~10x less memory.
Review summary: Albeit the results seem interesting, the paper lacks detailed experimental results, and is of limited interest for the ICLR audience.
Pros:
* 10x faster
* 10x smaller
* Design rationale described in detail
Cons:
* The quality of the reference baseline is low. For instance, cityscapes results are 58.3 IoU while state of the art is ~80 IoU. Thus the results are of limited interest.
* The results that support the design rationale are not provided. It is important to provide the experimental evidence to support each claim.
Quality: the work is interesting but feels incomplete. If your model is 10x faster and smaller, why not try build a model 10x longer to obtain improved results ? The paper focuses only on  nimbleness at the cost of quality (using a weak baseline). This limits the interest for the ICLR audience.
Clarity: the overall text is somewhat clear, but the model description (section 3) could be more clear.
Originality: the work is a compendium of “practitioners wisdom” applied to a specific task. It has thus limited originality.
Significance: I find the work that establishes a new “best practices all in one” quite interesting, but however these must shine in all aspects. Being fast at the cost of quality, will limit the impact of this work.
Minor comments:
* Overall the text is proper english but the sentences constructions is often unsound, specific examples below.
* To improve the chances of acceptance, I invite the authors to also explore bigger models and show that the same “collected wisdom” can be used both to reach high speed and high quality (with the proper trade-off curve being shown). Aiming for only one end of the quality versus speed curve limits too much the paper.
* Section 1: “mobile or battery powered … require rates > 10 fps“. 10 fps with which energy budget ? Should not this be  > 10 fps && < X Watt.
* “Rules and ideas” -> rules seem too strong of a word, “guidelines” ?
* “Is of utmost importance” -> “is of importance” (important is already important)
* “Presents a trainable network … therefore we compare to … the large majority of inference the same way”; the sentence makes no sense to me, I do not see the logical link between before and after “therefore”
* Scen-parsing -> scene-parsing
* It is arguable if encoder and decoder can be called “separate”
* “Unlike in Noh” why is that relevant ? Make explicit or remove
* “Real-time” is vague, you mean X fps @ Y W ?
* Other existing architectures -> Other architectures
* Section 3, does not the BN layer include a bias term ? Can you get good results without any bias term ?
* Table 1: why is the initial layer a downsampling one, since the results has half the size of the input ?
* Section 4, non linear operations. What do you mean by “settle to recurring pattern” ?
* Section 4, dimensionality changes. “Computationally expensive”, relative to what ?
* Section 4, dimensionality changes. “This technique ... speeds-up ten times”, but does not provide the same results. Without an experimental validation changing an apple for an orange does not make the orange better than the apple.
* Section 4, dimensionality changes. “Found one problem”, problem would imply something conceptually wrong. This is more an “issue” or an “miss-match” when using ResNet for semantic labelling.
* Section 4, factorizing filters. I am unsure of why you call nx1 filter asymmetric. A filter could be 1xn yet be symmetric (e.g. -2 -1 0 1 2). Why not simply call them rectangular filters ?
* Section 4, factorizing filters. Why would this change increase the variety ? I would have expected the opposite.
* Section 4, regularization. Define “much better”.
* Section 5.1; “640x360 is adequate for practical applications”; for _some_ applications.
* Section 5.2, “very quickly” is vague and depends on the reader expectations, please be quantitative.
* Section 5.2, Haver -> have
* Section 5.2, in this work -> In this work
* Section 5.2, unclear what you use the class weighting for. Is this for class balancing ?
* Section 5.2, Cityscapes was -> Cityscapes is
* Section 5.2, weighted by the average -> is each instance weighted relative the average object size.
* Section 5.2, fastest model in the Cityscapes -> fastest model in the public Cityscapes


REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS
We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.
We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.
1 INTRODUCTION
Large neural networks with millions of parameters achieve strong performance on image classification (Szegedy et al., 2015a), machine translation (Wu et al., 2016), language modeling (Jozefowicz et al., 2016), and speech recognition (Graves et al., 2013). However, despite using large datasets, neural networks are still prone to overfitting. Numerous techniques have been proposed to prevent overfitting, including early stopping, L1/L2 regularization (weight decay), dropout (Srivastava et al., 2014), and batch normalization (Ioffe & Szegedy, 2015). These techniques, along with most other forms of regularization, act on the hidden activations or weights of a neural network. Alternatively, regularizing the output distribution of large, deep neural networks has largely been unexplored.
To motivate output regularizers, we can view the knowledge of a model as the conditional distribution it produces over outputs given an input (Hinton et al., 2015) as opposed to the learned values of its parameters. Given this functional view of knowledge, the probabilities assigned to class labels that are incorrect (according to the training data) are part of the knowledge of the network. For example, when shown an image of a BMW, a network that assigns a probability of 10−3 to “Audi” and 10−9 to “carrot” is clearly better than a network that assigns 10−9 to “Audi” and 10−3 to carrot, all else being equal. One reason it is better is that the probabilities assigned to incorrect classes are an indication of how the network generalizes. Distillation (Hinton et al., 2015; Bucilu et al., 2006) exploits this fact by explicitly training a small network to assign the same probabilities to incorrect classes as a large network or ensemble of networks that generalizes well. Further, by operating on the output distribution that has a natural scale rather than on internal weights, whose significance depends on the values of the other weights, output regularization has the property that it is invariant to the parameterization of the underlying neural network.
∗Work done as part of the Google Brain Residency Program †Equal Contribution
In this paper, we systematically evaluated two output regularizers: a maximum entropy based confidence penalty and label smoothing (uniform and unigram) for large, deep neural networks on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters.
2 RELATED WORK
The maximum entropy principle (Jaynes, 1957) has a long history with deep connections to many areas of machine learning including unsupervised learning, supervised learning, and reinforcement learning. In supervised learning, we can search for the model with maximum entropy subject to constraints on empirical statistics, which naturally gives rise to maximum likelihood in log-linear models (see (Berger et al., 1996) for a review). Deterministic annealing Rose (1998) is a general approach for optimization that is widely applicable, avoids local minima, and can minimize discrete objectives, and it can be derived from the maximum entropy principle. Closely related to our work, Miller et al. (1996) apply deterministic annealing to train multilayer perceptrons, where an entropy based regularizer is introduced and slowly annealed. However, their focus is avoiding poor initialization and local minima, and while they find that deterministic annealing helps, the improvement diminishes quickly as the number of hidden units exceeds eight.
In reinforcement learning, encouraging the policy to have an output distribution with high entropy has been used to improve exploration (Williams & Peng, 1991). This prevents the policy from converging early and leads to improved performance (Mnih et al., 2016). Penalizing low entropy has also been used when combining reinforcement learning and supervised learning to train a neural speech recognition model to learn when to emit tokens (Luo et al., 2016). When learning to emit, the entropy of the emission policy was added to the training objective and was annealed throughout training. Indeed, in recent work on reward augmented maximum likelihood (Norouzi et al., 2016), this entropy augmented reinforcement learning objective played a direct role in linking maximum likelihood and reinforcement learning objectives.
Penalizing the entropy of a network’s output distribution has not been evaluated for large deep neural networks in supervised learning, but a closely related idea, label smoothing regularization, has been shown to improve generalization (Szegedy et al., 2015b). Label smoothing regularization estimates the marginalized effect of label-dropout during training, reducing overfitting by preventing a network from assigning full probability to each training example and maintaining a reasonable ratio between the logits of the incorrect classes. Simply adding label noise has also been shown to be effective at regularizing neural networks (Xie et al., 2016). Instead of smoothing the labels with a uniform distribution, as in label smoothing, we can smooth the labels with a teacher model (Hinton et al., 2015) or the model’s own distribution (Reed et al., 2014). Distillation and self-distillation both regularize a network by incorporating information about the ratios between incorrect classes.
Virtual adversarial training (VAT) (Miyato et al., 2015) is another promising smoothing regularizer. However, we did not compare to VAT because it has multiple hyperparameters and the approximated gradient of the local distributional smoothness can be computed with no more than three pairs of forward and back propagations, which is significantly more computation in grid-searching and training than the other approaches we compared to.
3 DIRECTLY PENALIZING CONFIDENCE
Confident predictions correspond to output distributions that have low entropy. A network is overconfident when it places all probability on a single class in the training set, which is often a symptom of overfitting (Szegedy et al., 2015b). The confidence penalty constitutes a regularization term that prevents these peaked distributions, leading to better generalization.
A neural network produces a conditional distribution pθ(y|x) over classes y given an input x through a softmax function. The entropy of this conditional distribution is given by
H(pθ(y|x)) = − ∑ i pθ(yi|x) log(pθ(yi|x)).
To penalize confident output distributions, we add the negative entropy to the negative log-likelihood during training
L(θ) = − ∑ log pθ(y|x)− βH(pθ(y|x)),
where β controls the strength of the confidence penalty. Notably, the gradient of the entropy term with respect to the logits is simple to compute. Denoting the ith logit by zi, then
∂H(pθ)
∂zi = pθ(yi|x) (− log pθ(yi|x)−H(pθ)) ,
which is the weighted deviation from the mean.
3.1 ANNEALING AND THRESHOLDING THE CONFIDENCE PENALTY
In reinforcement learning, penalizing low entropy distributions prevents a policy network from converging early and encourages exploration. However, in supervised learning, we typically want quick convergence, while preventing overfitting near the end of training, suggesting a confidence penalty that is weak at the beginning of training and strong near convergence. A simple way to achieve this is to anneal the confidence penalty.
Another way to strengthen the confidence penalty as training progresses is to only penalize output distributions when they are below a certain entropy threshold. We can achieve this by adding a hinge loss to the confidence penalty, leading to an objective of the form
L(θ) = − ∑ log pθ(y|x)− βmax(0,Γ−H(pθ(y|x)),
where Γ is the entropy threshold below which we begin applying the confidence penalty.
Initial experiments suggest that thresholding the confidence penalty leads to faster convergence at the cost of introducing an extra hyper-parameter. For the majority of our experiments, we were able to achieve comparable performance without using the thresholded version. For the sake of simplicity, we focus on the single hyper-parameter version in our experiments.
3.2 CONNECTION TO LABEL SMOOTHING
Label smoothing estimates the marginalized effect of label noise during training. When the prior label distribution is uniform, label smoothing is equivalent to adding the KL divergence between the uniform distribution u and the network’s predicted distribution pθ to the negative log-likelihood
L(θ) = − ∑ log pθ(y|x)−DKL(u‖pθ(y|x)).
By reversing the direction of the KL divergence, DKL(pθ(y|x)‖u), we recover the confidence penalty. This interpretation suggests further confidence regularizers that use alternative target distributions instead of the uniform distribution. We leave the exploration of these regularizers to future work.
4 EXPERIMENTS
We evaluated the confidence penalty and label smoothing on MNIST and CIFAR-10 for image classification, Penn Treebank for language modeling, WMT’14 English-to-German for machine translation, and TIMIT and WSJ for speech recognition. All models were implemented using TensorFlow (Abadi et al., 2016) and trained on NVIDIA Tesla K40 or K80 GPUs.
4.1 IMAGE CLASSIFICATION
4.1.1 MNIST
As a preliminary experiment, we evaluated the approaches on the standard MNIST digit recognition task. We used the standard split into 60k training images and 10k testing images. We use the last 10k images of the training set as a held-out validation set for hyper-parameter tuning and then retrained the models on the entire dataset with the best configuration.
We trained fully-connected, ReLu activation neural networks with 1024 units per layer and two hidden layers. Weights were initialized from a normal distribution with standard deviation 0.01. Models were optimized with stochastic gradient descent with a constant learning rate 0.05 (except for dropout where we set the learning rate to 0.001).
For label smoothing, we varied the smoothing parameter in the range [0.05, 0.1, 0.2, 0.3, 0.4, 0.5], and found 0.1 to work best for both methods. For the confidence penalty, we varied the weight values over [0.1, 0.3, 0.5, 1.0, 2.0, 4.0, 8.0] and found a confidence penalty weight of 1.0 to work best.
We also plotted the norm of the gradient as training progressed in Figure 2. We observed that label smoothing and confidence penalty had smaller gradient norms and converged more quickly than models regularized with dropout. If the output distributions is peaked on a misclassified example, the model receives a large gradient. This may explain why the regularized models have smaller gradient norms.
4.1.2 CIFAR-10
CIFAR-10 is an image classification dataset consisting of 32x32x3 RGB images of 10 classes. The dataset is split into 50k training images and 10k testing images. We use the last 5k images of the training set as a held-out validation set for hyper-parameter tuning, as is common practice.
For our experiments, we used a densely connected convolutional neural network, which represents the current state-of-the-art on CIFAR-10 (Huang et al., 2016a). We use the small configuration from (Huang et al., 2016a), which consists of 40-layers, with a growth rate of 12. All models were trained for 300 epochs, with a batch-size of 50 and a learning rate 0.1. The learning rate was reduced by a factor of 10 at 150 and 225 epochs. We present results for training without data-augmentation. We found that the confidence penalty did not lead to improved performance when training with data augmentation, however neither did other regularization techniques, including dropout.
For our final test scores, we trained on the entire training set. For label smoothing, we tried smoothing parameter values of [0.05, 0.1, 0.2, 0.3, 0.4, 0.5], and found 0.1 to work best. For the confidence penalty, we performed a grid search over confidence penalty weight values of [0.1, 0.25, 0.5, 1.0, 1.5] and found a confidence penalty weight of 0.1 to work best.
4.2 LANGUAGE MODELING
For language modeling, we found that confidence penalty significantly outperforms label noise and label smoothing. We performed word-level language modeling experiments using the Penn Treebank dataset (PTB) (Marcus et al., 1993). We used the hyper-parameter settings from the large configuration in (Zaremba et al., 2014). Briefly, we used a 2-layer, 1500-unit LSTM, with 65% dropout applied on all non-recurrent connections. We trained using stochastic gradient descent for 55 epochs, decaying the learning rate by 1.15 after 14 epochs, and clipped the norm of the gradients when they were larger than 10.
For label noise and label smoothing, we performed a grid search over noise and smoothing values of [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5]. For label noise, we found 0.1 to work best. For label smoothing, we found 0.1 to work best. For the confidence penalty, we performed a grid search over confidence penalty weight values of [0.1, 0.5, 1.0, 2.0, 3.0]. We found a confidence penalty weight of 2.0 to work best, which led to an improvement of 3.7 perplexity points over the baseline.
For reference, we also include results of the existing state-of-the-art models for the word-level language modeling task on PTB. Variational dropout (Gal, 2015) applies a fixed dropout mask (stochastic for each sample) at each time-step, instead of resampling at each time-step as in traditional dropout. Note, that we do not include the variational dropout results that use Monte Carlo (MC) model averaging, which achieves lower perplexity on the test set but requires 1000 model evaluations, which are then averaged. Recurrent highway networks (Zilly et al., 2016) currently represent the state-of-the-art performance on PTB.
4.3 MACHINE TRANSLATION
For machine translation, we evaluated the confidence penalty on the WMT’14 English-to-German translation task using Google’s production-level translation system Wu et al. (2016). The training set consists of 5M sentence pairs, and we used newstest2012 and newtests2013 for validation and newstest2014 for testing. We report tokenized BLEU scores as computed by the multi-bleu.perl script from the Moses translation machine translation package.
Our model was an 8-layer sequence-to-sequence model with attention (Bahdanau et al., 2014). The first encoder was a bidirectional LSTM, the remaining encoder and decoder layers were unidirectional LSTMs, and the attention network was a single layer feed-forward network. Each layer had 512 units (compared to 1024 in (Wu et al., 2016)). The model was trained using 12 replicas running concurrently with asynchronous updates. Dropout of 30% was applied as described in (Zaremba et al., 2014). Optimization used a mix of Adam and SGD with gradient clipping. Unlike (Wu et al., 2016), we did not use reinforcement learning to fine-tune our model. We used a beam size of 12 during decoding. For more details, see (Wu et al., 2016).
For label smoothing, we performed a grid search over values [0.05, 0.1, 0.2, 0.3, 0.4, 0.5] and found 0.1 to work best for both uniform and unigram smoothing. For the confidence penalty, we searched over values of [0.5, 2.5, 4.5] and found a value of 2.5 to work best . For machine translation, we found label smoothing slightly outperformed confidence penalty. When applied without dropout, both lead to an improvement of just over 1 BLEU point (dropout leads to an improvement of just over 2 BLEU points). However, when combined with dropout, the effect of both regularizers was diminished.
4.4 SPEECH RECOGNITION
4.4.1 TIMIT
In the TIMIT corpus, the training set consists of 3512 utterances, the validation set consists of 184 utterances and the test set consists of 192 utterances. All 61 phonemes were used during training and decoding, and during scoring, these 61 phonemes were reduced to 39 to compute the phoneme error rate (PER).
As our base model, we used a sequence-to-sequence model with attention. The encoder consisted of 3 bidirectional LSTM layers, the decoder consisted of a single unidirectional LSTM layer, and the attention network consisted of a single layer feed-forward network. All layers consisted of 256 units. Dropout of 15% was applied as described in Zaremba et al. (2014). We trained the model with asynchronous SGD with 5 replicas. We used a batch size of 32, a learning rate of 0.01, and momentum of 0.9. Gradients were clipped at 5.0. For more details, see Norouzi et al. (2016).
For label smoothing, we performed a grid search over values [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8] and found 0.2 to work best. For the confidence penalty, we performed a grid search over values of [0.125, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0] and found a value of 1.0 to work best. Label smoothing led to an absolute improvement over the dropout baseline of 1.6%, while the confidence penalty led to an absolute improvement of 1.2%.
4.4.2 WALL STREET JOURNAL
For the WSJ corpus we used attention-based sequence-to-sequence networks that directly predicted characters. We used the SI284 subset for training, DEV93 for validation, and EVAL92 for testing. We used 240-dimensional vectors consisting of 80-bin filterbank features augmented with their deltas and delta-deltas with per-speaker normalized mean and variances computed with Kaldi Povey et al. (2011). We did not use text-only data or separate language models during decoding.
Network architecture details were as follows. The encoder of the network consisted of 4 bidirectional LSTM layers each having 256 units, interleaved with 3 time-subsampling layers, configured to drop every second frame (Bahdanau et al., 2016; Chan et al., 2015). The decoder used a single LSTM layer with 256 units. The attention vectors were computed with a single layer feedforward network having 64 hidden units and the convolutional filters as described in Chorowski et al. (2015). Weights were initialized from a uniform distribution [−0.075, 0.075]. All models used weight decay of 10−6, additive Gaussian weight noise with standard deviation 0.075, applied after 20K steps, and were trained for 650K steps. We used the ADAM optimizer asynchronously over 8 GPUs. We used a learning rate of 10−3, which was reduced to 10−4 after 400K and 10−5 after 500K steps.
We tested three methods of increasing the entropy of outputs: the confidence penalty and two variants of label smoothing: uniform and unigram. All resulted in improved Word Error Rates (WER), however the unigram smoothing resulted in the greatest WER reduction, and we found it to be least sensitive to its hyperparameter (the smoothing value). Furthermore, uniform smoothing and confidence penalty required masking network outputs corresponding to tokens that never appeared as labels, such as the start-of-sequence token.
Table 6 compares the performance of the regularized networks with several recent results. We observe that the benefits of label smoothing (WER reduction from 14.2 to 11) improve over the recently proposed Latent Sequence Decompositions (LSD) method (Chan et al., 2016) which reduces the WER from 14.7 to 12.9 by extending the space of output tokens to dynamically chosen character n-grams.
5 CONCLUSION
Motivated by recent successes of output regularizers (Szegedy et al., 2015b; Xie et al., 2016), we conduct a systematic evaluation of two output regularizers: the confidence penalty and label smoothing. We show that this form of regularization, which has been shown to improve exploration in reinforcement learning, also acts as a strong regularizer in supervised learning. We find that both the confidence penalty and label smoothing improve a wide range of state-of-the-art models, without the need to modify hyper-parameters.
ACKNOWLEDGMENTS
We would like to thank Sergey Ioffe, Alex Alemi and Navdeep Jaitly for helpful discussions. We would also like to thank Prajit Ramachandran, Barret Zoph, Mohammad Norouzi, and Yonghui Wu for technical help with the various models used in our experiments. We thank the anonymous reviewers for insightful comments.
6 GRADIENT NORMS

The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be?
The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning.
The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point. In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy. Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective. But the paper doesn't indicate precisely what.
I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms?
The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing.
At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications.
Typo:
In related work: "Penalizing entropy" - you mean penalizing low entropy