Text classification in general is a well studied area. However, classifying short and noisy text remains challenging. Feature sparsity is a major issue. The quality of document representation here has a great impact on the classification accuracy. Existing methods represent text using bag-of-word model, with TFIDF or other weighting schemes. Recently word embedding and even document embedding are proposed to represent text. The purpose is to capture features at both word level and sentence level. However, the character level information are usually ignored. In this paper, we take word morphology and word semantic meaning into consideration, which are represented by character-aware embedding and word distributed embedding. By concatenating both character-level and word distributed embedding together and arranging words in order, a sentence representation matrix could be obtained. To overcome data sparsity problem of short text, sentence representation vector is then derived based on different views from sentence representation matrix. The various views contributes to the construction of an enriched sentence embedding. We employ a residual network on the sentence embedding to get a consistent and refined sentence representation. Evaluated on a few short text datasets, our model outperforms state-of-the-art models.
1 INTRODUCTION
For text classification, a popular feature representation method is bag-of-word. However, this representation has an intrinsic disadvantage that two separate features will be generated for two words with the same root or of different tenses. Lemmatization and stemming could be applied to partially address this problem, but may not always leads to correct results. For example, “meaningful” and “meaningless” would both be considered as “meaning” after applying lemmatization or stemming algorithms, while they are of opposite meanings. Thus, word morphology could also provide useful information in document understanding, particular in short text where the information redundancy is low.
For short text, an important issue is data sparsity, particularly when utilizing feature representation method like bag-of-word, regardless the weighting scheme. Therefore, various distributed word representation like Word2Vec (Mikolov et al., 2013) and document representation Doc2Vec (Le & Mikolov, 2014) have been proposed to address the problem. However, this kind of method miss the word morphology information and word combination information. To deal with these issues, we propose a model which could capture various kinds of features that could benefit classification task.
In this paper, we look deep into characters. We learn character representation and combine both character-level (Zhang et al., 2015) and word-level embedding to represent a word. Thus both morphology and semantic properties of the word are captured. As we know, not all the words in a sentence contribute the same when predicting the sentence’s label. Therefore, highlight the relatively pertinent information would give better chance of correct prediction. Attention mechanism (Mnih et al., 2014; Bahdanau et al., 2016) which focuses on specific part of input could help achieve this goal. The applications of attention mechanism are mostly on sequential model, while we employ
∗The two authors contribute the same for the work.
the idea of attention on a feed-forward network (Raffel & Ellis, 2015). By multiplying the weight assigned by attention mechanism to its corresponding word vector, a weighted feature matrix could be constructed by concatenating the sequence of word embeddings in a sentence.
Short text usually could not provide much useful information for class prediction. We try different views to extract as much information as possible to construct an enriched sentence representation vector. Specifically, to convert a sentence representation matrix to an enriched vector, we draw two types of features. The first one is based on word feature space and the other one is based on n-gram. However, not all the features contribute the same on sentence classification. Attention mechanism is applied to focus on the significant features. Since these features come from different views, we need a method to make the elements consistent. The residual network proposed in (He et al., 2015; 2016) achieve much better results on image classification task. In other words, the residual mechanism could construct better image representation. Therefore, we adopt residual network to refine the sentence representation vector. Once we obtain a good quality representation for the sentence, it will be delivered to a classifier.
2 RELATED WORK
There are many traditional machine learning methods for text classification and most of them could achieve quite good results on formal text datasets. Recently, many deep learning methods are proposed to solve the text classification task (Zhang et al., 2015; dos Santos & Gatti, 2014; Kim, 2014).
Deep convolutional neural network suggests benefits in image classification (Krizhevsky et al., 2012; Sermanet et al., 2013). Therefore, many research also try to apply it on text classification problem. Kim (2014) propose a model similar to Collobert et al. (2011) architecture. However, they employ two channels of word vectors. One is static throughout training and the other is fine-tuned via backpropagation. Various size of filters are conducted on both channel, and the results are concatenated together. Then max-pooling over time is taken to select the most significant feature among each filter. The selected features are concatenated as the sentence vector.
Similarly, Zhang et al. (2015) also employ the convolutional networks but add character-level information for text classification. They design two networks, one large and one small. Both of them have nine layers including six convolutional layer and three fully-connected layers. Between the three fully connected layers they insert two dropout for regularization. For both convolution and max-pooling layers, they employ 1-D version (Boureau et al., 2010). After convolution, they add the sum over all the results from one filter as the output. Specially, they claim 1-D max-pooling enable them to train a relatively deep network (Boureau et al.).
Besides applying models directly on testing datasets, more aspects are considered when extracting features. Character-level feature is adopted in many tasks besides Zhang et al. (2015) and most of them achieve quite good performance. dos Santos & Zadrozny (2014) take word morphology and shape into consideration which have been ignored for part-of-speech tagging task. They suggest the intra-word information is extremely useful when dealing with morphologically rich languages. They adopt neural network model to learn the character-level representation which is further delivered to help word embedding learning. Kim et al. (2016) construct neural language model by analysis of word representation obtained from character composition. Results suggest the model could be encode semantic and orthographic information from character level.
Attention model is also utilized in our model, which is used to assign weights for each parts of components. Usually, attention model is used in sequential model (Rocktäschel et al., 2015; Mnih et al., 2014; Bahdanau et al., 2016; Kadlec et al., 2016). The attention mechanism includes sensor, internal state, actions, and reward. At each time, the sensor will capture a glimpse network which only focus on a small part of the network. Internal state will summarize the extracted information. Actions decides the location for the next step and reward suggests the benefit when taking the action. In our condition, we adopt a simplified attention network as (Raffel & Ellis, 2015; 2016). We do not need to guess the next step location and just give a weight on each components which indicates the significance of the element.
Residual network (He et al., 2015; 2016; Chen et al., 2016) is known to be able to make neural network deeper and relieve degradation problem at the same time. And residual network in (He et al., 2015) outperforms the state-of-the-art models on image recognition. He et al. (2016) introduces
how to make the residual block more efficient on image classification. Similarly, for short text classification problem, the quality of sentence representation is also quite important for the final result. Thus, we try to adopt the residual block as in (He et al., 2015; 2016) to refine the sentence vector.
3 CHARACTER-AWARE ATTENTION RESIDUAL NETWORK
In this paper, we propose a character-aware attention residual network to generate sentence representation. Figure 1 illustrates the model. For each word, the word representation vector is constructed by concatenating both character-level embedding and word semantic embedding. Thus a sentence is represented by a matrix. Then two types of features are extracted from the sentence matrix to construct the enriched sentence representation vector for short text. However, not all the features contribute the same for classification. Attention mechanism is employed to target on pertinent parts. To make features extracted from different views consistent, a residual network is adopt to refine the sentence representation vector. Thus, an enriched sentence vector is obtained to do text classification.
3.1 WORD REPRESENTATION CONSTRUCTION
Let C be the vocabulary of characters, and E ∈ Rdc×|C| is the character embedding matrix, where dc is the dimensionality of character embedding. Given a word, which is composed of a sequence of characters [c1, c2, ..., cnc ], its corresponding character-level embedding matrix would be E
w ∈ Rdc×nc . Herein, Ew·i = E · vi (1) where vi is a binary column vector with 1 only at the ci-th place and 0 for other positions. Here, we fix the word length dc and take zero-padding when necessary.
For each of such matrix Ew, a convolution operation (Le Cun et al., 1990) with m filters (i.e., kernels) P ∈ Rdc×k is applied on Ew, and a set of feature maps could be obtained. Instead of adopting max-pooling over time (Collobert et al., 2011), we adopt max-pooling over filters operation to capture local information of words as shown in Figure 1. Similar operation is adopted in (Shen et al., 2014). That is we get the max feature value over results of m filters at the same window position, which depicts the most significant feature over the k characters. Thus, a vector qc for the word which captures the character-level information is constructed.
Note that embedding vector qc could only capture the word morphological features, while it can not reflect word semantic and syntactic characteristics. Therefore, we concatenate the distributed word representative vector qw (i.e., Word2Vec) (Mikolov et al., 2013) to qc as the word’s final representation q ∈ R(dc+dw), where dw is the dimensionality of Word2Vec. Given a sentence, which consists of a sequence of words [w1, w2, ..., wnw ], its representation matrix is Q ∈ R(dc+dw)×nw .
3.2 SENTENCE REPRESENTATION VECTOR CONSTRUCTION
To overcome the lack of information issue for short text, we explore various kinds of useful information from limited context. From higher level, we adopt two types of features as shown in
Figure 2 (i.e., type 1 feature and type 2 feature). They capture different views of information for the short text, which could be considered as results from horizontal view and vertical view on sentence representation matrix Q separately.
Type 1 feature takes word’s feature space (i.e., horizontal view on Q) into consideration. The feature space is the composition of both character-level embedding and word semantic embedding. Each word is a point in the feature space. We formulate the summation over all words appearing in the sentence as the sentence’s representation, inspired by (Zhang et al., 2015). In fact, not all the words in a sentence contribute the same for prediction. Therefore, we want to highlight the significant words and this is realized by weighting the word’s representation features. To assign the weights, we employ attention mechanism, and multiply the weight to the word feature vector as Equation 2. Specifically, we follow Raffel & Ellis (2015) and Bahdanau et al. (2014) as shown in Figure 3(a). For each word representation vector qi, we apply a Tanh function on the linear transformation of qi as g(qi) = Tanh(Wqhqi + bqh), where Wth ∈ R1×(dc+dw), bth ∈ R. Then a softmax function on g(qi) is used to assign a weight si for each qi, which indicates the significance of word i in the sentence.
si = exp(g(qi))∑j=1 nw exp(g(qj)) , q̃i = siqi. (2)
As a result, we can get a weighted sentence representation matrix Q̃ ∈ R(dc+dw)×nw . Then we employ an average over words in the sentence at the same feature position and obtain a sentence representation vector r0.
Type 2 feature models the word level features (i.e., vertical view on Q). As we know, sometimes continuous words combination is meaningful and pertinent for sentence classification. To capture n-gram information, we apply convolution operation on Q, which is followed by a max-pooling over time. We adopt several different kernel sizes to model various n-grams. Different n-grams contribute differently. The attention mechanism is utilized again on the vectors of n-gram representations, and the resulting weights indicate their significance. We get the weighted feature vectors r1, r2, r3. Concatenating r0, r1, r2, r3, the complete sentence vector v is constructed.
3.3 RESIDUAL NETWORK FOR REFINING SENTENCE REPRESENTATION
The residual learning (He et al., 2015; 2016) is reported to outperform state-of-the-art models in image classification task and object detection task. This suggests residual learning could help to capture and refine the embedding. To make the features of sentence vector v from different views consistent, we employ residual learning to v.
Let the desired mapping as H(v), instead of making each layer directly optimize H(v), residual learning (He et al., 2015) turns to fit the residual function:
F(v) := H(v)− v. (3)
Thus, the original target mapping becomes:
y = F(v) + v. (4)
Both residual function F(v) and the added input form v are flexible. In our model, we construct the building block by two fully connected layers connected by a ReLU (Nair & Hinton, 2010) operation as shown in Figure 3(b). Meanwhile, the identity mapping is adopted by performing a shortcut connection and element-wise addition:
v′ = F(v, G) + v (5)
where v′ is the refined sentence vector, G is the weight matrix to be learned. After getting the sentence embedding v′ from the building block, it is further delivered to a softmax classifier for text classification.
4 EXPERIMENT
4.1 DATASETS
We adopt testing datasets from different sources. There are three datasets, including Tweets, Question, AG news. All of them are relatively short.
Tweets are typical short text with only 140 characters limitation. We crawl the tweets from Twitter with a set of keywords, which is specifically about some products. We label them as positive, negative, neutral, question and spam.
Question dataset is a small dataset. The content is short questions, and the labels are question types.
AG News dataset is from (Zhang et al., 2015). The reason we choose this is because the length of text is much shorter than others. The news here only contains the title and description fields.
4.2 EXPERIMENT SETTINGS
In this paper, we take 128 ASCII characters as character set, by which most of the testing documents are composite. We define word length nc as 20 and character embedding length dc as 100. If a word with characters less than 20, zero padding is taken. If the length is larger than 20, just take the first 20 characters. We train the word distributed embedding using training data and the feature dimension is 300. We take sentence length as 20, which is enough to cover most of crucial words. We add 5 residual blocks to refine the sentence vector.
4.3 BASELINE MODEL
We select both traditional models and deep learning models on classification as baselines.
TF-SVM is the bag-of-word feature weighted by counting the term frequency in a sentence. Then deliver the feature matrix to a SVM classifier.
TFIDF-SVM is taken as traditional baseline model. Since SVM classifier is robust and state-ofthe-art traditional classifier, and TFIDF usually assign good weights for bag-of-words in documents even for tough inputs. So this is a competitive baseline model.
Lg. Conv, Sm. Conv are proposed in (Zhang et al., 2015) which also consider character-level embedding, and they concatenate all the characters’ embeddings in a sentence in order as sentence’s representation matrix. For fair comparison, we do not include thesaurus to help clear documents.
4.4 COMPARISON RESULTS
Table 3 shows the comparison results on the testing datasets. As we can see, the proposed model could outperform baseline models on Tweets and Question datasets. For AG news dataset, our method could give comparable results as the best baseline model, TFIDF-SVM. The TFIDF-SVM model can achieve relatively better results than others. However, both Lg. Conv and Sm. Conv do not perform well on Tweets dataset. This may because these two models are relatively deep network with several down sampling operations (i.e., max-pooling) and this dramatically decreases the short text representation. And short text does not contain much information. Thus Lg. Conv and Sm. Conv could not give good results. The TF-SVM model also does not perform well on Tweets dataset. This may because the tweet text is too short and term frequencies are mostly 1 which is not enough to provide information on classification. Similar to result of CAR-1 on Tweets data. When removing type 1 feature, the performance drops dramatically. However, for other datasets, in which the document length is longer and the content is relatively formal, removing type 1 feature does not influence the performance that much. Hence, these results suggest the word character-level feature and semantic feature (i.e., type 1 feature) are rather important for short, free-style text. On the other hand, by adding type 2 features can also improve the performance according to results of CAR2. Consequently, when dealing with short text, either formal or informal, including character-level feature, word-semantic feature and n-gram feature would benefit the performance.
Another comparison is adding the residual network or not. As we can see from Table 3, residual network could refine the vector representation. When removing residual block, performances on three datasets all decrease. In particular, the improvement for shorter and noisy text (Tweets dataset) is more than those relatively longer and formatted documents. Thus, for short noisy text classification problem, one adopts residual building block would improve the performance.
5 CONCLUSION
We propose a character-aware attention residual network for short text classification. We construct the sentence representation vector by two kinds of features. The first is focusing on feature space, which include both character-level characteristics and semantic characteristics. The other is n-gram features. To make them consistent, the residual network helps refine the vector representation. Experiment results suggest both extracted features and the residual network helps on short text clas-
sification. Our proposed method could outperform the state-of-the-art traditional models and deep learning models.

This paper proposes a new neural network model for sentence representation. This new model is inspired by the success of residual network in Computer Vision and some observation of word morphology in Natural Language Processing. Although this paper shows that this new model could give the best results on several datasets, it lacks a strong evidence/intuition/motivation to support the network architecture.
To be specific:
- I was confused by the contribution of this paper: character-aware word embedding or residual network or both?
- The claim of using residual network in section 3.3 seems pretty thin, since it ignores some fundamental difference between image representation and sentence representation. Even though the results show that adding residual network could help, I was still not be convinced. Is there any explanation about what is captured in the residual component from the perspective of sentence modeling?
- This paper combines several components in the classification framework, including character-aware model for word embedding, residual network and attention weight in Type 1 feature. I would like to see the contribution from each of them to the final performance, while in Table 3 I only saw one of them. Is it possible to add more results on the ablation test?
- In equation (5), what is the meaning of $i$ in $G_i$?
- The citation format is impropriate


ATTENTIVE RECURRENT COMPARATORS
Attentive Recurrent Comparators (ARCs) are a novel class of neural networks built with attention and recurrence that learn to estimate the similarity of a set of objects by cycling through them and making observations. The observations made in one object are conditioned on the observations made in all the other objects. This allows ARCs to learn to focus on the salient aspects needed to ascertain similarity. Our simplistic model that does not use any convolutions performs comparably to Deep Convolutional Siamese Networks on various visual tasks. However using ARCs and convolutional feature extractors in conjunction produces a model that is significantly better than any other method and has superior generalization capabilities. On the Omniglot dataset, ARC based models achieve an error rate of 1.5% in the One-Shot classification task a 2-3x reduction compared to the previous best models. This is also the first Deep Learning model to outperform humans (4.5%) and surpass the state of the art accuracy set by the highly specialized Hierarchical Bayesian Program Learning (HBPL) system (3.3%).
1 INTRODUCTION
Advancing Deep Learning systems to solve Artificial Intelligence tasks requires that models be capable of performing continual meta-learning[ (Lake et al., 2016), (Schaul & Schmidhuber, 2010)]. But top-down hierarchical designs of models (Santoro et al., 2016) to perform such tasks are not very successful on real world data and there are many reasons for this. First, most datasets are generally not designed with such higher order tasks in mind, thus researchers either work with synthetic data or fabricate higher level tasks based on traditional datasets - both of which constrain their utility. Second, hierarchical or meta models suffer from reduced supervision during training due to their inherent design. Third, with our experiments we found that the foundational architectures like Memory Augmented Neural Networks are still in their infancy and not ripe enough to be utilized in complex hierarchical systems. Therefore, in this paper, we present an alternative way of bridging this gap by building models in a bottom-up fashion. Comparing two or more inputs and estimating their similarity is a primal task using which more sophisticated models can be designed - an idea that has been well exploited in traditional Machine Learning for long (Bellet et al., 2013). Using the modern developments of attention mechanisms and by combining it with recurrent neural networks, we first built better comparators called Attentive Recurrent Comparators (ARCs) 1. Using ARCs as a foundational element, we were then able to build more complex models and achieve qualitatively better results on tasks like one-shot learning. Thus, this work is proof of concept for the bottom-up design approach that can be applied to almost any dataset.
When a person is asked to compare two objects and estimate their similarity, the person does so by repeatedly looking back and forth between the two objects. With each glimpse of an object, a specific observation is made. These observations made in both objects are then cumulatively used to come to a conclusion about their similarity. A crucial characteristic of this process is that new observations are made conditioned on the previous context that has been investigated so far by the observer. The observation and it’s contextual location are based on intermediate deductions. These intermediate deductions are themselves based on the observations made so far in the two objects.
∗Other Affiliation: Student at R V College of Engineering, Bengaluru 1Code available at https://github.com/pranv/ARC
A series of such guided observations and the entailing inferences are accumulated and finally the judgement on similarity is made.
In stark contrast to this, current similarity estimating systems in Deep Learning are analogues of the Siamese similarity learning system (Bromley et al., 1993). In this system, a fixed set of features is detected in both the objects. Detection of features is independent of the features present in the other object. The two objects are compared based on the mutual agreement in the detected features. More concretely, comparison between two objects in this system consists of measuring the distance between their vector embeddings. A neural network defines the mapping from the object to the corresponding embedding vector in target space. This neural network is trained to extract the most salient features from the object for the specific task in hand.
There is major underlying difference between the human approach discussed above and the siamese approach to the problem. In the human way, the information from the two objects is fused from the very beginning and this combined information primes the subsequent steps in comparison. There are multiple lookups on each of the objects and each of these lookups are conditioned on the observations of both the objects so far. In the siamese way, when the embeddings in the target space are compared the information fuses mostly at an abstract level and only in the last stage.
We were interested to see the utility of the human way of comparing objects. For this, we used the modern tools of attention and recurrence to build an end-to-end differentiable model that can learn to compare objects called Attentive Recurrent Comparators (ARCs). ARCs judge similarity of objects similar to the way people do as discussed above.
We tested ARCs across many visual tasks and compared it against strong baselines of prevalent methods. ARCs which did not use any convolutions showed superior performance compared to Deep Convolutional Siamese Neural Networks on challenging tasks. Though Dense ARCs are as capable as ConvNets, a combination of both ARCs and convolutions produces more superior models (hereafter referred to as ConvARCs), capable of better generalization and performance. In the task of estimating the similarity of two characters from the Omniglot dataset for example, ARCs and Deep Convnets both achieve about 93.4% accuracy, whereas ConvARCs achieve 96.10% accuracy.
Further, as discussed above, similarity estimation is a generic and a primal task in many other higherlevel cognitive tasks. Evaluating our model on these higher-level tasks also lets us explore the generalisation capacity of ARCs. In this work, we study the performance of models designed to perform One Shot Learning with ARCs as building blocks. On the Omniglot one-shot classification task, our model achieved 98.5% accuracy significantly surpassing the current state of the art set by Deep Learning methods or other systems.
Fundamentally, the performance of ARCs shows the value of early fusion of information across the entire context of the task. Further, it also strengthens the view that attention and recurrence together can be as good as convolutions in some cases.
2 ATTENTIVE RECURRENT COMPARATORS
The ARC model can be directly derived by distilling the vital aspects from the human way discussed in Section 1. In the following paragraphs we describe the ARC model for the binary image case - where there are two images whose similarity has to be judged. It is trivial to generalise it to more objects or other modalities. See Figure 1 for a visual depiction of the model.
The model operates on given two images over the span of an episode. The images are given at the beginning of the episode and the ARC is expected to emit a token of similarity at the end of this episode. Given two images {xa, xb}, the model repeatedly cycles through the both, attending to only one image at one time step. Thus the sequence of presentations is xa, xb, xa, xb, ... and so on for a finite number of presentations of each image. An episode is nothing more than a collection of time-steps, with an action being taken in each time-step.
For time step t the input image presented is given by:
It ←− xa if t % 2 is 0 else xb
The model functionally consists of a recurrent core and an attention mechanism. During the span of the episode, the model iteratively focuses its attention on the current input. At each time step of the episode, the model attends to only one input, but over the course of many time steps it would have observed many aspects of all the inputs. The observations are made by the model at each time step by directing its attention to a region of interest in each input. Since the core of the model is a recurrent neural network, this round robin like cyclic presentation of inputs allows for early fusion of information from all the inputs. This makes the model aware of the context in which it is operating. Consequently, this provides feedback to the attention mechanism to attend on the relevant and crucial parts of each sample considering the context of all the inputs and observations made so far.
If there are n inputs and we allow for g glimpses of each input, then the episode length L is ng. The hidden state of the RNN controller at the final time step hL can be then used for subsequent processing.
The attention mechanism focuses on a specific region of the image It to get the glimpse Gt.
Gt ←− attend(It,Ωt) Ωt = Wght−1
attend(.) is the attention mechanism described in the sub section below, that acts on image It. Ωt are the attention glimpse parameters which specify the location and size of the attention window. At each step, we use the previous hidden state of the RNN core ht−1 to compute Ωt. Wg is the projection matrix that maps the hidden state to the required number of attention parameters.
Next, both the glimpse and previous hidden state are combined to form the next hidden state.
ht ←− RNN(Gt, ht−1)
RNN(.) is the update function for the recurrent core being used. This could be simple RNN or an LSTM.
The above 4 equations describe the Binary ARC. We arrived at the iterative cycling of input paradigm after trying out many approaches to attend to multiple images at once. Iterative cycling turned out to
be more computationally efficient, scalable and statistically more consistent than other approaches we tested. Note that It for some t alternates between xa and xb, while the rest of the equations are exactly the same for all time steps.
2.1 ATTENTION MECHANISM
The attention mechanism is based on DRAW (Gregor et al., 2015), zoomable and differentiable. The attention window is defined by an N ×N 2D grid of Cauchy kernels. We found that the heavy tail of the Cauchy curve to aids in alleviating some of the vanishing gradient issues and it sped up training.
The grid’s location and size is defined based on the glimpse parameters. The N ×N grid of kernels is placed at (x, y) on the S × S image, with the central Cauchy kernel being located at (x, y). The distance between two Cauchy kernals either in the vertical or horizontal direction is δ. In other words, the elemental square of the 2D grid is δ×δ in size. The glimpse parameter set Ωt is unpacked to get Ωt → (x̂, ŷ, δ̂). x, y and δ are computed from x̂, ŷ and,δ̂ using the following transforms:
x = (S − 1) (x̂+ 1) 2
y = (S − 1) (ŷ + 1) 2
δ = S
N (1− |δ̂|) γ = e1−2|δ̂|
The location of a ith row, jth column’s Cauchy kernel in terms of the pixel coordinates of the image is given by:
µiX = x+ (i− (N + 1)/2) δ and µ j Y = y + (j − (N + 1)/2) δ
The horizontal and vertical filterbank matrices are then calculated as:
FX [i, a] = 1
ZX
{ πγ [ 1 + ( a− µiX γ )2]}−1 and FY [j, b] = 1
ZY
πγ 1 +(b− µjY γ )2 −1
ZX and ZY are normalization constants such that they make ΣaFX [i, a] = 1 and ΣbFX [j, b] = 1
Final result of the attending to the image is given by:
attend(It,Ωt) = FY ItF T X
attend thus gets an N ×N patch of the image, which is flattened and used in the model.
2.2 USE OF CONVOLUTIONS
As seen in the experimental section below, while simple attention over raw images performs as well as Deep ResNets, we found large improvements by using Convolutional feature extractors. Applying several layers of Convolution produces a 3D solid of activations (or a stack of 2D feature maps). Attention over this corresponds to applying the same 2D attention over the entire depth of the 3D feature map and outputting the flattened glimpse.
3 PRACTICAL EXPERIMENTS AND ANALYSIS
Understanding the empirical functioning of an ARC and identifying factors affecting its performance requires both qualitative and quantitative studies. Qualitative analysis tells us what the model is doing when it is comparing 2 images and how this relates to human ways of comparison. Quantitative analysis shows the variations in performance when certain aspects of the model are changed and thus provide an estimate of their importance. For the analysis presented below, we use the simple ARC model (without convolutions) described in Section 2 above trained for the verification task on the Omniglot dataset. Data samples in the Omniglot dataset have an understandable structure with characters being composed of simple strokes drawn on a clean canvas. The dataset is also very diverse, which allows us to study various characteristics of our model under a wide range of conditions. Since our main result in the paper is also on the Omniglot dataset (Sections 4 and 5), we train
the ARC on this same dataset for this analysis to get an insight into the type of performance gains brought about by this architecture.
The verification task is a binary classification problem wherein the model is trained to predict whether the 2 drawings of characters provided belong to the same character or not (see Section 4 for more details). The final hidden state of the RNN Controller hL is given to a single logistic neuron to estimate the probability of similarity. The whole setup is trained end to end with back-propagation and SGD. The particular model under consideration had an LSTM controller (Hochreiter & Schmidhuber, 1997) with forget gates (Gers et al., 2000). The number of glimpses per image was fixed to 8, thus the total number of recurrent steps being 16. 32 × 32 greyscale images of characters were used and the attention glimpse resolution is 4× 4.
3.1 QUALITATIVE ANALYSIS
The following inferences were made after studying several cases of ARC’s operation (see Figure 2 for an example):
1. The observations in one image are definitely being conditioned on the observations in the other image. This can be seen in figures 2a and 2b.
2. The ARC seems to have learnt a fairly regular left to right parsing strategy, during which the attention window gradually reduces in size. This is quite similar to strategies found in other sequential attentive models like DRAW (Gregor et al., 2015).
3. Deviation from such regular ordered parsing occurs if model finds some interesting feature in either character. This results in attention being fixated to that particular region of the character for a few subsequent glimpses.
4. There is no strict coordination or correspondence chronologically between the attended regions of the two images. While instances of ARC focussing on the same aspect/stroke of two characters were common, there were plenty more instances wherein the ARC attended to different aspects/strokes in each image during an interval. We hypothesise that the RNN
controller could be utilizing turns of glimpsing at an image to observe some other aspects which are not of immediate consequence.
5. We also frequently encountered cases wherein the attention window, after parsing as described in point 2, would end up focusing on some blank, stroke-less region, as if it had stopped looking at the sample. We hypothesize that the model is preferring to utilize its recurrent transitions and not to be disturbed by any input stimuli.
3.2 QUANTITATIVE ANALYSIS
We performed a simple yet very insightful ablation study to understand ARC’s dynamics. ARC accumulates information about both the input images by a series of attentive observations. We trained 8 separate binary classifiers to classify images as being similar or not based on hidden states of the LSTM controller at every even time step correspondingly . The performance of these binary classifiers are correlated with the information contained in the hidden states. The performance of these classifiers is reported in Table 1. Since the ARC has an attention window of only 4× 4 pixels, it can barely see anything in the first time step, where its attention is spread throughout the whole image. With more glimpses, finer observations bring in more precise information into the ARC and the recurrent transitions make use of this knowledge, leading to higher accuracies. We also used the 8 binary classifiers to study how models confidence grows with more glimpses and one such good example is provided in Figure 3.
4 SIMILARITY LEARNING
Verification is a generic and common task in Machine Learning. The verification task essentially requires models that can predict whether the two inputs are the same or different, for some notion of same (such as unique facial identity, objects of same class etc.,). Specifically, here we restrict ourselves to the task of estimating the similarity of the given pair of images. When given two images the models are required to output a single logistic value, which is expected to be 1 for very similar inputs and 0 for very dissimilar inputs. We compare our ARC model with several baselines and report performance on two challenging datasets.
4.1 DATASETS
4.1.1 OMNIGLOT
The dataset is thoroughly detailed in the next section which is on one shot classification on this dataset. And this task acts as a precursor to the more sophisticated next task. We use 32×32 images and similar/dissimilar pairs of character drawings are randomly chosen only within alphabet to make the task more challenging. Out of the 50 alphabets provided in the dataset, 30 were used for training, 10 for validation and the last 10 for testing.
4.2 BASELINES AND IMPLEMENTATION DETAILS
We consider strong convolutional baselines, which have been shown time and again to excel at such visual tasks. We particularly use Wide Resnets (WRNs) (Zagoruyko & Komodakis, 2016) which are the current state of the art models in image classification. Independent nets were tuned for each dataset. Hyper parameters were set for reasonable values for all our ARC models and no hyperparameter tuning of any kind was employed. For the Omniglot dataset, we also include the result from (Koch et al.) We used moderate data augmentation consisting of translation, flipping, rotation and shearing, which we found to be critical for training ARC models.
4.3 RESULTS
The results are in Table 1 for Omniglot respectively. Our simple ARC model without using any convolutional layers obtains a performance that matches a AlexNet style 6 layer Deep Convnet with millions of parameters. Using convolutional feature extractors, ARCs outperform the Wide ResNet based Siamese ConvNet baselines, even the ones containing an order of magnitude more parameters.
5 OMNIGLOT ONE SHOT CLASSIFICATION
One shot learning requires Machine Learning models to be at the apotheosis of data efficiency. In case of classification, only a single example of each individual class is given and the model is
expected to generalise to new samples. A classic example is of a human kid learning about the animal giraffe (Vinyals et al., 2016). The kid does not need to see thousands of images of a Giraffe to learn to detect it. Rather, just from a single example, the kid can not only recognize it at a future point, but going further, she can also speculate on its other characteristics. While humans excel at this task, current Deep Learning systems are at the opposite end of the spectrum, where they are trained on millions of samples to achieve the kind of results that they are well known for. With ARCs we have developed a generic method to comparing objects. We have also shown that our model generalizes extremely well. So we decided to test ARC on the challenging Omniglot dataset.
Omniglot is a dataset by Lake et al. (2015) that specially designed to compare and contrast the learning abilities of humans and machines. The dataset contains handwritten characters of 50 of the world’s languages/alphabets. Though there are 1623 characters, there are only 20 samples for each which is drawn by 20 individuals. So this is in a diagonally opposite position when compared to MNIST or ImageNet. One Shot Classification on this dataset is very challenging one as most Deep Learning systems do not work well in such extreme conditions. Lake et al. (2015) developed a dedicated system for such rapid knowledge acquisition called Hierarchical Bayesian Programming Learning, which surpasses human performance and is the current state of the art of all methods.
5.1 TASK
The dataset is divided into a background set and an evaluation set. Background set contains 30 alphabets (964 characters) and only this set should be used to perform all learning (e.g. hyper-parameter inference or feature learning). The remaining 20 alphabets are for pure evaluation purposes only. Each character is a 105× 105 image. A one shot classification task episode is as follows: from a randomly chosen alphabet, 20 characters are chosen which becomes the support set. One character among these 20 becomes the test character. 2 drawers are chosen, one each for the support set and the test character. The task is to match the test drawing to the correct character’s drawing in the support set. Assigning an image to one of the 20 characters given results in a 20-way, 1-shot classification task.
5.2 MODELS
5.2.1 NAIVE ARC MODEL
This is a trivial extension of ARC for used for verification to this task. The test image from the first set is chosen and compared against all 20 images from the second set. It is matched to the character with maximum similarity. This is done for 20 times for each character in the first set.
5.2.2 FULL CONTEXT ARC
Our whole hypothesis in this work has been about the value of providing the full context to the model. And we have shown to that models which are aware of the context of operation are better than the ones that aren’t. While Naive ARC model is simple and efficient, it does not incorporate the whole context in which our model is expected to make the decision of similarity. When the character is being compared to 20 other characters from the support set, the comparisons are all independently done. That is, the model is not aware available options for matching, so it assigns the similarity score to each pair independently.
It is highly desirable to have a 20-way ARC, where each observation is conditioned on the all images. Unfortunately, such a model is not practical. The recurrent controller has memory limitations in its state. Scaling up the memory incurs a huge parameter burden. So instead, we use a hierarchical setup, which decomposes the comparisons to be at two levels - first local pairwise comparison and a second global comparison. We found that this model reduces the information that has to be crammed in the controller state, while still providing sufficient context.
As with the Naive method, we compare one image from set A with one from set B in pairs. But instead of emitting a similarity score immediately, we collect the comparison embeddings of each comparison. Comparison embeddings are the final hidden state of the controller when the test image T is being compared to image Sj from the support set B: ej = hLT,Sj . These embeddings are further processed by a Bi-Directional LSTM layer. This merges the information from all comparisons, thus
providing the necessary context before score emission. This is also the method used in Matching Networks (Vinyals et al., 2016).
cj = [ −−−−→ LSTM(ej); ←−−−− LSTM(ej) ] ∀j ∈ [1, 20]
Each embedding is mapped to a single score sj = f(cj), where f(.) is an affine transform followed by a non-linearity. The final output is the normalized similarity with respect to all similarity scores.
pj = softmax(sj) ∀j ∈ [1, 20]
This whole process is to make sure that we adhere to the fundamental principle of deep learning, which is to optimise objectives that directly reflect the task. The normalisation allows for the expression of relative similarity rather than absolute similarity.
5.3 BASELINES AND OTHER METHODS
We compare the two models discussed above with other methods in literature: starting from the simplest baseline of k-Nearest Neighbours to the latest meta-learning methods. The training and evaluation practises are non consistent.
5.3.1 ACROSS ALPHABETS
Many papers recently, like Matching Networks Vinyals et al. (2016) and MANNs Santoro et al. (2016) have used 1200 chars for background set (instead of 964 specified by Lake et al. (2015)). The remaining 423 characters are used for testing. Most importantly, the characters sampled for both training and evaluation are across all the alphabets in the training set.
5.3.2 WITHIN ALPHABETS
This corresponds to standard Omniglot setting where characters are sampled within an alphabet and only the 30 background characters are used for training and validation.
The across alphabet task is much more simpler as it is easy to distinguish characters belonging to different languages, compared to distinguishing characters belonging to the same language. Further, across alphabet methods use a lot more data which is a particularly advantageous entity for Deep Learning Methods.
There are large variations in the resolution of the images used as well. The Deep Siamese Network of Koch et al. uses 105x105 images and thus not comparable to out model, but we include it as it is the current best result using deep neural nets. The performance of MANNs in this standard setup is interpreted from the graph in the paper as the authors did not report it. It should also be noted that HBPL incorporates human stroke data into the model. Lake et al estimate the human performance to be at 95.5%.
5.4 RESULTS
Results are presented in Table 2. Our ARC models outperform all previous methods according to both of the testing protocols and establish the corresponding state of the art results.
6 RELATED WORK
Deep Neural Networks (Schmidhuber, 2015) (LeCun et al., 2015) are very complex parametrised functions which can be adapted to have the required behaviour by specifying a suitable objective function. Our overall model is a simple combination of the attention mechanism and recurrent neural networks (RNNs). We test our model by analysing its performance in similarity learning. We also test its generalisation ability by using it in a model built for the challenging task of one shot classification on hand-written character symbols.
6.1 ATTENTION
It is known that attention brings in selectivity in processing information while reducing the processing load (Desimone & Duncan, 1995). Attention and (Recurrent) Neural Networks were combined in Schmidhuber & Huber (1991) to learn fovea trajectories. Later attention was used in conjunction with RBMs to learn what and where to attend in Larochelle & Hinton (2010) and in Denil et al. (2012). Hard Attention mechanism based on Reinforcement Learning was used in Mnih et al. (2014) and further extended to multiple objects in Ba et al. (2014); both of these models showed that the computation required at inference is significantly less compared to highly parallel Convolutional Networks, while still achieving good performance. A soft or differentiable attention mechanisms have been used in Graves (2013). A specialised form of location based soft attention mechanism, well suited for 2D images was developed for the DRAW architecture (Gregor et al., 2015), and this forms the basis of our attention mechanism in ARC.
6.2 SIMILARITY LEARNING
A survey of the methods and importance of measuring similarity of samples in Machine Learning is presented in Bellet et al. (2013). With respect to deep learning methods, the most popular architecture family is that of Siamese Networks (Bromley et al., 1993). The energy based derivation is presented in Chopra et al. (2005) and since then they have been used across wide range of modalities - in vision (Zagoruyko & Komodakis, 2015) (Bertinetto et al., 2016), for face recognition and verification (Taigman et al., 2014) and in Natural Language Processing (Lu & Li, 2013) (Hu et al., 2014). Recently Triplet Losses (Hoffer & Ailon, 2015) are being used to achieve higher performance and is similar to our Ternary ARC model at an abstract level.
6.3 ONE SHOT LEARNING
A bayesian framework for one shot visual recognition was presented in Fe-Fei et al. (2003). Lake et al. (2015) extensively study One Shot Learning and present a novel probabilistic framework called Hierarchical Bayesian Program Learning (HBPL) for rapid learning. They have also released the Omniglot dataset, which has become a testing ground for One Shot learning techniques. Recently, many Deep Learning methods have been developed to do one shot learning: Koch et al. use Deep Convolutional Siamese Networks for performing one shot classification. Matching Networks (Vinyals et al., 2016) and Memory Augmented Neural Networks (Santoro et al., 2016) are other approaches to perform continual or meta learning in the low data regime. All the models except the HBPL have inferior one shot classification performance compared to humans on the Omniglot Dataset.
7 CONCLUSION AND FUTURE WORK
We presented a model that uses attention and recurrence to cycle through a set images repeatedly and estimate their similarity. We showed that this model is not only viable but also much better than the siamese neural networks in wide use today in terms of performance and generalization. Our main
result is in the task of One Shot classification on the Omniglot dataset, where we achieved state of the art performance surpassing HBPL’s and human performance.
One potential downside of this model is that due to sequential execution of the recurrent core and by the very design of the model, it might be more computationally expensive than a distance metric method. But we believe that advancing hardware speeds, such costs will be outweighed by the benefits of ARCs.
Though presented in the context of images, ARCs can be used in any modality. There are innumerable ways to extend ARCs. Better attention mechanisms, higher resolution images, different datasets, hyper-parameter tuning, more complicated controllers etc are the simple things which could be employed to achieve better performance.
More interesting extensions would involve developing more complex architectures using this bottom-up approach to solve even more challenging AI tasks.
ACKNOWLEDGEMENTS
We would like to thank all the members of the Statistics and Machine Learning Lab at the Indian Institute of Science for their support and feedback. We would like to specifically thank Akshay Mehrotra for his extensive help with everything from the implementation to discussing results. We would also like to thank Siddharth Agrawal and Gaurav Pandey for their helpful feedback throughout the process. We would like to thank Soumith Chintala for his feedback on this manuscript and the idea.

This paper introduces an attention-based recurrent network that learns to compare images by attending iteratively back and forth between a pair of images. Experiments show state-of-the-art results on Omniglot, though a large part of the performance gain comes from when extracted convolutional features are used as input.
The paper is significantly improved from the original submission and reflects changes based on pre-review questions. However, while there was an attempt made to include more qualitative results e.g. Fig. 2, it is still relatively weak and could benefit from more examples and analysis. Also, why is the attention in Fig. 2 always attending over the full character?  Although it is zooming in, shouldn’t it attend to relevant parts of the character?  Attending to the full character on a solid background seems a trivial solution where it is then unclear where the large performance gains are coming from.
While the paper is much more polished now, it is still lacking in details in some respects, e.g. details of the convolutional feature extractor used that gives large performance gain.


LIPNET: END-TO-END SENTENCE-LEVEL LIPREADING
Lipreading is the task of decoding text from the movement of a speaker’s mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).
1 INTRODUCTION
Lipreading plays a crucial role in human communication and speech understanding, as highlighted by the McGurk effect (McGurk & MacDonald, 1976), where one phoneme’s audio dubbed on top of a video of someone speaking a different phoneme results in a third phoneme being perceived.
Lipreading is a notoriously difficult task for humans, specially in the absence of context1. Most lipreading actuations, besides the lips and sometimes tongue and teeth, are latent and difficult to disambiguate without context (Fisher, 1968; Woodward & Barber, 1960). For example, Fisher (1968) gives 5 categories of visual phonemes (called visemes), out of a list of 23 initial consonant phonemes, that are commonly confused by people when viewing a speaker’s mouth. Many of these were asymmetrically confused, and observations were similar for final consonant phonemes.
Consequently, human lipreading performance is poor. Hearing-impaired people achieve an accuracy of only 17±12% even for a limited subset of 30 monosyllabic words and 21±11% for 30 compound words (Easton & Basala, 1982). An important goal, therefore, is to automate lipreading. Machine lipreaders have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces, security, speech recognition in noisy environments, biometric identification, and silent-movie processing.
Machine lipreading is difficult because it requires extracting spatiotemporal features from the video (since both position and motion are important). Recent deep learning approaches attempt to extract those features end-to-end. Most existing work, however, performs only word classification, not sentence-level sequence prediction.
†These authors contributed equally to this work. 1LipNet video: https://youtube.com/playlist?list=PLXkuFIFnXUAPIrXKgtIpctv2NuSo7xw3k
In this paper, we present LipNet, which is to the best of our knowledge, the first end-to-end sentence-level lipreading model. As with modern deep learning based automatic speech recognition (ASR), LipNet is trained end-to-end to make sentence-level predictions. Our model operates at the character-level, using spatiotemporal convolutional neural networks (STCNNs), recurrent neural networks (RNNs), and the connectionist temporal classification loss (CTC) Graves et al. (2006).
Our empirical results on the GRID corpus (Cooke et al., 2006), one of the few public sentence-level datasets, show that LipNet attains a 95.2% sentence-level word accuracy, in a overlapped speakers split that is popular for benchmarking lipreading methods. The previous best accuracy reported on an aligned word classification version of this task was 86.4% (Gergen et al., 2016). Furthermore, LipNet can generalise across unseen speakers in the GRID corpus with an accuracy of 88.6%.
We also compare the performance of LipNet with that of hearing-impaired people who can lipread on the GRID corpus task. On average, they achieve an accuracy of 52.3%, in contrast to LipNet’s 1.69× higher accuracy in the same sentences. Finally, by applying saliency visualisation techniques (Zeiler & Fergus, 2014; Simonyan et al., 2013), we interpret LipNet’s learned behaviour, showing that the model attends to phonologically important regions in the video. Furthermore, by computing intra-viseme and inter-viseme confusion matrices at the phoneme level, we show that almost all of LipNet’s few erroneous predictions occur within visemes, since context is sometimes insufficient for disambiguation.
2 RELATED WORK
In this section, we outline various existing approaches to automated lipreading.
Automated lipreading: Most existing work on lipreading does not employ deep learning. Such work requires either heavy preprocessing of frames to extract image features, temporal preprocessing of frames to extract video features (e.g., optical flow or movement detection), or other types of handcrafted vision pipelines (Matthews et al., 2002; Zhao et al., 2009; Gurban & Thiran, 2009; Papandreou et al., 2007; 2009; Pitsikalis et al., 2006; Lucey & Sridharan, 2006; Papandreou et al., 2009). The automated lipreading literature is too vast to adequately cover, so we refer the reader to Zhou et al. (2014) for an extensive review.
Notably, Goldschen et al. (1997) were the first to do visual-only sentence-level lipreading using hidden Markov models (HMMs) in a limited dataset, using hand-segmented phones. Later, Neti et al. (2000) were the first to do sentence-level audiovisual speech recognition using an HMM combined with hand-engineered features, on the IBM ViaVoice (Neti et al., 2000) dataset. The authors improve speech recognition performance in noisy environments by fusing visual features with audio ones. The dataset contains 17111 utterances of 261 speakers for training (about 34.9 hours) and is not publicly available. As stated, their visual-only results cannot be interpreted as visual-only recognition, as they are used as rescoring of the noisy audio-only lattices. Using a similar approach, Potamianos et al. (2003) report speaker independent and speaker adapted 91.62%, 82.31% WER in the same dataset respectively, and 38.53%, 16.77% WER in the connected DIGIT corpus, which contains sentences of digits.
Furthermore, Gergen et al. (2016) use speaker-dependent training on an LDA-transformed version of the Discrete Cosine Transforms of the mouth regions in an HMM/GMM system. This work holds the previous state-of-the-art on the GRID corpus with a speaker-dependent accuracy of 86.4%. Generalisation across speakers and extraction of motion features is considered an open problem, as noted in (Zhou et al., 2014). LipNet addresses both of these issues.
Classification with deep learning: In recent years, there have been several attempts to apply deep learning to lipreading. However, all of these approaches perform only word or phoneme classification, whereas LipNet performs full sentence sequence prediction. Approaches include learning multimodal audio-visual representations (Ngiam et al., 2011; Sui et al., 2015; Ninomiya et al., 2015; Petridis & Pantic, 2016), learning visual features as part of a traditional speech-style processing pipeline (e.g. HMMs, GMM-HMMs, etc.) for classifying words and/or phonemes (Almajai et al., 2016; Takashima et al., 2016; Noda et al., 2014; Koller et al., 2015), or combinations thereof (Takashima et al., 2016). Many of these approaches mirror early progress in applying neural networks for acoustic processing in speech recognition (Hinton et al., 2012).
Chung & Zisserman (2016a) propose spatial and spatiotemporal convolutional neural networks, based on VGG, for word classification. The architectures are evaluated on a word-level dataset BBC TV (333 and 500 classes), but, as reported, their spatiotemporal models fall short of the spatial architectures by an average of around 14%. Additionally, their models cannot handle variable sequence lengths and they do not attempt sentence-level sequence prediction.
Chung & Zisserman (2016b) train an audio-visual max-margin matching model for learning pretrained mouth features, which they use as inputs to an LSTM for 10-phrase classification on the OuluVS2 dataset, as well as a non-lipreading task.
Wand et al. (2016) introduce LSTM recurrent neural networks for lipreading but address neither sentence-level sequence prediction nor speaker independence.
Garg et al. (2016) apply a VGG pre-trained on faces to classifying words and phrases from the MIRACL-VC1 dataset, which has only 10 words and 10 phrases. However, their best recurrent model is trained by freezing the VGGNet parameters and then training the RNN, rather than training them jointly. Their best model achieves only 56.0% word classification accuracy, and 44.5% phrase classification accuracy, despite both of these being 10-class classification tasks.
Sequence prediction in speech recognition: The field of automatic speech recognition (ASR) would not be in the state it is today without modern advances in deep learning, many of which have occurred in the context of ASR (Graves et al., 2006; Dahl et al., 2012; Hinton et al., 2012). The connectionist temporal classification loss (CTC) of Graves et al. (2006) drove the movement from deep learning as a component of ASR, to deep ASR systems trained end-to-end (Graves & Jaitly, 2014; Maas et al., 2015; Amodei et al., 2015). As mentioned earlier, much recent lipreading progress has mirrored early progress in ASR, but stopping short of sequence prediction.
LipNet is the first end-to-end model that performs sentence-level sequence prediction for visual speech recogntion. That is, we demonstrate the first work that takes as input as sequence of images and outputs a distribution over sequences of tokens; it is trained end-to-end using CTC and thus also does not require alignments.
Lipreading Datasets: Lipreading datasets (AVICar, AVLetters, AVLetters2, BBC TV, CUAVE, OuluVS1, OuluVS2) are plentiful (Zhou et al., 2014; Chung & Zisserman, 2016a), but most only contain single words or are too small. One exception is the GRID corpus (Cooke et al., 2006), which has audio and video recordings of 34 speakers who produced 1000 sentences each, for a total of 28 hours across 34000 sentences. Table 1 summarises state-of-the-art performance in each of the main lipreading datasets.
We use the GRID corpus to evaluate LipNet because it is sentence-level and has the most data. The sentences are drawn from the following simple grammar: command(4) + color(4) + preposition(4) + letter(25) + digit(10) + adverb(4), where the number denotes how many word choices there are for each of the 6 word categories. The categories consist of, respectively, {bin, lay, place, set}, {blue, green, red, white}, {at, by, in, with}, {A, . . . , Z}\{W}, {zero, . . . , nine}, and
{again, now, please, soon}, yielding 64000 possible sentences. For example, two sentences in the data are “set blue by A four please” and “place red at C zero again”.
3 LIPNET
LipNet is a neural network architecture for lipreading that maps variable-length sequences of video frames to text sequences, and is trained end-to-end. In this section, we describe LipNet’s building blocks and architecture.
3.1 SPATIOTEMPORAL CONVOLUTIONS
Convolutional neural networks (CNNs), containing stacked convolutions operating spatially over an image, have been instrumental in advancing performance in computer visions tasks such as object recognition that receive an image as input (Krizhevsky et al., 2012). A basic 2D convolution layer from C channels to C ′ channels (without a bias and with unit stride) computes
[conv(x,w)]c′ij = C∑ c=1 kw∑ i′=1 kh∑ j′=1 wc′ci′j′xc,i+i′,j+j′ ,
for input x and weights w ∈ RC′×C×kw×kh where we define xcij = 0 for i, j out of bounds. Spatiotemporal convolutional neural networks (STCNNs) can process video data by convolving across time, as well as the spatial dimensions (Karpathy et al., 2014; Ji et al., 2013). Hence similarly,
[stconv(x,w)]c′tij = C∑ c=1 kt∑ t′=1 kw∑ i′=1 kh∑ j′=1 wc′ct′i′j′xc,t+t′,i+i′,j+j′ .
3.2 GATED RECURRENT UNIT
Gated Recurrent Unit (GRU) (Chung et al., 2014) is a type of recurrent neural network (RNN) that improves upon earlier RNNs by adding cells and gates for propagating information over more timesteps and learning to control this information flow. It is similar to the Long Short-Term Memory (LSTM) RNN (Hochreiter & Schmidhuber, 1997). We use the standard formulation:
[ut, rt] T = sigm(Wzzt +Whht−1 + bg)
h̃t = tanh(Uzzt +Uh(rt ht−1) + bh) ht = (1− ut) ht−1 + ut h̃t
where z := {z1, . . . , zT } is the input sequence to the RNN, denotes element-wise multiplication, and sigm(r) = 1/(1 + exp(−r)). We use a bidirectional GRU (Bi-GRU) as introduced by Graves & Schmidhuber (2005) in the context of LSTMs: one RNN maps {z1, . . . , zT } 7→ { −→ h1, . . . , −→ hT }, and another {zT , . . . , z1} 7→ { ←− h1, . . . , ←− hT }, then ht := [ −→ ht, ←− ht]. The Bi-GRU ensures that ht depends on zt′ for all t′. To parameterise a distribution over sequences, at time-step t let p(ut|z) = softmax(mlp(ht;Wmlp)), where mlp is a feed-forward network with weights Wmlp. Then we can define the distribution over length-T sequences as p(u1, . . . , uT |z) = ∏ 1≤t≤T p(ut|z), where T is determined by z, the input to the GRU. In LipNet, z is the output of the STCNN.
3.3 CONNECTIONIST TEMPORAL CLASSIFICATION
The connectionist temporal classification (CTC) loss (Graves et al., 2006) is widely used in modern speech recognition as it eliminates the need for training data that aligns inputs to target outputs (Amodei et al., 2015; Graves & Jaitly, 2014; Maas et al., 2015). Given a model that outputs a sequence of discrete distributions over the token classes (vocabulary) augmented with a special “blank” token, CTC computes the probability of a sequence by marginalising over all sequences that are defined as equivalent to this sequence. This simultaneously removes the need for alignments and addresses variable-length sequences. Let V denote the set of tokens that the model classifies at a single time-step of its output (vocabulary), and the blank-augmented vocabulary Ṽ = V ∪ { }
where denotes the CTC blank symbol. Define the function B : Ṽ ∗ → V ∗ that, given a string over Ṽ , deletes adjacent duplicate characters and removes blank tokens. For a label sequence y ∈ V ∗, CTC defines p(y|x) = ∑ u∈B−1(y) s.t. |u|=T p(u1, . . . , uT |x), where T is the number of time-steps in the sequence model. For example, if T = 3, CTC defines the probability of a string “am” as p(aam) + p(amm) + p( am) + p(a m) + p(am ). This sum is computed efficiently by dynamic programming, allowing us to perform maximum likelihood.
3.4 LIPNET ARCHITECTURE
Figure 1 illustrates the LipNet architecture, which starts with 3×(spatiotemporal convolutions, channel-wise dropout, spatial max-pooling). Subsequently, the features extracted are followed by two Bi-GRUs. The Bi-GRUs are crucial for efficient further aggregation of the STCNN output. Finally, a linear transformation is applied at each time-step, followed by a softmax over the vocabulary augmented with the CTC blank, and then the CTC loss. All layers use rectified linear unit (ReLU) activation functions. More details including hyperparameters can be found in Table 3 of Appendix A.
4 LIPREADING EVALUATION
In this section, we evaluate LipNet on the GRID corpus. The augmentation methods employed don’t make use of external data and rely purely on the GRID corpus.
4.1 DATA AUGMENTATION
Preprocessing: The GRID corpus consists of 34 subjects, each narrating 1000 sentences. The videos for speaker 21 are missing, and a few others are empty or corrupt, leaving 32746 usable videos. We employ a split (unseen speakers; not previously used in the literature) holding out the data of two male speakers (1 and 2) and two female speakers (20 and 22) for evaluation (3971 videos). The remainder is used for training (28775 videos). We also use a sentence-level variant of the split (overlapped speakers) similar to Wand et al. (2016), where 255 random sentences from each speaker are used for evaluation. All remaining data from all speakers is pooled together for training. All videos are 3 seconds long with a frame rate of 25fps. The videos were processed with the DLib face detector, and the iBug face landmark predictor (Sagonas et al., 2013) with 68 landmarks coupled with an online Kalman Filter. Using these landmarks, we apply an affine transformation to extract a mouth-centred crop of size 100 × 50 pixels per frame. We standardise the RGB channels over the whole training set to have zero mean and unit variance.
Augmentation: We augment the dataset with simple transformations to reduce overfitting. First, we train on both the regular and the horizontally mirrored image sequence. Second, since the dataset provides word start and end timings for each sentence video, we augment the sentence-level training data with video clips of individual words as additional training instances. These instances have a
decay rate of 0.925. Third, to encourage resilience to varying motion speeds by deletion and duplication of frames, this is performed with a per-frame probability of 0.05. The same augmentation methods were followed in all proposed baselines and models.
4.2 BASELINES
To evaluate LipNet, we compare its performance to that of three hearing-impaired people who can lipread, as well as three ablation models inspired by recent state-of-the-art work (Chung & Zisserman, 2016a; Wand et al., 2016).
Hearing-Impaired People: This baseline was performed by three members of the Oxford Students’ Disability Community. After being introduced to the grammar of the GRID corpus, they observed 10 minutes of annotated videos from the training dataset, then annotated 300 random videos from the evaluation dataset. When uncertain, they were asked to pick the most probable answer.
Baseline-LSTM: Using the sentence-level training setup of LipNet, we replicate the model architecture of the previous deep learning GRID corpus state-of-the-art (Wand et al., 2016). See Appendix A for more implementation details.
Baseline-2D: Based on the LipNet architecture, we replace the STCNN with spatial-only convolutions similar to those of Chung & Zisserman (2016a). Notably, contrary to the results we observe with LipNet, Chung & Zisserman (2016a) report 14% and 31% poorer performance of their STCNNs compared to the 2D architectures in their two datasets.
Baseline-NoLM: Identical to LipNet, but with the language model used in beam search disabled.
4.3 PERFORMANCE EVALUATION
To measure the performance of LipNet and the baselines, we compute the word error rate (WER) and the character error rate (CER), standard metrics for the performance of ASR models. We produce approximate maximum-probability predictions from LipNet by performing CTC beam search. WER (or CER) is defined as the minimum number of word (or character) insertions, substitutions, and deletions required to transform the prediction into the ground truth, divided by the number of words (or characters) in the ground truth. Note that WER is usually equal to classification error when the predicted sentence has the same number of words as the ground truth, particularly in our case since almost all errors are substitution errors.
Table 2 summarises the performance of LipNet compared to the baselines. According to the literature, the accuracy of human lipreaders is around 20% (Easton & Basala, 1982; Hilder et al., 2009). As expected, the fixed sentence structure and the limited subset of words for each position in the GRID corpus facilitate the use of context, increasing performance. On the unseen speakers split, the three hearing-impaired people achieve 57.3%, 50.4%, and 35.5% WER respectively, yielding an average of 47.7% WER.
For both unseen and overlapped speakers evaluation, the highest performance is achieved by the architectures enhanced with convolutional stacks. LipNet exhibits a 2.3× higher performance in the overlapped compared to the unseen speakers split. For unseen speakers, Baseline-2D and LipNet achieve 1.8× and 4.2× lower WER, respectively, than hearing-impaired people.
The WER for unseen speakers Baseline-2D is 26.7%, whereas for LipNet it is 2.3× lower, at 11.4%. Similarly, the error rate for overlapped speakers was 2.4× lower for LipNet compared to Baseline2D. Both results demonstrate the importance of combining STCNNs with RNNs. This performance difference confirms the intuition that extracting spatiotemporal features using a STCNN is better than aggregating spatial-only features. This observation contrasts with the empirical observations of Chung & Zisserman (2016a). Furthermore, LipNet’s use of STCNN, RNNs, and CTC cleanly allow processing both variable-length input and variable-length output sequences, whereas the architectures of Chung & Zisserman (2016a) and Chung & Zisserman (2016b) only handle the former.
Baseline-LSTM exhibits the lowest performance, in both unseen and overlapped speakers, with 52.8% and 26.3% WER, respectively. Interestingly, although Baseline-LSTM replicates the architecture of Wand et al. (2016), and despite the numerous data augmentation methods, the model performs 1.3× lower than the reported 79.6% word-level accuracy illustrating the difficulty of a sentence-level task even in a restricted grammar.
Finally, by disabling the language model, the Baseline-NoLM exhibits approximately 1.2× higher WER than our proposed model.
4.4 LEARNED REPRESENTATIONS
In this section, we analyse the learned representations of LipNet from a phonological perspective. First, we create saliency visualisations (Simonyan et al., 2013; Zeiler & Fergus, 2014) to illustrate where LipNet has learned to attend. In particular, we feed an input into the model and greedily decode an output sequence, yielding a CTC alignment û ∈ Ṽ ∗ (following the notation of Sections 3.2 and 3.3). Then, we compute the gradient of ∑ t p(ût|x) with respect to the input video frame sequence, but unlike Simonyan et al. (2013), we use guided backpropagation (Springenberg et al., 2014). Second, we train LipNet to predict ARPAbet phonemes, instead of characters, to analyse visual phoneme similarities using intra-viseme and inter-viseme confusion matrices.
4.4.1 SALIENCY MAPS
We apply saliency visualisation techniques to interpret LipNet’s learned behaviour, showing that the model attends to phonologically important regions in the video. In particular, in Figure 2 we analyse two saliency visualisations for the words please and lay for speaker 25, based on Ashby (2013).
The production of the word please requires a great deal of articulatory movement at the beginning: the lips are pressed firmly together for the bilabial plosive /p/ (frame 1). At the same time, the blade of the tongue comes in contact with the alveolar ridge in anticipation of the following lateral /l/. The lips then part, allowing the compressed air to escape between the lips (frame 2). The jaw and lips then open further, seen in the distance between the midpoints of the upper and lower lips, and the lips spread (increasing the distance between the corners of the mouth), for the close vowel /iy/ (frame 3–4). Since this is a relatively steady-state vowel, lip position remains unchanged for the rest of its duration (frames 4–8), where the attention level drops considerably. The jaw and the lips then close slightly, as the blade of the tongue needs to be brought close to the alveolar ridge, for /z/ (frames 9–10), where attention resumes.
Lay is interesting since the bulk of frontally visible articulatory movement involves the blade of the tongue coming into contact with the alveolar ridge for /l/ (frames 2–6), and then going down for the vowel /ey/ (frames 7–9). That is exactly where most of LipNet’s attention is focused, as there is little change in lip position.
4.4.2 VISEMES
According to DeLand (1931) and Fisher (1968), Alexander Graham Bell first hypothesised that multiple phonemes may be visually identical on a given speaker. This was later verified, giving rise to the concept of a viseme, a visual equivalent of a phoneme (Woodward & Barber, 1960; Fisher, 1968). For our analysis, we use the phoneme-to-viseme mapping of Neti et al. (2000), clustering the phonemes into the following categories: Lip-rounding based vowels (V), Alveolar-semivowels (A), Alveolar-fricatives (B), Alveolar (C), Palato-alveolar (D), Bilabial (E), Dental (F), Labio-dental (G), and Velar (H). The full mapping can be found in Table 4 in Appendix A. The GRID corpus contain 31 out of the 39 phonemes in ARPAbet. We compute confusion matrices between phonemes and then
group phonemes into viseme clusters, following Neti et al. (2000). Figure 3 shows the confusion matrices of the 3 most confused viseme categories, as well as the confusions between the viseme categories. The full phoneme confusion matrix is in Figure 4 in Appendix B.
Given that the speakers are British, the confusion between /aa/ and /ay/ (Figure 3a) is most probably due to the fact that the first element, and the greater part, of the diphthong /ay/ is articulatorily identical with /aa/: an open back unrounded vowel (Ferragne & Pellegrino, 2010). The confusion of /ih/ (a rather close vowel) and /ae/ (a very open vowel) is at first glance surprising, but in fact in the sample /ae/ occurs only in the word at, which is a function word normally pronounced with a reduced, weak vowel /ah/. /ah/ and /ih/ are the most frequent unstressed vowels and there is a good deal of variation within and between them, e.g. private and watches (Cruttenden, 2014).
The confusion within the categories of bilabial stops /p b m/ and alveolar stops /t d n/ (Figures 3b-c) is unsurprising: complete closure at the same place of articulation makes them look practically identical. The differences of velum action and vocal fold vibration are unobservable from the front.
Finally, the quality of the viseme categorisation of Neti et al. (2000) is confirmed by the fact that the matrix in Figure 3d is diagonal, with only minor confusion between alveolar (C) and palatoalveolar (D) visemes. Articulatorily, alveolar /s z/ and palato-alveolar /sh zh/ fricatives are distinguished by only a small difference in tongue position: against the palate just behind the alveolar ridge, which is not easily observed from the front. The same can be said about dental /th/ and alveolar /t/.
5 CONCLUSIONS
We proposed LipNet, the first model to apply deep learning to end-to-end learning of a model that maps sequences of image frames of a speaker’s mouth to entire sentences. The end-to-end model eliminates the need to segment videos into words before predicting a sentence. LipNet requires neither hand-engineered spatiotemporal visual features nor a separately-trained sequence model.
Our empirical evaluation illustrates the importance of spatiotemporal feature extraction and efficient temporal aggregation, confirming the intuition of Easton & Basala (1982). Furthermore, LipNet greatly outperforms a human lipreading baseline, exhibiting 4.1× better performance, and 4.8% WER which is 2.8× lower than the word-level state-of-the-art (Gergen et al., 2016) in the GRID corpus.
While LipNet is already an empirical success, the deep speech recognition literature (Amodei et al., 2015) suggests that performance will only improve with more data. In future work, we hope to demonstrate this by applying LipNet to larger datasets, such as a sentence-level variant of that collected by Chung & Zisserman (2016a).
Some applications, such as silent dictation, demand the use of video only. However, to extend the range of potential applications of LipNet, we aim to apply this approach to a jointly trained audiovisual speech recognition model, where visual input assists with robustness in noisy environments.
ACKNOWLEDGMENTS
This work was supported by an Oxford-Google DeepMind Graduate Scholarship, the EPSRC, and CIFAR. We would also like to thank: NVIDIA for their generous donation of DGX-1 and GTX Titan X GPUs, used in our experiments; Áine Jackson, Brittany Klug and Samantha Pugh for helping us measure the experienced lipreader baseline; Mitko Sabev for his phonetics guidance; Odysseas Votsis for his video production help; and Alex Graves and Oiwi Parker Jones for helpful comments.
A ARCHITECTURE DETAILS
In this appendix, we provide additional details about the implementation and architecture.
A.1 IMPLEMENTATION
LipNet is implemented using Torch, the warp-ctc CTC library (Amodei et al., 2015), and StanfordCTC’s decoder implementation. The network parameters were initialised using He initialisation (He et al., 2015), apart from the square GRU matrices that were orthogonally initialised, as described in (Chung et al., 2014). The models were trained with channel-wise dropout (dropout rate p = 0.5) after each pooling layer and mini-batches of size 50. We used the optimiser Adam (Kingma & Ba, 2014) with a learning rate of 10−4, and the default hyperparameters: a first-moment momentum coefficient of 0.9, a second-moment momentum coefficient of 0.999, and the numerical stability parameter = 10−8.
The CER and WER scores were computed using CTC beam search with the following parameters for Stanford-CTC’s decoder: beam width 200, α = 1, and β = 1.5. On top of that, we use a character 5-gram binarised language model, as suggested in (Graves & Jaitly, 2014).
A.2 LIPNET ARCHITECTURE
The videos were processed with DLib face detector (King, 2009) and the iBug face shape predictor with 68 landmarks (Sagonas et al., 2013). The RGB input frames were normalised using the following per-channel means and standard deviations: [µR = 0.7136, σR = 0.1138, µG = 0.4906, σG = 0.1078, µB = 0.3283, σB = 0.0917].
Table 3 summarises the LipNet architecture hyperparameters, where T denotes time, C denotes channels, F denotes feature dimension, H and W denote height and width and V denotes the number of words in the vocabulary including the CTC blank symbol.
Note that spatiotemporal convolution sizes depend on the number of channels, and the kernel’s three dimensions. Spatiotemporal kernel sizes are specified in the same order as the input size dimensions. The input dimension orderings are given in parentheses in the input size column.
Layers after the Bi-GRU are applied per-timestep.
A.3 BASELINE-LSTM ARCHITECTURE
Baseline-LSTM replicates the setup of Wand et al. (2016), and is trained the same way as LipNet. The model uses two LSTM layers with 128 neurons. The input frames were converted to grayscale and were down-sampled to 50×25px, dropout p = 0, and the parameters were initialised uniformly with values between [−0.05, 0.05].
B PHONEMES AND VISEMES
Table 4 shows the phoneme to viseme clustering of Neti et al. (2000) and Figure 4 shows LipNet’s full phoneme confusion matrix.

UPDATE:  I have read the authors' responses.  I did not read the social media comments about this paper prior to reviewing it.
I appreciate the authors' updates in response to the reviewer comments.  Overall, however, my review stands.  The authors have taken a task that had not yet been addressed with a straightforward modern deep learning approach, and addressed it with such an approach.  I assume that if we pick up any task that hasn't been worked on for a while, and give it a solid deep learning treatment, we will do well.  I do not see such papers as a contribution to ICLR, unless they also provide new insights, analysis, or surprising results (which, to my mind, this paper does not).  This is a general point and the program chairs may disagree with it, of course.
I have removed my recommendation that this be accepted as a workshop paper, as I have since noticed that the workshop track this year has a different focus.
************************
ORIGINAL REVIEW:
The authors show that an appropriately engineered LSTM+CNN+CTC network does an excellent job of lipreading on the GRID corpus.  This is a nice result to know about--yet another example of a really nice result that one can get the first time one applies such methods to an old task--and all of the work that went into getting it looks solid (and likely involved some significant engineering effort).  However, this in itself is not sufficiently novel for publication at ICLR.  The paper also needs to be revised to better represent prior work, and ideally remove some of the vague motivational language.  Some specifics on what I think needs to be revised:
- First, the claim of being the first to do sentence-level lipreading.  As mentioned in a pre-review comment, this is not true.  The paper should be revised to discuss the prior work on this task (even though much of it used data that is not public).  Ideally the title should also be changed in light of this.
- The comparison with human lipreaders needs to be qualified a bit.  This task is presumably very unnatural for humans because of the unusual grammar, so perhaps what you are showing is that a machine can better take into account the strong contraints.  This is great, but not a general statement about LipNet vs. humans.
- The paper contains some unnecessary motivational platitudes.  We do not need to invoke Easton and Basala 1982 to motivate modeling context in a linguistic sequence prediction task, and prior work using older sequence models (e.g. HMMs) for lipreading has modeled context as well.  The McGurk effect does not show that lipreading plays a crucial role in human communication.
- It is worth noting that even without the spatial convolution, your Baseline-2D already does extremely well.  So I am not sure about the "importance of spatiotemporal feature extraction" as stated in the conclusion.
Some more minor comments, typos, etc.:
- citations for LSTMs, CTC, etc. should be provided the first time they are mentioned.
- I did not quite follow the justification for upsampling.
- what is meant by "lip-rounding vowels"?  They seem to include almost all English vowels.
- Did you consider keeping the vowel visemes V1-V4 separate rather than collapsing them into one?  Since you list Neti et al.'s full viseme set, it is worth mentioning why you modified it.
- "Given that the speakers are British, the confusion between /aa/ and /ay/..." -- I am not sure what this has to do with British speakers, as the relationship between these vowels exists in other English dialects as well (e.g. American).
- The discussion about confusions within bilabial stops and within alveolar stops is a bit mismatched with the actual confusion data in Fig. 3(b,c).  For example, there does not seem to be any confusion between /m/ and /b/ or between /m/ and /p/.
- "lipreading actuations":  I am not sure what "actuations" means in this context
- "palato-alvealoar" --> "palato-alveolar"
- "Articulatorily alveolar" --> "Alveolar"?