UOUS AND DISCRETE ADDRESSING SCHEMES
In this paper, we extend neural Turing machine (NTM) into a dynamic neural Turing machine (D-NTM) by introducing a trainable memory addressing scheme. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies including both linear and nonlinear ones. We implement the D-NTM with both continuous, differentiable and discrete, non-differentiable read/write mechanisms. We investigate the mechanisms and effects for learning to read and write to a memory through experiments on Facebook bAbI tasks using both a feedforward and GRU-controller. The D-NTM is evaluated on a set of Facebook bAbI tasks and shown to outperform NTM and LSTM baselines. We also provide further experimental results on sequential MNIST, associative recall and copy tasks.
1 INTRODUCTION
Designing general-purpose learning algorithms is one of the long-standing goals of artificial intelligence. Despite the success of deep learning in this area (see, e.g., (Goodfellow et al., 2016)) there are still a set of complex tasks that are not well addressed by conventional neural networks. Those tasks often require a neural network to be equipped with an explicit, external memory in which a larger, potentially unbounded, set of facts need to be stored. They include, but are not limited to, episodic question-answering (Weston et al., 2015b; Hermann et al., 2015; Hill et al., 2015), compact algorithms (Zaremba et al., 2015), dialogue (Serban et al., 2016; Vinyals & Le, 2015) and video caption generation (Yao et al., 2015).
Recently two promising approaches based on neural networks to this type of tasks have been proposed. Memory networks (Weston et al., 2015b) explicitly store all the facts, or information, available for each episode in an external memory (as continuous vectors) and use the attention-based mechanism to index them when returning an output. On the other hand, neural Turing machines (NTM, (Graves et al., 2014)) read each fact in an episode and decides whether to read, write the fact or do both to the external, differentiable memory.
A crucial difference between these two models is that the memory network does not have a mechanism to modify the content of the external memory, while the NTM does. In practice, this leads to easier learning in the memory network, which in turn resulted in it being used more in real tasks (Bordes et al., 2015; Dodge et al., 2015). On the contrary, the NTM has mainly been tested on a series of small-scale, carefully-crafted tasks such as copy and associative recall. The NTM, however is more expressive, precisely because it can store and modify the internal state of the network as it processes an episode.
The original NTM supports two modes of addressing (which can be used simultaneously.) They are content-based and location-based addressing. We notice that the location-based strategy is based on linear addressing. The distance between each pair of consecutive memory cells is fixed to a constant. We address this limitation, in this paper, by introducing a learnable address vector for each memory cell of the NTM with least recently used memory addressing mechanism, and we call this variant a dynamic neural Turing machine (D-NTM).
We evaluate the proposed D-NTM on the full set of Facebook bAbI task (Weston et al., 2015b) using either continuous, differentiable attention or discrete, non-differentiable attention (Zaremba & Sutskever, 2015) as an addressing strategy. Our experiments reveal that it is possible to use the discrete, non-differentiable attention mechanism, and in fact, the D-NTM with the discrete attention and GRU controller outperforms the one with the continuous attention. After we published our paper on arXiv, a new extension of NTM called DNC (Graves et al., 2016) has also provided results on bAbI task as well.
We also provide results on sequential-MNIST and algorithmic tasks proposed by (Graves et al., 2014) in order to investigate the ability of our model when dealing with long-term dependencies.
Our Contributions
1. We propose a generalization of Neural Turing Machine called a dynamic neural Turing machine (D-NTM) which employs a learnable and location-based addressing. 2. We demonstrate the application of neural Turing machines on a more natural and less toyish task: episodic question-answering besides the toy tasks. We provide detailed analysis of our model on this task. 3. We propose to use the discrete attention mechanism and empirically show that, it can outperform the continuous attention based addressing for episodic QA task. 4. We propose a curriculum strategy for our model with the feedforward controller and discrete attention that improves our results significantly.
2 DYNAMIC NEURAL TURING MACHINE
The proposed dynamic neural Turing machine (D-NTM) extends the neural Turing machine (NTM, (Graves et al., 2014)) which has a modular design. The NTM consists of two main modules, a controller and, a memory. The controller, which is often implemented as a recurrent neural network, issues a command to the memory so as to read, write to and erase a subset of memory cells. Although the memory was originally envisioned as an integrated module, it is not necessary, and the memory may be an external, black box (Zaremba & Sutskever, 2015).
2.1 CONTROLLER
At each time step t, the controller (1) receives an input value xt, (2) addresses and reads the memory and creates the content vector φt, (3) erases/writes a portion of the memory, (4) updates its own hidden state ht, and (5) outputs a value yt (if needed.) In this paper, we use both a gated recurrent unit (GRU, (Cho et al., 2014)) and a feedforward-controller to implement the controller such that for a GRU controller
ht = GRU(xt,ht−1, φt) (1)
or for a feedforward-controller
ht = σ(xt, φt). (2)
2.2 MEMORY
We use a rectangular matrix M ∈ RN×(dc+da) to denote N memory cells. Unlike the original NTM, we partition each memory cell vector into two parts:
M = [A;C] .
The first part A ∈ RN×da is a learnable address matrix, and the second C ∈ RN×dc a content matrix. In other words, each memory cell mi is now
mi = [ai; ci] .
The address part ai is considered a model parameter that is updated during training. During inference, the address part is not overwritten by the controller and remains constant. On the other hand, the content part ci is both read and written by the controller both during training and inference. At the beginning of each episode, the content part of the memory is refreshed to be an all-zero matrix, C0 = 0. This introduction of the learnable address portion for each memory cell allows the model to learn sophisticated location-based addressing strategies. A similar addressing mechanism is also explored in (Reed & de Freitas, 2015) in the context of learning program traces.
2.3 MEMORY ADDRESSING
Memory addressing in the D-NTM is equivalent to computing an N -dimensional address vector. The DNTM computes three such vectors for respectively reading wt ∈ RN , erasing et ∈ Rdc and writing ut ∈ RN . Specifically for writing, the controller further computes a candidate memory content vector c̄t ∈
Rdc based on its current hidden state of the controller ht ∈ Rdh and the input of the controller scaled with a scalar gate αt which is a function of the hidden state and the input of the controller as well, see Eqn 4.
αt = f(ht,xt), (3)
c̄t = ReLU(Wmht + αtWxxt + bm). (4)
Reading With the read vector wt, the content vector read from the memory φt ∈ Rda+dc is retrieved by
φt = (wt)>Mt−1, (5)
where wt is a row vector.
Erasing and Writing Given the erase, write and candidate memory content vectors (et, utj , and c̄t respectively) generated by a simple MLP conditioned on the hidden state of the controller ht, the memory matrix is updated by,
Ct[j] = (1− etutj) Ct−1[j] + utj c̄t. (6)
where the subscript j in Ct[j] denotes the j-th row of the content part Ct of the memory matrix Mt.
No Operation (NOP) As found in (Joulin & Mikolov, 2015), an additional NOP action might be beneficial for the controller not to access the memory once in a while. We model this situation by designating one memory cell as a NOP cell. Reading or writing from this memory cell is ignored.
2.4 LEARNING
Once the proposed D-NTM is executed, it returns the output distribution p(y|x1, . . . ,xT ). As a result, we define a cost function as the negative log-likelihood:
C(θ) = 1
N N∑ n=1 − log p(yn|xn1 , . . . ,xnT ), (7)
where θ is a set of all the parameters. As the proposed D-NTM, just like the original NTM, is fully end-to-end differentiable, we can compute the gradient of this cost function by using backpropagation and learn the parameters of the model with a gradient-based optimization algorithm, such as stochastic gradient descent, to train it end-to-end.
3 ADDRESSING MECHANISM
3.1 ADDRESS VECTORS
Each of the address vectors (both read and write) is computed in the same way. The way they are computed are very similar to the content based addressing in (Graves et al., 2014). First, the controller computes a key vector:
kt = W>k h t + bk,
where Wk ∈ RN×(da+dc) and bk ∈ Rda+dc if the read head is being computed, otherwise Wk ∈ RN×dc and bk ∈ Rdc if the write head weights are being computed. They can be the parameters for a specific head (either read or write.) Also, the sharpening factor βt ∈ R≥1 is computed as:
softplus(x) = log(exp(x) + 1) (8)
βt = softplus(u>β h t + bβ) + 1. (9)
uβ and bβ are the parameters of the sharpening βt.
The address vector is then computed by, zti = β tS ( kt,mti ) (10)
wti = exp(zti)∑ j exp(z t j) , (11)
where the similarity function S ∈ R≥0 is defined as
S (x,y) = x · y
(||x||||y||+ ) .
3.2 MULTI-STEP ADDRESSING
At each time-step, controller may require more than one-step for accessing to the memory. The original NTM addresses this by implementing multiple sets of read, erase and write heads. In this paper, we explore an option of allowing each head to operate more than once at each time step, similar to the multi-hop mechanism from the end-to-end memory network (Sukhbaatar et al., 2015).
3.3 DYNAMIC LEAST RECENTLY USED ADDRESSING
We introduce a memory addressing schema that can learn to put more emphasis on the least recently used (LRU) memory locations. As observed in (Santoro et al., 2016; Rae et al., 2016), we find it easier to learn the write operations with the use of LRU addressing.
To learn a LRU based addressing, first we compute the exponentially moving averages of the logits (zt) as vt, vt = 0.1vt−1 + 0.9zt. We rescale the accumulated vt with γt, such that the controller adjusts the influence of how much previously written memory locations should effect the attention weights of a particular time-step. Next, we subtract vt from zt in order to reduce the weights of previously read or written memory locations. γt is a shallow MLP with a scalar output and it is conditioned on the hidden state of the controller. γt is parametrized with the parameters uγ and bγ ,
γt = sigmoid(u>γ ht + bγ), (12)
wt = softmax(zt − γtvt−1). (13)
This addressing method increases the weights of the least recently used rows of the memory. The magnitude of the influence of the least-recently used memory locations is being learned and adjusted with γt. Our LRU addressing is dynamic due to the model’s ability to switch between pure content-based addressing and LRU. During the training, we do not backpropagate through vt. Due to the dynamic nature of this addressing mechanism, it can be used for both read and write operations. If needed, the model will automatically learn to disable LRU while reading from the memory.
4 GENERATING DISCRETE ADDRESS VECTORS
In this section, we describe the discrete attention based addressing strategy.
Discrete Addressing Let us use w to denote an address vector (either read, write or erase) at time t. By definition in Eq. (10), every element in this address vector is positive and sums up to one. In other words, we can treat this vector as the probabilities of a categorical distribution C(w) with dim(w) choices:
p(j) = wj ,
where wj is the j-th element of w. We can readily sample from this categorical distribution and form an one-hot vector w̃ such that
w̃k = I(k = j),
where j ∼ C(w), and I is an indicator function.
Training We use this sampling-based strategy for all the heads during training. This clearly makes the use of backpropagation infeasible to compute the gradient, as the sampling procedure is not differentiable. Thus, we use REINFORCE (Williams, 1992) together with the three variance reduction techniques–global baseline, input-dependent baseline and variance normalization– suggested in (Mnih & Gregor, 2014).
Let us define R(x) = log p(y|x1, . . . ,xT ) as a reward. We first center and re-scale the reward by
R̃(x) = R(x)− b√ σ2 + ,
where b and σ is running average and standard deviation of R. We can further center it for each input x separately, i.e.,
R̃(x)← R̃(x)− b(x), where b(x) is computed by a baseline network which takes as input x and predicts its estimated reward. The baseline network is trained to minimize the Huber loss (Huber, 1964) between the true reward R̃(x)∗ and the predicted reward b(x). We use the Huber loss, which is defined by
Hδ(x) = { x2 for |x| ≤ δ, δ(2|x| − δ), otherwise,
due to its robustness. As a further measure to reduce the variance, we regularize the negative entropy of all those category distributions to facilitate a better exploration during training (Xu et al., 2015).
Then, the cost function for each training example is approximated as Cn(θ) =− log p(y|x1:T , w̃1:J , ũ1:J , ẽ1:J)
− J∑ j=1 R̃(xn)(log p(w̃j |x1:T ) + log p(ũj |x1:T ) + log p(ẽj |x1:T )) − λH J∑ j=1 (H(wj |x1:T ) +H(uj |x1:T ) +H(ej |x1:T )).
where J is the number of addressing steps, λH is the entropy regularization coefficient, andH denotes the entropy.
Inference Once training is over, we switch to a deterministic strategy. We simply choose an element of w with the largest value to be the index of the target memory cell, such that
w̃k = I(k = argmax(w)).
Curriculum Learning for the Discrete Attention Training discrete attention with feed-forward controller and REINFORCE is challenging. We propose to use a curriculum strategy for training with the discrete attention in order to tackle this problem. For each minibatch, we sample π from a binomial distribution with the probability pt, πt ∼ Bin(pt). The model will either use the discrete or the continuous-attention based on the πt. We start the training procedure with p0 = 1 and during the training pt is annealed to 0 by setting pt = p 0
√ 1+t .
We can rewrite the weights wt as in Equation 14, where it is expressed as the combination of continuous attention weights w̄t and discrete attention weights w̃t with πt being a binary variable that chooses to use one of them,
wt ← πtw̄t + (1− πt)w̃t. (14)
By using this curriculum learning strategy, at the beginning of the training, the model learns to use the memory mainly with the continuous attention. As we anneal the pt, the model will rely more on the discrete attention.
5 REGULARIZING DYNAMIC NEURAL TURING MACHINES
When the controller of D-NTM is a powerful recurrent neural network, it is important to regularize training of the D-NTM so as to avoid suboptimal solutions in which the D-NTM ignores the memory and works as a simple recurrent neural network.
Read-Write Consistency Regularizer One such suboptimal solution we have observed in our preliminary experiments with the proposed D-NTM is that the D-NTM uses the address part A of the memory matrix simply as an additional weight matrix, rather than as a means to accessing the content part C. We found that this pathological case can be effectively avoided by encouraging the read head to point to a memory cell which has also been pointed by the write head. This can be implemented as the following regularization term:
Rrw(w,u) = λ T∑ t′=1 ||1− ( 1 t′ t′∑ t=1 ut) >wt′ ||22 (15)
In the equations above, ut is the write and wt is the read weights.
Next Input Prediction as Regularization Temporal structure is a strong signal that should be exploited by the controller based on a recurrent neural network. We exploit this structure by letting the controller predict the input in the future. We maximize the predictability of the next input by the controller during training. This is equivalent to minimizing the following regularizer:
Rpred(W) = − log p(ft+1|ft,wt,ut,Mt;W))
where ft is the current input and ft+1 is the input at next timestep. We found this regularizer to be effective in our preliminary experiments and use it for bAbI tasks.
6 RELATED WORK
A recurrent neural network (RNN), which is used as a controller in the proposed D-NTM, has an implicit memory in the form of recurring hidden states. Even with this implicit memory, a vanilla RNN is however known to have difficulties in storing information for long time-spans (Bengio et al., 1994; Hochreiter, 1991). Long short-term memory (LSTM, (Hochreiter & Schmidhuber, 1997)) and gated recurrent units (GRU, (Cho et al., 2014)) have been found to address this issue. However all these models based solely on RNNs have been found to be limited when they are used to solve, e.g., algorithmic tasks and episodic question-answering.
In addition to the finite random access memory of the neural Turing machine, based on which the D-NTM is designed, other data structures have been proposed as external memory for neural networks. In (Sun et al., 1997; Grefenstette et al., 2015; Joulin & Mikolov, 2015), a continuous, differentiable stack was proposed. In (Zaremba et al., 2015; Zaremba & Sutskever, 2015), grid and tape storages are used. These approaches differ from the NTM in that their memory is unbounded and can grow indefinitely. On the other hand, they are often not randomly accessible.
Memory networks (Weston et al., 2015b) form another family of neural networks with external memory. In this class of neural networks, information is stored explicitly as it is (in the form of its continuous representation) in the memory, without being erased or modified during an episode. Memory networks and their variants have been applied to various tasks successfully (Sukhbaatar et al., 2015; Bordes et al., 2015; Dodge et al., 2015; Xiong et al., 2016). Miller et al. (2016) have also independently proposed the idea of having separate key and value vectors for memory networks.
Another related family of models is the attention-based neural networks. Neural networks with continuous or discrete attention over an input have shown promising results on a variety of challenging tasks, including machine translation (Bahdanau et al., 2015; Luong et al., 2015), speech recognition (Chorowski et al., 2015), machine reading comprehension (Hermann et al., 2015) and image caption generation (Xu et al., 2015).
The latter two, the memory network and attention-based networks, are however clearly distinguishable from the D-NTM by the fact that they do not modify the content of the memory.
7 EXPERIMENTS
We provide experimental results to demonstrate the abilities of our model, first on Facebook bAbI task (Weston et al., 2015a). We give detailed analysis and experimental results on this task. We also compare different variations of NTM on bAbI tasks. We have performed experiments on sequential permuted MNIST (Le et al., 2015) and on toy tasks to compare other published models on these tasks with a recurrent controller. The details of our experiments are provided in the supplementary material.
7.1 EPISODIC QUESTION-ANSWERING: BABI TASKS
In this section, we evaluate the proposed D-NTM on the recently proposed episodic question-answering task called Facebook bAbI. We use the dataset with 10k training examples per sub-task provided by Facebook.1 For each episode, the D-NTM reads a sequence of factual sentences followed by a question, all of which are given as natural language sentences. The D-NTM is expected to store and retrieve relevant information in the memory in order to answer the question based on the presented facts. Exact implementation details and hyper-parameter settings are provided in the appendix.
7.1.1 GOALS
The goal of this experiment is three-fold. First, we present for the first time the performance of a memory-based network that can both read and write dynamically on the Facebook bAbI tasks2. We aim to understand whether a model that has to learn to write an incoming fact to the memory, rather than storing it as it is, is able to work well, and to do so, we compare both the original NTM and proposed D-NTM against an LSTM-RNN.
Second, we investigate the effect of having to learn how to write. The fact that the NTM needs to learn to write likely has adverse effect on the overall performance, when compared to, for instance, end-to-end memory networks (MemN2N, (Sukhbaatar et al., 2015)) and dynamic memory network (DMN+, (Xiong et al., 2016)) both of which simply store the incoming facts as they are. We quantify this effect in this experiment. Lastly, we show the effect of the proposed learnable addressing scheme.
We further explore the effect of using a feedforward controller instead of the GRU controller. In addition to the explicit memory, the GRU controller can use its own internal hidden state as the memory. On the other hand, the feedforward controller must solely rely on the explicit memory, as it is the only memory available.
7.1.2 RESULTS AND ANALYSIS
In Table 1, we first observe that the NTMs are indeed capable of solving this type of episodic question-answering better than the vanilla LSTM-RNN. Although the availability of explicit memory in the NTM has already suggested this result, we note that this is the first time neural Turing machines have been used in this specific task.
All the variants of NTM with the GRU controller outperform the vanilla LSTM-RNN. However, not all of them perform equally well. First, it is clear that the proposed dynamic NTM (D-NTM) using the GRU controller outperforms the original NTM with the GRU controller (NTM, CBA only NTM vs. continuous D-NTM, Discrete D-NTM). As discussed earlier, the learnable addressing scheme of the D-NTM allows the controller to access the memory slots by location in a potentially nonlinear way. We expect it to help with tasks that have non-trivial access patterns, and as anticipated, we see a large gain with the D-NTM over the original NTM in the tasks of, for instance, 12 - Conjunction and 17 - Positional Reasoning.
Among the recurrent variants of the proposed D-NTM, we notice significant improvements by using discrete addressing over using continuous addressing. We conjecture that this is due to certain types of tasks that require precise/sharp retrieval of a stored fact, in which case continuous addressing is in disadvantage over discrete addressing. This is evident from the observation that the D-NTM with discrete addressing significantly outperforms that with continuous addressing in the tasks of 8 -
1 https://research.facebook.com/researchers/1543934539189348 2Similar experiments were done in the recently published (Graves et al., 2016), but D-NTM results for bAbI
tasks were already available in arxiv by that time.
Lists/Sets and 11 - Basic Coreference. Furthermore, this is in line with an earlier observation in (Xu et al., 2015), where discrete addressing was found to generalize better in the task of image caption generation.
In Table 2, we also observe that the D-NTM with the feedforward controller and discrete attention performs worse than LSTM and D-NTM with continuous-attention. However, when the proposed curriculum strategy from Sec. 4 is used, the average test error drops from 68.30 to 37.79.
We empirically found training of the feedforward controller more difficult than that of the recurrent controller. We train our feedforward controller based models four times longer (in terms of the number of updates) than the recurrent controller based ones in order to ensure that they are converged for most of the tasks. On the other hand, the models trained with the GRU controller overfit on bAbI tasks very quickly. For example, on tasks 3 and 16 the feedforward controller based model underfits (i.e., high training loss) at the end of the training, whereas with the same number of units the model with the GRU controller can overfit on those tasks after 3,000 updates only.
When our results are compared to the variants of the memory network Weston et al. (2015b) (MemN2N and DMN+), we notice a significant performance gap. We attribute this gap to the difficulty in learning to manipulate and store a complex input.
We also provide further experiments investigating different extensions on D-NTM in the appendix.
7.2 SEQUENTIAL pMNIST
In sequential MNIST task, the pixels of the MNIST digits are provided to the model in scan line order, left to right and top to bottom (Le et al., 2015). At the end of sequence of pixels, the model predicts the label of the digit in the sequence of pixels. We experiment D-NTM on the variation of sequential MNIST where the order of the pixels is randomly shuffled, we call this task as permuted MNIST (pMNIST). An important contribution of this task to our paper, in particular, is to measure the model’s ability to perform well when dealing with long-term dependencies. We report our results in Table 33, we observe improvements over other models that we compare against. In Table 3, ”discrete addressing with MAB” refers to D-NTM model using REINFORCE with baseline computed from moving averages of the reward. Discrete addressing with IB refers to D-NTM using REINFORCE with input-based baseline.
7.3 NTM TOY TASKS
We explore the possibility of using D-NTM to solve algorithmic tasks such as copy and associative recall tasks. We train our model on the same lengths of sequences that is experimented in (Graves et al., 2014). We report our results in Table 4. We find out that D-NTM using continuous-attention can successfully learn the ”Copy” and ”Associative Recall” tasks.
In Table 4, we train our model on sequences of the same length as the experiments in (Graves et al., 2014) and test the model on the sequences of the maximum length seen during the training. We consider model to be successful on copy or associative recall if its validation cost (binary cross-entropy) is lower than 0.02 over the sequences of maximum length seen during the training. We set the threshold to 0.02 to determine whether a model is successful on a task. Because empirically we observe that the models have higher validation costs perform badly in terms of generalization over the longer sequences. ”D-NTM discrete” model in this table is trained with REINFORCE using moving averages to estimate the baseline.
Test Acc
D-NTM discrete MAB 89.6 D-NTM discrete IB 92.3 Soft D-NTM 93.4 NTM 90.9
I-RNN (Le et al., 2015) 82.0 Zoneout (Krueger et al., 2016) 93.1 LSTM (Krueger et al., 2016) 89.8 Unitary-RNN (Arjovsky et al., 2015) 91.4 Recurrent Dropout (Krueger et al., 2016) 92.5
Table 3: Sequential pMNIST.
Copy Tasks Associative Recall
Soft D-NTM Success Success D-NTM discrete Success Failure NTM Success Success
Table 4: NTM Toy Tasks.
8 CONCLUSION AND FUTURE WORK
In this paper we extend neural Turing machines (NTM) by introducing a learnable addressing scheme which allows the NTM to be capable of performing highly nonlinear location-based addressing. This extension, to which we refer by dynamic NTM (D-NTM), is extensively tested with various configurations, including different addressing mechanisms (continuous vs. discrete) and different number of addressing steps, on the Facebook bAbI tasks. This is the first time an NTM-type model was tested on this task, and we observe that the NTM, especially the proposed D-NTM, performs better than vanilla LSTM-RNN. Furthermore, the experiments revealed that the discrete, discrete addressing works better than the continuous addressing with the GRU controller, and our analysis reveals that this is the case when the task requires precise retrieval of memory content.
Our experiments show that the NTM-based models can be weaker than other variants of memory networks which do not learn but have an explicit mechanism of storing incoming facts as they are. We conjecture that this is due to the difficulty in learning how to write, manipulate and delete the content of memory. Despite this difficulty, we find the NTM-based approach, such as the proposed D-NTM,
3Let us note that, the current state of art on this task is recurrent batch normalization with LSTM (Cooijmans et al., 2016) with 95.6% accuracy. It is possible to use recurrent batch normalization in our model and potentially improve our results on this task as well.
to be a better, future-proof approach, because it can scale to a much longer horizon (where it becomes impossible to explicitly store all the experiences.)
On pMNIST task, we show that our model can outperform other similar type of approaches proposed to deal with the long-term dependencies. On copy and associative recall tasks, we show that our model can solve the algorithmic problems that are proposed to solve with NTM type of models.
The success of both the learnable address and the discrete addressing scheme suggests two future research directions. First, we should try both of these schemes in a wider array of memory-based models, as they are not specific to the neural Turing machines. Second, the proposed D-NTM needs to be evaluated on a diverse set of applications, such as text summarization (Rush et al., 2015), visual questionanswering (Antol et al., 2015) and machine translation, in order to make a more concrete conclusion.
A EXPERIMENTAL DETAILS
A.1 MODEL AND TRAINING DETAILS FOR BABI
We use the same hyperparameters for all the tasks for a given model.
A.1.1 FACT REPRESENTATION
We use a recurrent neural network with GRU units to encode a variable-length fact into a fixed-size vector representation. This allows the D-NTM to exploit the word ordering in each fact, unlike when facts are encoded as bag-of-words vectors.
A.1.2 CONTROLLER
We experiment with both a recurrent and feedforward neural network as the controller that generates the read and write weights. The controller has 180 units. We train our feed-forward controller using noisy-tanh activation function (Gulcehre et al., 2016) since we were experiencing training difficulties with sigmoid and tanh activation functions. We use both single-step and three-steps addressing with our GRU controller.
A.1.3 MEMORY
The memory contains 120 memory cells. Each memory cell consists of a 16-dimensional address part and 28-dimensional content part.
A.1.4 TRAINING DETAILS
We set aside a random 10% of the training examples as a validation set for each sub-task and use it for early-stopping and hyperparameter search. We train one D-NTM for each sub-task, using Adam (Kingma & Ba, 2014) with its learning rate set to 0.003 and 0.007 respectively for GRU and Feedforward controller. The size of each minibatch is 160, and each minibatch is constructed uniform-randomly from the training set.
A.2 MODEL AND TRAINING DETAILS FOR SEQUENTIAL MNIST
On sequential MNIST task we try to keep the capacity of our model to be close to our baselines. We use 100 GRU units in the controller and each content vector of size 8 and with address vectors of size 8. We use a learning rate of 1e − 3 and trained the model with adam optimizer. We did not use the read and write consistency regularization in any of our models.
A.3 MODEL AND TRAINING DETAILS FOR TOY TASKS
On both copy and associative recall tasks, we try to keep the capacity of our model to be close to our baselines. We use 100 GRU units in the controller and each content vector of has a size of 8 and using address vector of size 8. We use a learning rate of 1e− 3 and trained the model with adam optimizer. We did not use the read and write consistency regularization in any of our models. For the model with the discrete attention we use REINFORCE with baseline computed using moving averages.
B VISUALIZATION OF DISCRETE ATTENTION
We visualize the attention of D-NTM with GRU controller with discrete attention in Figure 2. From this example, we can see that D-NTM has learned to find the correct supporting fact even without any supervision for the particular story in the visualization.
C LEARNING CURVES FOR THE RECURRENT CONTROLLER
In Figure 3, we compare the learning curves of the continuous and discrete attention D-NTM model with recurrent controller on Task 1. Surprisingly, the discrete attention D-NTM converges faster than the continuous-attention model. The main difficulty of learning continuous-attention is due to the fact that learning to write with continuous-attention can be challenging.
D A COMPARISON BETWEEN THE LEARNING
CURVES OF INPUT BASED BASELINE AND REGULAR BASELINE ON pMNIST
In Figure 4, we show the learning curves of input-based-baseline (ibb) and regular REINFORCE with moving averages baseline (mab) on the pMNIST task. We observe that input-based-baseline in general is much easier to optimize and converges faster as well. But it can quickly overfit to the task as well.
E TRAINING WITH CONTINUOUS-ATTENTION AND TESTING WITH DISCRETE-ATTENTION
In Table 5, we provide results investigating the effects of using discrete attention model at the test-time for a model trained with feed-forward controller and continuous attention. Discrete∗ D-NTM model bootstraps the discrete attention with the continuous attention, using the curriculum method that we have
introduced in Section ”Curriculum Learning for the Discrete Attention”. Discrete† D-NTM model is the continuous-attention model which uses discrete-attention at the test time. We observe that the Discrete† D-NTM model which is trained with continuous-attention outperforms Discrete D-NTM model.
F D-NTM WITH BOW FACT REPRESENTATION
In Table 6, we provide results for D-NTM using BoW with positional encoding (PE) Sukhbaatar et al. (2015) as the representation of the input facts. The facts representations are provided as an input to the GRU controller. In agreement to our results with the GRU fact representation, with the BoW fact representation we observe improvements with multi-step of addressing over single-step and discrete addressing over continuous addressing.

The model described in his paper is quite complicated and reproducing the method from its description may be challenging: Are you planing to release the code and what is your estimate release date?
Different cost functions and regularization are introduce in the paper. Would it be possible to summarize the overall cost function minimized by this model?
Some variables seems to have different definition. In particular w_t and b, would it be possible to clarify this?
“gamma_t is a shallow MLP”: What does it mean? Is gamma_t a function or a variable? It seems from eq. (10) that it is a vector (or a scalar?).
"curriculum learning for the discrete attention": Can you compare this to simpler schemes? Like rounding the continuous attention?
In your introduction, you state that "it is possible to use the discrete non-differentiable attention mechanism", referencing Zaremba & Sutskever, 2016 that use REINFORCE and a simple controller. However, in your sec. 4, you are stating that  "Training discrete attention with feed-forward controller and REINFORCE is challenging". It seems that these two statements contradict each other, could you comment on it?
In the introduction, you state that "Memory network (Weston et al. 2015b) [..] uses an attention-based mechanism to index them". To the best of my knowledge, it is actually Sukhbaatar et al., 2015 that has introduce the attention mechanism to memory networks.
In the introduction, the authors state that “memory networks […] [are] used in real tasks (Bordes et al. 2015, Dodge et al. 2015)”. However in both papers, they use memory network systems different from Weston et al. The current statement is quite misleading, would it be possible to clarify it?


Classifying products into categories precisely and efficiently is a major challenge in modern e-commerce. The high traffic of new products uploaded daily and the dynamic nature of the categories raise the need for machine learning models that can reduce the cost and time of human editors. In this paper, we propose a decision level fusion approach for multi-modal product classification using text and image inputs. We train input specific state-of-the-art deep neural networks for each input source, show the potential of forging them together into a multi-modal architecture and train a novel policy network that learns to choose between them. Finally, we demonstrate that our multi-modal network improves the top-1 accuracy % over both networks on a real-world large-scale product classification dataset that we collected from Walmart.com. While we focus on image-text fusion that characterizes e-commerce domains, our algorithms can be easily applied to other modalities such as audio, video, physical sensors, etc.
1 INTRODUCTION
Product classification is a key issue in e-commerce domains. A product is typically represented by metadata such as its title, image, color, weight and so on, and most of them are assigned manually by the seller. Once a product is uploaded to an e-commerce website, it is typically placed in multiple categories. Categorizing products helps e-commerce websites to provide costumers a better shopping experience, for example by efficiently searching the products catalog or by developing recommendation systems. A few examples of categories are internal taxonomies (for business needs), public taxonomies (such as groceries and office equipment) and the product’s shelf (a group of products that are presented together on an e-commerce web page). These categories vary with time in order to optimize search efficiency and to account for special events such as holidays and sports events. In order to address these needs, e-commerce websites typically hire editors and use crowdsourcing platforms to classify products. However, due to the high amount of new products uploaded daily and the dynamic nature of the categories, machine learning solutions for product classification are very appealing as means to reduce the time and economic costs. Thus, precisely categorizing items emerges as a significant issue in e-commerce domains.
A shelf is a group of products presented together on an e-commerce website page, and usually contain products with a given theme/category (e.g., Women boots, folding tables). Product to shelf classification is a challenging problem due to data size, category skewness, and noisy metadata and labels. In particular, it presents three fundamental challenges for machine learning algorithms. First, it is typically a multi-class problem with thousands of classes. Second, a product may belong to multiple shelves making it a multi-label problem. And last, a product has both an image and a text input making it a multi-modal problem.
Products classification is typically addressed as a text classification problem because most metadata of items are represented as textual features (Pyo et al., 2010). Text classification is a classic topic for natural language processing, in which one needs to assign predefined categories to text inputs.
Standard methods follow a classical two-stage scheme of extraction of (handcrafted) features, followed by a classification stage. Typical features include bag-of-words or n-grams, and their TF-IDF. On the other hand, Deep Neural Networks use generic priors instead of specific domain knowledge (Bengio et al., 2013) and have been shown to give competitive results on text classification tasks (Zhang et al., 2015). In particular, Convolutional neural networks (CNNs) (Kim, 2014; Zhang et al., 2015; Conneau et al., 2016) and Recurrent NNs (Lai et al., 2015; Pyo et al., 2010; Xiao & Cho, 2016) can efficiently capture the sequentiality of the text. These methods are typically applied directly to distributed embedding of words (Kim, 2014; Lai et al., 2015; Pyo et al., 2010) or characters (Zhang et al., 2015; Conneau et al., 2016; Xiao & Cho, 2016), without any knowledge on the syntactic or semantic structures of a language. However, all of these architectures were only applied on problems with a small amount of labels (∼ 20) while e-commerce shelf classification problems typically have thousands of labels with multiple labels per product.
In Image classification, CNNs are widely considered the best models, and achieve state-of-theart results on the ImageNet Large-Scale Visual Recognition Challenge (Russakovsky et al., 2015; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2015). However, as good as they are, the classification accuracy of machine learning systems is often limited in problems with many classes of object categories. One remedy is to leverage data from other sources, such as text data. However, the studies on multi-modal deep learning for large-scale item categorization are still rare to the best of our belief. In particular in a setting where there is a significant difference in discriminative power between the two types of signals.
In this work, we propose a multi-modal deep neural network model for product classification. Our design principle is to leverage the specific prior for each data type by using the current state-of-
the-art classifiers from the image and text domains. The final architecture has 3 main components (Figure 2, Right): a text CNN (Kim, 2014), an image CNN (Simonyan & Zisserman, 2014) and a policy network that learns to choose between them. We collected a large-scale data set of 1.2 million products from the Walmart.com website. Each product has a title and an image and needs to be classified to a shelf (label) with 2890 possible shelves. Examples from this dataset can be seen in Figure 1 and are also available on-line at the Walmart.com website. For most of the products, both the image and the title of each product contain relevant information for customers. However, it is interesting to observe that for some of the products, both input types may not be informative for shelf prediction (Figure 1). This observation motivates our work and raises interesting questions: which input type is more useful for product classification? is it possible to forge the inputs into a better architecture?
In our experiments, we show that the text CNN outperforms the image one. However, for a relatively large number of products (∼ 8%), the image CNN is correct while the text CNN is wrong, indicating a potential gain from using a multi-modal architecture. We also show that the policy is able to choose between the two models and give a performance improvement over both state-of-the-art networks.
To the best of our knowledge, this is the first work that demonstrates a performance improvement on top-1 classification accuracy by using images and text on a large-scale classification problem. In particular, our main contributions are:
• We demonstrate that the text classification CNN (Kim, 2014) outperforms the VGG network (Simonyan & Zisserman, 2014) on a real-world large-scale product to shelf classification problem.
• We analyze the errors made by the different networks and show the potential gain of multimodality.
• We propose a novel decision-level fusion policy that learns to choose between the text and image networks and improve over both.
2 MULTI-MODALITY
Over the years, a large body of research has been devoted to improving classification using ensembles of classifiers (Kittler et al., 1998; Hansen & Salamon, 1990). Inspired by their success, these methods have also been used in multi-modal settings (e.g.,Guillaumin et al. (2010); Poria et al. (2016)), where the source of the signals, or alternatively their modalities, are different. Some examples include audio-visual speech classification (Ngiam et al., 2011), image and text retrieval (Kiros et al.), sentiment analysis and semi-supervised learning (Guillaumin et al., 2010).
Combining classifiers from different input sources presents multiple challenges. First, classifiers vary in their discriminative power, thus, an optimal unification method should be able to adapt itself for specific combinations of classifiers. Second, different data sources have different state-ofthe-art architectures, typically deep neural networks, which vary in depth, width, and optimization algorithm; making it non-trivial to merge them. Moreover, a multi-modal architecture potentially has more local minima that may give unsatisfying results. Finally, most of the publicly available real-world big data classification datasets, an essential building block of deep learning systems, typically contain only one data type.
Nevertheless, the potential performance boost of multi-modal architectures has motivated researchers over the years. Frome et al. (2013) combined an image network (Krizhevsky et al., 2012) with a Skip-gram Language Model in order to improve classification results on ImageNet. However, they were not able to improve the top-1 accuracy prediction, possibly because the text input they used (image labels) didn’t contain a lot of information. Other works, used multi-modality to learn good embedding but did not present results on classification benchmarks (Lynch et al., 2015; Kiros et al.; Gong et al., 2014). Kannan et al. (2011) suggested to improve text-based product classification by adding an image signal, training an image classifier and learning a decision rule between the two. However, they only experimented with a small dataset and a low number of labels, and it is not clear how to scale their method for extreme multi-class multi-label applications that characterize real-world problems in e-commerce.
Adding modalities can improve the classification of products that have a non-informative input source (e.g., image or text). In e-commerce, for example, classifiers that rely exclusively on text suffer from short and non-informative titles, differences in style between vendors and overlapping text across categories (i.e., a word that helps to classify a certain class may appear in other classes). Figure 1 presents a few examples of products that have only one informative input type. These examples suggest that a multi-modal architecture can potentially outperform a classifier with a single input type.
Most unification techniques for multi-modal learning are partitioned between feature-level fusion techniques and decision-level fusion techniques (Figure 2, Left).
2.1 FEATURE LEVEL FUSION
Feature-level fusion is characterized by three phases: (a) learning a representation, (b) supervised training, and (c) testing. The different unification techniques are distinguished by the availability of the data in each phase (Guillaumin et al., 2010). For example, in cross-modality training, the representation is learned from all the modalities, but only one modality is available for supervised training and testing. In other cases, all of the modalities are available at all stages but we may want (or not) to limit their usage given a certain budget. Another source for the distinction is the order in which phases (a) and (b) are made. For example, one may first learn the representation and then learn a classifier from it, or learn both the representation and the classifier in parallel. In the deep learning context, there are two common approaches. In the first approach, we learn an end-to-end deep NN; the NN has multiple input-specific pipes that include a data source followed by input specific layers. After a certain depth, the pipes are concatenated followed by additional layers such that the NN is trained end-to-end. In the second approach, input specific deep NNs are learned first, and a multi-modal representation vector is created by concatenating the input specific feature vectors (e.g., the neural network’s last hidden layer). Then, an additional classifier learns to classify from the multi-modal representation vector. While multi-modal methods have shown potential to boost performance on small datasets (Poria et al., 2016), or on top-k accuracy measures (Frome et al., 2013), we are not familiar with works that succeeded with applying it on a large-scale classification problem and received performance improvement in top-1 accuracy.
2.2 DECISION-LEVEL FUSION
In this approach, an input specific classifier is learned for each modality, and the goal is to find a decision rule between them. The decision rule is typically a pre-defined rule (Guillaumin et al., 2010) and is not learned from the data. For example, Poria et al. (2016) chose the classifier with the maximal confidence, while Krizhevsky et al. (2012) average classifier predictions. However, in this work we show that learning the decision rule yields significantly better results on our data.
3 METHODS AND ARCHITECTURES
In this section, we give the details of our multi-modal product classification architecture. The architecture is composed of a text CNN and an image CNN which are forged together by a policy network, as can be seen in Figure 2, Right.
3.1 MULTI LABEL COST FUNCTION
Our cost function is the weighted sigmoid cross entropy with logits, a common cost function for multi-label problems. Let x be the logits, z be the targets, q be a positive weight coefficient, used as a multiplier for the positive targets, and σ(x) = 11+exp(−x) . The loss is given by:
Cost(x,z;q) = − qz · log(σ(x))− (1− z) · log(1− σ(x)) = (1− z) · x+ (1 + (q − 1) · z) · log(1 + exp(−x)).
The positive coefficient q, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error. We found it to have a significant effect in practice.
3.2 TEXT CLASSIFICATION
For the text signal, we use the text CNN architecture of Kim (2014). The first layer embeds words into low-dimensional vectors using random embedding (different than the original paper). The next layer performs convolutions over time on the embedded word vectors using multiple filter sizes (3, 4 and 5), where we use 128 filters from each size. Next, we max-pool-over-time the result of each convolution filter and concatenated all the results together. We add a dropout regularization layer (0.5 dropping rate), followed by a fully connected layer, and classify the result using a softmax layer. An illustration of the Text CNN can be seen in Figure 2.
3.3 IMAGE CLASSIFICATION
For the image signal, we use the VGG Network (Simonyan & Zisserman, 2014). The input to the network is a fixed-size 224 x 224 RGB image. The image is passed through a stack of convolutional layers with a very small receptive field: 3 x 3. The convolution stride is fixed to 1 pixel; the spatial padding of the convolutional layer is 1 pixel. Spatial pooling is carried out by five max-pooling layers, which follow some of the convolutional layers. Max-pooling is performed over a 2 x 2 pixel window, with stride 2. A stack of convolutional layers is followed by three Fully-Connected (FC) layers: the first two have 4096 channels each, the third performs 2890-way product classification and thus contains 2890 channels (one for each class). All hidden layers are followed by a ReLu non-linearity. The exact details can be seen in Figure 2.
3.4 MULTI-MODAL ARCHITECTURE
We experimented with four types of multi-modal architectures. (1) Learning decision-level fusion policies from different inputs. (1a) Policies that use the text and image CNNs class probabilities as input (Figure 2). We experimented with architectures that have one or two fully connected layers (the two-layered policy is using 10 hidden units and a ReLu non-linearity between them). (1b) Policies that use the text and/or image as input. For these policies, the architecture of policy network was either the text CNN or the VGG network. In order to train policies, labels are collected from the image and text networks predictions, i.e., the label is 1 if the image network made a correct prediction while the text network made a mistake, and 0 otherwise. On evaluation, we use the policy predictions to select between the models, i.e., if the policy prediction is 1 we use the image network, and use the text network otherwise. (2) Pre-defined policies that average the predictions of the different CNNs or choose the CNN with the highest confidence. (3) End-to-end feature-level fusion, each input type is processed by its specific CNN. We concatenate the last hidden layers of the CNNs and add one or two fully connected layers. All the layers are trained together end-to-end (we also tried to initialize the input specific weights from pre-trained single-modal networks). (4) Multistep feature-level fusion. As in (3), we create shared representation vector by concatenating the last hidden layers. However, we now keep the shared representation fixed and learn a new classifier from it.
4 EXPERIMENTS
4.1 SETUP
Our dataset contains 1.2 million products (title image and shelf) that we collected from Walmart.com (offered online and can be viewed at the website) and were deemed the hardest to classify by the current production system. We divide the data into training (1.1 million) validation (50k) and test (50k). We train both the image network and the text network on the training dataset and evaluate them on the test dataset. The policy is trained on the validation dataset and is also evaluated on the test dataset. The objective is to classify the product’s shelf, from 2890 possible choices. Each product is typically assigned to more than one shelf (3 on average), and the network is considered accurate if its most probable shelf is one of them.
4.2 TRAINING THE TEXT ARCHITECTURE
Preprocess: we build a dictionary of all the words in the training data and embed each word using a random embedding into a one hundred dimensional vector. We trim titles with more than 40 words and pad shorter titles with nulls.
We experimented with different batch sizes, dropout rates, and filters stride, but found that the vanilla architecture (Kim, 2014) works well on our data. This is consistent with Zhang & Wallace (2015), who showed that text CNNs are not very sensitive to hyperparameters. We tuned the cost function positive coefficient parameter q, and found out that the value 30 performed best in practice (we will also use this value for the image network). The best CNN that we trained classified 70.1% of the products from the test set correctly (Table 1).
4.3 TRAINING THE IMAGE ARCHITECTURE
Preprocess: we re-size all the images into 224 x 224 pixels and reduce the image mean.
The VGG network that we trained classified 57% of the products from the test set correctly. This is a bit disappointing if we compare it to the performance of the VGG network on ImageNet (∼ 75%). There are a few differences between these two datasets that may explain this gap. First, our data has 3 times more classes and contains multiple labels per image making the classification harder, and second, Figure 1 implies that some of our images are not informative for shelf classification. Some works claim that the features learned by VGG on ImageNet are global feature extractors (Lynch et al., 2015). We therefore decided to use the weights learned by VGG on ImageNet and learn only the last layer. This configuration yielded only 36.7% accuracy. We believe that the reason is that some of the ImageNet classes are irrelevant for e-commerce (e.g., vehicles and animals) while some relevant categories are misrepresented (e.g., electronics and office equipment). It could also be that our images follow some specific pattern of white background, well-lit studio etc., that characterizes e-commerce.
4.4 ERROR ANALYSIS
Is a picture worth a thousand words? Inspecting Figure 3, we can see that the text network outperformed the image network on this dataset, classifying more products correctly. Similar results were reported before (Pyo et al., 2010; Kannan et al., 2011) but to the best of our knowledge, this is the first work that compares state-of-the-art text and image CNNs on a real-world large-scale ecommerce dataset. What is the potential of multi-modality? We identified that for 7.8% of the products the image network made a correct prediction while the text network was wrong. This observation is encouraging since it implies that there is a relative big potential to harness via multi-modality. We find this large gap surprising since different neural networks applied to the same problem tend to make the same mistakes (Szegedy et al., 2013). Unification techniques for multi-modal problems typically use the last hidden layer of each network as features (Frome et al., 2013; Lynch et al., 2015; Pyo et al., 2010). We therefore decided to visualize the activations of this layer using a tSNE map (Maaten & Hinton, 2008). Figure 3, depicts such a map for the activations of the text model (the image model yielded similar results). In particular,
we were looking for regions in the tSNE map where the image predictions are correct and the text is wrong (Figure 3, green). Finding such a region will imply that a policy network can learn good decision boundaries. However, we can see that there are no well-defined regions in the tSNE maps where the image network is correct and the title is wrong (green), thus implying that it might be hard to identify these products using the activations of the last layers.
4.5 MULTI-MODAL UNIFICATION TECHNIQUES
Our error analysis experiment highlights the potential of merging image and text. Still, we found it hard to achieve the upper bound provided by the error analysis in practice. We now describe the policies that managed to achieve performance boost in top-1 accuracy % over the text and image networks, and then provide discussion on other approaches that we tried but didn’t work.
Decision-level fusion: We trained policies from different data sources (e.g., title, image, and each CNN class probabilities), using different architectures and different hyperparameters. Looking at Table 1, we can see that the best policies were trained using the class probabilities (the softmax probabilities) of the image and text CNNs as inputs. The amount of class probabilities that were used (top-1, top-3 or all) did not have a significant effect on the results, indicating that the top-1 probability contains enough information to learn good policies. This result makes sense since the top-1 probability measures the confidence of the network in making a prediction. Still, the top-3 probabilities performed slightly better, indicating that the difference between the top probabilities may also matter. We can also see that the 2-layer architecture outperformed the 1-layer, indicating that a linear policy is too simple, and deeper models can yield better results. Last, the cost function positive coefficient q had a big impact on the results. We can see that for q = 1, the policy network is more accurate in its prediction however it achieves worse results on shelf classification. For q = 5 we get the best results, while higher values of q (e.g., 7 or 10) resulted in inaccurate policies that did not perform well in practice.
While it may not seem surprising that combining text and image will improve accuracy, in practice we found it extremely hard to leverage this potential. To the best of our knowledge, this is the first work that demonstrates a direct performance improvement on top-1 classification accuracy from using images and text on a large-scale classification problem. We experimented with pre-defined policies that do not learn from the data. Specifically, we tried to average the logits, following (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), and to choose the network with the maximal confidence following (Poria et al., 2016). Both of these experiments yielded significantly worse results, probably, since the text network is much more accurate than the image one (Table 1). We also tried to learn policies from the text and/or the image input, using a policy network which is either a text CNN, a VGG network or a combination. However, all of these experiments resulted in policies that overfit the data and performed worse than the title model on the test data (Table 1). We also experimented with early stopping criteria, various regularization methods (dropout, l1, l2) and reduced model size but none could make the policy network generalize.
Feature-level fusion: Training a CNN end-to-end can be tricky. For example, each input source has its own specific architecture, with specific learning rate and optimization algorithm. We experimented with training the network end-to-end, but also with first training each part separately and then learning the concatenated parts. We tried different unification approaches such as gating functions (Srivastava et al., 2015), cross products and a different number of fully connected layers after the concatenation. These experiments resulted in models that were inferior to the text model. While this may seem surprising, the only successful feature level fusion that we are aware of (Frome et al., 2013), was not able to gain accuracy improvement on top-1 accuracy.
5 CONCLUSIONS
In this work, we investigated a multi-modal multi-class multi-label product classification problem and presented results on a challenging real-world dataset that we collected from Walmart.com. We discovered that the text network outperforms the image network on our dataset, and observed a big potential of fusing text and image inputs. Finally, we suggested a multi-modal decision-level fusion approach that leverages state-of-the-art results from image and text classification and forges them into a multi-modal architecture that outperforms both.
State-of-the-art image CNNs are much larger than text CNNs, and take more time to train and to run. Thus, extracting image features during run time, or getting the image network predictions may be prohibitively expensive. In this context, an interesting observation is that feature level fusion methods require using the image signal for each product, while decision level fusion methods require using the image network selectively making them more appealing. Moreover, our experiments suggest that decision-level fusion performs better than feature-level fusion in practice.
Finally, we were only able to realize a fraction of the potential of multi-modality. In the future, we plan to investigate deeper policy networks and more sophisticated measures of confidence. We also plan to investigate ensembles of image networks (Krizhevsky et al., 2012) and text networks (Pyo et al., 2010). We believe that the insights from training policy networks will eventually lead us to train end to end differential multi-modal networks.

This paper introduces a large-scale multi-model product classification system. The model consists of three modules, Image CNN (VGG 16 architecture), text CNN (Kim 2014) and decision-level fusion policies. The authors have tried several fusion methods: including policies taking inputs from text and image CNN probabilities; choose either CNN; average the predictions; end-to-end training. Experimental results show that text CNN alone works better than image CNN and multi-model fusion can improve the accuracy by a small margin. It is a little bit surprising that end-to-end feature level fusion works worse than text CNN alone. The writing is clear and there are a lot of useful practical experiences of learning large-scale model. However, I lean toward rejecting the paper because the following:
1) No other dataset reported. The authors haven't mentioned releasing the walmart dataset and it is going to be really hard to reproduce the results without the dataset.
2) Technical novelty is limited. All the decision-level fusion policies have been investigated by some previous methods before.
3) Performance gain is also limited.