Many powerful machine learning models are based on the composition of multiple processing layers, such as deep nets, which gives rise to nonconvex objective functions. A general, recent approach to optimise such “nested” functions is the method of auxiliary coordinates (MAC). MAC introduces an auxiliary coordinate for each data point in order to decouple the nested model into independent submodels. This decomposes the optimisation into steps that alternate between training single layers and updating the coordinates. It has the advantage that it reuses existing single-layer algorithms, introduces parallelism, and does not need to use chain-rule gradients, so it works with nondifferentiable layers. We describe ParMAC, a distributed-computation model for MAC. This trains on a dataset distributed across machines while limiting the amount of communication so it does not obliterate the benefit of parallelism. ParMAC works on a cluster of machines with a circular topology and alternates two steps until convergence: one step trains the submodels in parallel using stochastic updates, and the other trains the coordinates in parallel. Only submodel parameters, no data or coordinates, are ever communicated between machines. ParMAC exhibits high parallelism, low communication overhead, and facilitates data shuffling, load balancing, fault tolerance and streaming data processing. We study the convergence of ParMAC and its parallel speedup, and implement ParMAC usingMPI to learn binary autoencoders for fast image retrieval, achieving nearly perfect speedups in a 128-processor cluster with a training set of 100 million high-dimensional points.
Many powerful machine learning models are based on the composition of multiple processing layers, such as deep nets, which gives rise to nonconvex objective functions. A general, recent approach to optimise such “nested” functions is the method of auxiliary coordinates (MAC). MAC introduces an auxiliary coordinate for each data point in order to decouple the nested model into independent submodels. This decomposes the optimisation into steps that alternate between training single layers and updating the coordinates. It has the advantage that it reuses existing single-layer algorithms, introduces parallelism, and does not need to use chain-rule gradients, so it works with nondifferentiable layers. We describe ParMAC, a distributed-computation model for MAC. This trains on a dataset distributed across machines while limiting the amount of communication so it does not obliterate the benefit of parallelism. ParMAC works on a cluster of machines with a circular topology and alternates two steps until convergence: one step trains the submodels in parallel using stochastic updates, and the other trains the coordinates in parallel. Only submodel parameters, no data or coordinates, are ever communicated between machines. ParMAC exhibits high parallelism, low communication overhead, and facilitates data shuffling, load balancing, fault tolerance and streaming data processing. We study the convergence of ParMAC and its parallel speedup, and implement ParMAC usingMPI to learn binary autoencoders for fast image retrieval, achieving nearly perfect speedups in a 128-processor cluster with a training set of 100 million high-dimensional points.
1 INTRODUCTION
Serial computing has reached a plateau and parallel, distributed architectures are becoming widely available, from machines with a few cores to cloud computing with 1000s of machines. The combination of powerful nested models with large datasets is a key ingredient to solve difficult problems in machine learning, computer vision and other areas, and it underlies recent successes in deep learning (Hinton et al., 2012; Le et al., 2012; Dean et al., 2012). Unfortunately, parallel computation is not easy, and many good serial algorithms do not parallelise well. The cost of communicating (through the memory hierarchy or a network) greatly exceeds the cost of computing, both in time and energy, and will continue to do so for the foreseeable future. Thus, good parallel algorithms must minimise communication and maximise computation per machine, while creating sufficiently many subproblems (ideally independent) to benefit from as many machines as possible. The load (in runtime) on each machine should be approximately equal. Faults become more frequent as the number of machines increases, particularly if they are inexpensivemachines. Machines may be heterogeneous and differ in CPU and memory; this is the case with initiatives such as SETI@home, which may become an important source of distributed computation in the future. Big data applications have additional restrictions. The size of the data means it cannot be stored on a single machine, so distributedmemory architectures are necessary. Sending data between machines is prohibitive because of the size of the data and the high communication costs. In some applications, more data is collected than can be stored, so data must be regularly discarded. In others, such as sensor networks, limited battery life and computational power imply that data must be processed locally.
In this paper, we focus on machine learning models of the form y = FK+1(. . .F2(F1(x)) . . . ), i.e., consisting of a nested mapping from the inputx to the outputy. Such nested models involvemultiple parameterised layers of processing and include deep neural nets, cascades for object recognition in computer vision or for phoneme classification in speech processing, wrapper approaches to classification or regression, and various combinations of feature extraction/learning and preprocessing prior to some learning task. Nested and hierarchical models are ubiquitous in machine learning because they provide a way to construct complex models by the composition of simple layers. However, training nested models is difficult even in the serial case because function composition produces inherently nonconvex functions, which makes gradient-based optimisation difficult and slow, and sometimes inapplicable (e.g. with nonsmooth or discrete layers).
Our starting point is a recently proposed technique to train nested models, the method of auxiliary coordinates (MAC) (Carreira-Perpiñán and Wang, 2012; 2014). This reformulates the optimisation into an iterative procedure that alternates training submodels independently with coordinating them. It introduces significant model and data parallelism, can often train the submodels using existing algorithms, and has convergence guarantees with differentiable functions to a local stationary point, while it also applies with nondifferentiable or even discrete layers. MAC has been applied to various nested models (Carreira-Perpiñán and Wang, 2014; Wang and Carreira-Perpiñán, 2014; Carreira-Perpiñán and Raziperchikolaei, 2015; Raziperchikolaei and Carreira-Perpiñán, 2016; Carreira-Perpiñán and Vladymyrov, 2015). However, the original papers proposing MAC (Carreira-Perpiñán and Wang, 2012; 2014) did not address how to run MAC on a distributed computing architecture, where communication between machines is far costlier than computation. This paper proposes ParMAC, a parallel, distributed framework to learn nested models using MAC, analyses its parallel speedup and convergence, implements it in MPI for the problem of learning binary autoencoders, and demonstrates its ability to train on large datasets and achieve large speedups on a distributed cluster.
Related work Distributed optimisation and large-scale machine learning have been steadily gaining interest in recent years, with the development of parallel computation abstractions tailored to machine learning, such as Spark (Zaharia et al., 2010), GraphLab (Low et al., 2012), Petuum (Xing et al., 2015) or TensorFlow (Abadi et al., 2015), which have the goal of making cloud computing easily available to train machine learning models. Most work has centred on convex optimisation, particularly when the objective function has the form of empirical risk minimisation (data fitting term plus regulariser) (Cevher et al., 2014). This includes many important models in machine learning, such as linear regression, LASSO, logistic regression or SVMs. Such work is typically based on stochastic gradient descent (SGD) (Bottou, 2010), coordinate descent (CD) (Wright, 2016) or the alternating direction method of multipliers (ADMM) (Boyd et al., 2011). This has resulted in several variations of parallel SGD (Bertsekas, 2011; Zinkevich et al., 2010; Niu et al., 2011), parallel CD (Bradley et al., 2011; Richtárik and Takáč, 2013; Liu and Wright, 2015) and parallel ADMM (Boyd et al., 2011; Ouyang et al., 2013; Zhang and Kwok, 2014).
Little work has addressed nonconvex models. Most of it has focused on deep nets (Dean et al., 2012; Le et al., 2012). Google’s DistBelief (Dean et al., 2012) uses asynchronous parallel SGD (with gradients for the full model computed with backpropagation) to achieve data parallelism, and some form of model parallelism. The latter is achieved by carefully partitioning the neural net into pieces and allocating them to machines to compute gradients. This is difficult to do and requires a careful match of the neural net structure (number of layers and hidden units, connectivity, etc.) to the target hardware. Also, parallel SGD can diverge with nonconvex models, which requires heuristics to make sure we average replica models that are close in parameter space and thus associated with the same optimum. Although this has managed to train huge nets on huge datasets by using tens of thousands of CPU cores, the speedups achieved were very modest. Other work has used similar techniques but for GPUs (Coates et al., 2013; Seide et al., 2014).
Finally, there also exist specific approximation techniques for certain types of large-scale machine learning problems, such as spectral problems, using the Nyström formula or other landmarkbased methods (Williams and Seeger, 2001; Bengio et al., 2004; Drineas and Mahoney, 2005; Talwalkar et al., 2008; Vladymyrov and Carreira-Perpiñán, 2013; 2016).
ParMAC is specifically designed for nested models, which are typically nonconvex and include deep nets and many other models, some of which have nondifferentiable layers. As we describe below, ParMAC has the advantages of being simple and relatively independent of the target hardware, while achieving high speedups.
2 OPTIMISING NESTED MODELS USING AUXILIARY COORDINATES (MAC)
Many optimisation problems in machine learning involve mathematically “nested” functions of the form F(x;W) = FK+1(. . .F2(F1(x;W1);W2) . . . ;WK+1) with parameters W, such as deep nets. Such problems are traditionally optimised using methods based on gradients computed using the chain rule. However, such gradients may sometimes be inconvenient to use, or may not exist (e.g. if some of the layers are nondifferentiable, as with binary autoencoders). Also, they are hard to parallelise, because of the inherent sequentiality in the chain rule. The method of auxiliary coordinates (MAC) (Carreira-Perpiñán and Wang, 2012; 2014) is designed to optimise nested models without using chain-rule gradients while introducing parallelism. The idea is to break nested functional relationships judiciously by introducing new variables (the auxiliary coordinates) as equality constraints. These are then solved by optimising a penalised function using alternating optimisation over the original parameters (which we call the W step) and over the coordinates (which we call the Z step). The result is a coordination-minimisation (CM) algorithm: the minimisation (W) step updates the parameters by splitting the nested model into independent submodels and training them using existing algorithms, and the coordination (Z) step ensures that corresponding inputs and outputs of submodels eventually match. MAC algorithms have been developed for several nested models so far: deep nets (Carreira-Perpiñán and Wang, 2014), low-dimensional SVMs (Wang and Carreira-Perpiñán, 2014), binary autoencoders (Carreira-Perpiñán and Raziperchikolaei, 2015), affinity-based loss functions for binary hashing (Raziperchikolaei and Carreira-Perpiñán, 2016) and parametric nonlinear embeddings (Carreira-Perpiñán and Vladymyrov, 2015). Although this paper proposes and analyses ParMAC in general, our MPI implementation is for the particular case of binary autoencoders. These define a nonconvex nondifferentiable problem, yet its MAC algorithm is simple and effective.
MAC algorithm for binary autoencoders A binary autoencoder (BA) is a usual autoencoder but with a binary code layer. It consists of an encoder h(x) that maps a real vector x ∈ RD onto a binary code vector with L < D bits, z ∈ {0, 1}L, and a linear decoder f(z) which maps z back to RD in an effort to reconstruct x. We will call h a binary hash function (see later). Let us write h(x) = (Ax) (A includes a bias by having an extra dimension x0 = 1 for each x) where A ∈ RL×(D+1) and (t) is a step function applied elementwise, i.e., (t) = 1 if t ≥ 0 and (t) = 0 otherwise. Given a dataset of D-dimensional patterns X = (x1, . . . ,xN ), our objective function, which involves the nested model y = f(h(x)), is the usual least-squares reconstruction error EBA(h, f) = ∑N n=1 ‖xn − f(h(xn))‖ 2 . Optimising this nonconvex, nonsmooth function is NP-complete. Where the gradients do exist wrt A they are zero, so optimisation of h using chainrule gradients does not apply. We introduce as auxiliary coordinates the outputs of h, i.e., the codes for each of the N input patterns, and obtain the following equality-constrained problem:
min h,f ,Z
N ∑
n=1
‖xn − f(zn)‖2 s.t. zn = h(xn), zn ∈ {0, 1}L, n = 1, . . . , N. (1)
Note the codes are binary. We now apply the quadratic-penalty method and minimise the following objective function while progressively increasing µ, so the constraints are eventually satisfied:
EQ(h, f ,Z;µ) =
N ∑
n=1
‖xn − f(zn)‖2 + µ ‖zn − h(xn)‖2 s.t. zn ∈ {0, 1}L, n = 1, . . . , N. (2)
Finally, we apply alternating optimisation over Z andW = (h, f). This gives the following steps:
• Over Z for fixed (h, f), this is a binary optimisation on NL variables, but it separates into N independent optimisations each on only L variables, with the form of a binary proximal operator (where we omit the index n): minz ‖x− f(z)‖2 + µ‖z− h(x)‖2 s.t. z ∈ {0, 1}L. This can be solved approximately by alternating optimisation over bits.
• Over W = (h, f) for fixed Z, we obtain L + D independent problems: for each of the L single-bit hash functions (which try to predict Z optimally from X), each solvable by fitting a linear SVM; and for each of theD linear decoders in f (which try to reconstructX optimally from Z), each a linear least-squares problem.
The user must choose a schedule for the penalty parameter µ (sequence of values 0 < µ1 < · · · < µ∞). This should increase slowly enough that the binary codes can change considerably and explore
better solutions before the constraints are satisfied and the algorithm stops. With BAs, MAC stops for a finite value of µ, which occurs whenever Z does not change compared to the previous Z step. This gives a practical stopping criterion. Carreira-Perpiñán and Raziperchikolaei (2015) give proofs of these statements and further details about the algorithm. Fig. 1 gives the MAC algorithm for BAs.
The BA was proposed as a way to learn good binary hash functions for fast, approximate information retrieval (Carreira-Perpiñán and Raziperchikolaei, 2015). Binary hashing (Grauman and Fergus, 2013) has emerged in recent years as an effective way to do fast, approximate nearest-neighbour searches in image databases. The real-valued, high-dimensional image vectors are mapped onto a binary space with L bits and the search is performed there using Hamming distances at a vastly faster speed and smaller memory (e.g. N = 109 points with D = 500 take 2 TB, but only 8 GB using L = 64 bits, which easily fits in RAM). As shown by Carreira-Perpiñán and Raziperchikolaei (2015), training BAs with MAC beats approximate optimisation approaches such as relaxing the codes or the step function in the encoder, and yields state-of-the-art binary hash functions h in unsupervised problems, improving over established approaches such as iterative quantisation (ITQ) (Gong et al., 2013). We focus mostly on linear hash functions because these are, by far, the most used type of hash functions in the literature of binary hashing, due to the fact that computing the binary codes for a test image must be fast at run time.
MAC in general With a nested function with K layers, we can introduce auxiliary coordinates at each layer. For example, with a neural net, this decouples the weight vector of every hidden unit in theW step, which can be solved as a logistic regression (see Carreira-Perpiñán and Alizadeh, 2016). For a large net with a large dataset, this affords an enormous potential for parallel computation.
MAC and EM MAC is very similar to expectation-maximisation (EM) at a conceptual level. EM (McLachlan and Krishnan, 2008) applies generally to many probabilistic models. The resulting algorithm can be very different (e.g. EM for Gaussian mixtures vs EM for hidden Markov models), but it always alternates two steps that conceptually do the following. The E step updates in parallel the posterior probabilities. This separates over data points and is like the Z step in MAC, where the posterior probabilities are the auxiliary coordinates, and where the step may be in closed-form or require optimisation, depending on the model. The M step updates in parallel the “submodels”. For a mixture with M components, these are the M Gaussians (means, covariances, proportions). This separates over submodels and is like the W step in MAC. For BAs, the submodels are the L encoders (linear SVMs) and theD decoders (linear regressors); for a neural net, each weight vector of a hidden unit is a submodel (a logistic regressor). For Gaussian mixtures, the M step can be done exactly in one “epoch” because it is a simple average. For MAC, it usually requires optimisation, and so multiple epochs. In fact, ParMAC applies to EM by using e = 1 epoch: in the W step, the Gaussians visit each machine circularly and (their averages) are updated on its data; in the Z step, each machine updates its posterior probabilities.
In the rest of the paper, some readers may find this analogy useful and think of EM for Gaussian mixtures instead of MAC, replacing “submodels” and “auxiliary coordinates” in MAC with “Gaussians” and “posterior probabilities” in EM, respectively.
3 PARMAC: A PARALLEL, DISTRIBUTED COMPUTATION MODEL FOR MAC
A specific MAC algorithm depends on the model and objective function and on how the auxiliary coordinates are introduced. We can achieve steps that are closed-form, convex, nonconvex, binary, or others. However, we will assume the following always hold: (1) Separability over data points. In the Z step, the N subproblems for z1, . . . , zN are independent, one per data point. Each zn step depends on the current model. (2) Separability over submodels. In the W step, there are M independent submodels, where M depends on the problem. For example, M is the number of hidden units in a deep net, or the number of hash functions and linear decoders in a BA. Each submodel depends on all the data and coordinates. We now show how to turn this into a distributed, low-communication ParMAC algorithm.
The basic idea in ParMAC is as follows. With large datasets in distributed systems, it is imperative to minimise data movement over the network because the communication time generally far exceeds the computation time in modern architectures. In MAC we have 3 types of data: the original training data (X,Y), the auxiliary coordinates Z, and the model parameters (the submodels). Usually, the latter type is far smaller. In ParMAC, we never communicate training or coordinate data; each machine keeps a disjoint portion of (X,Y,Z) corresponding to a subset of the points. Only model parameters are communicated, during the W step, following a circular topology, which implicitly implements a stochastic optimisation. The model parameters are the hash functions h and the decoder f for BAs, and the weight vector wh of each hidden unit h for deep nets. Let us see this in detail (refer to fig. 2).
Assume we have P identical processing machines, each with its own memory and CPU, connected through a network in a circular unidirectional topology. Each machine stores a subset of the data points and corresponding coordinates (xn,yn, zn) such that the subsets are disjoint and their union is the entire data. Before the Z step starts, each machine contains all the (just updated) submodels. This means that in the Z step each machine processes its auxiliary coordinates {zn} independently of all other machines, i.e., no communication occurs. The W step is more subtle. At the beginning of the W step, each machine will contain all the submodels and its portion of the data and (just updated) coordinates. Each submodel must have access to the entire data and coordinates in order to update itself and, since the data cannot leave its home machine, the submodel must go to the data. We achieve this in the circular topology with an asynchronous processing, as follows. Each machine keeps a queue of submodels to be processed, and repeatedly performs the following operations: extract a submodel from the queue, process it on its data and send it to the machine’s successor
(which will insert it in its queue). If the queue is empty, the machine waits until it is nonempty. The queue of each machine is initialised with a portionM/P of submodels associated with that machine (e.g. in fig. 2, machine 1’s queue contains submodels 1–3, machine 2 submodels 4–6, etc.). Each submodel carries a counter that is initially 1 and increases every time it visits a machine. When it reaches P , the submodel has visited all machines in sequence and has completed an epoch. We repeat this for e epochs and, to ensure all machines have all final submodels before starting the Z step, we run a communication-only epoch e + 1 (without computation), where submodels simply move from machine to machine.
Since each submodel is updated as soon as it visits a machine, rather than computing the exact gradient once it has visited all machines and then take a step, the W step is really carrying out stochastic steps for each submodel. For example, if the update is done by a gradient step, we are actually implementing stochastic gradient descent (SGD) where the minibatches are of size N/P (or smaller, if we subdivide a machine’s data portion into minibatches, which should be typically the case in practice). From this point of view, we can regard the W step as doing SGD on each submodel in parallel by having each submodel visit the minibatches in each machine.
As described, and as implemented in our experiments, the entire model parameters are communicated e + 1 times in a MAC iteration if running e epochs in the W step. We can also run e epochs with only 2 rounds of communication by having a submodel do e consecutive passes within each machine’s data. This reduces the amount of shuffling, but should not be a problem if the data are randomly distributed over machines.
Extensions of ParMAC Data shuffling, which improves the SGD convergence speed, can be achieved without data movement by accessing the local data in random order at each epoch (withinmachine), and by randomising the circular topology at each epoch (across-machine). Load balancing is simple because the work in both W and Z steps is proportional to the number of data points N . Hence, if the processing power of machine p is proportional to αp > 0, we allocate to it Nαp/(α1 + · · · + αP ) data points. Streaming, i.e., discarding old data and adding new data during training, can be done by adding/removing data within-machine, or by adding/removing machines and updating the circular topology. Fault tolerance is possible because we can still learn a good model even if we lose the data from a machine that fails, and because in the W step we can revert to older copies of the lost submodels residing in other machines. See further details in Carreira-Perpiñán and Alizadeh (2016).
A theoretical model of the parallel speedup We can estimate the runtime of the W and Z steps assuming there are M independent submodels of the same size in the W step, using e epochs, on a dataset with N training points, distributed over P identical machines (each with N/P points). Let tWr be the computation time per submodel and data point in the W step, t Z r the computation time per data point in the Z step, and tWc the communication time per submodel in theW step. Then the runtime of theW and Z steps is TW(P ) = ⌈M/P ⌉(tWr NP + tWc )Pe+⌈M/P ⌉tWc P and TZ(P ) = M NP t Z r , respectively. Hence the parallel speedup is (see details in Carreira-Perpiñán and Alizadeh, 2016):
S(P ) = T (1)
T (P ) = ρ 1⌈M/P⌉MP 1 N P 2 + ρ2P + ρ1 1 ⌈M/P⌉M
ρ1 = t Z r /(e+ 1)t W c , ρ2 = et W r /(e+ 1)t W c
ρ = ρ1 + ρ2 = (et W r + t Z r )/(e+ 1)t W c
(3)
where ρ, ρ1 and ρ2 are ratios of computation vs communication, dependent on the optimisation algorithm in theW and Z steps, and on the performace of the distributed system and MPI library.
Hence, if P ≤ M and M is divisible by P we have S(P ) = P/(1 + PρN ) and if P > M we have S(P ) = ρM/(ρ2+ρ1 M P + P N ). In practice, typically we have ρ ≪ 1 (because communication dominates computation in current architectures) and ρ2N ≫ 1 (large dataset). If we take P ≪ ρ2N , then S(P ) ≈ P if P ≤ M and S(P ) ≈ ρM/(ρ2 + ρ1MP ) if P > M . Hence, the speedup is nearly perfect if using fewer machines than submodels, and otherwise it peaks at S∗1 = ρM/(ρ2 + 2 √
ρ1M/N) > M for P = P ∗ 1 = √ ρ1MN > M and decreases thereafter. This affords very large speedups for large datasets and large models. This theoretical speedup matches well our measured ones (see the experiments section), and can be used to determine optimal values for the number of machines P to use in practice (subject to additional constraints, e.g. cost of the machines).
Eq. (3) also shows that we can leave the speedup unchanged by trading off dataset size and computation/communication times, as long as one of these holds: NtWr and Nt Z r remain constant; or N/tWc remains constant; or t W r /t W c and t Z r /t W c remain constant.
In the BA, we have submodels of different size: encoders of size D and decoders of size L < D. We can model this by “grouping” the D decoders into L groups of D/L decoders each, resulting in M = 2L equal-size submodels (assuming the ratio of computation and communication times of decoder vs encoder is L/D < 1).
Convergence of ParMAC The only approximation that ParMACmakes to the originalMAC algorithm is using SGD in theW step. Since we can guarantee convergence of SGD under certain conditions (e.g. Robbins-Monro schedules), we can recover the original convergence guarantees for MAC to a local stationary point with differentiable layers (see details in Carreira-Perpiñán and Alizadeh, 2016). This convergence guarantee is independent of the number of layers, models and processors. With nondifferentiable layers, the convergence properties of MAC (and ParMAC) are not well known. In particular, for the binary autoencoder the encoding layer is discrete and the problem is NP-complete. While convergence guarantees are important theoretically, in practical applications with large datasets in a distributed setting one typically runs SGD for just a few epochs, even one or less than one (i.e., we stop SGD before passing through all the data). This typically reduces the objective function to a good enough value as fast as possible, since each pass over the data is very costly. In our experiments, 1–2 epochs in the W step make ParMAC very similar to MAC using an exact step.
Circular vs parameter-server topologies We also considered implementing ParMAC using a parameter-server (PS) topology rather than a circular one, but the latter is better. With a PS we do parallel SGD on each submodel independently, i.e., each worker runs SGD on its own submodel replica for a while, sends it to the PS, and this broadcasts an “average” submodel back to the workers, asynchronously. The circular topology does true SGD on each submodel independently from the others. We can show the runtime per iteration using a PS is equal to that of the circular topology only if the server can communicate with P workers simultaneously (rather than sequentially), otherwise it is slower. The reason is the PS has more communication. The PS has some additional disadvantages: parallel SGD convergesmore slowly than true SGD and is difficult to apply if theW step is nonconvex; and it needs extra machine(s) to act as parameter server(s). The fundamental issue is that both topologies differ in how they employ the available parallelism: the circular topology updates different, independent submodels, while the PS updates replicas of the same submodels.
4 EXPERIMENTS
MPI implementation of ParMAC for BAs. We have used C/C++, the GSL and BLAS libraries for mathematical operations, and the Message Passing Interface (MPI) (Gropp et al., 1999) for interprocess communication. MPI is a widely used framework for high-performance parallel computing, available in multiple platforms. It is particularly suitable for ParMAC because of its support of the SPMD (single program,multiple data) model. In MPI, processes in differentmachines communicate through messages. To receive data, we use the synchronous blocking receive function MPI Recv; the process calling this blocks until the data arrives. To send data we use the buffered blocking send function MPI Bsend. We allocate enough memory and attach it to the system. The process calling MPI Bsend blocks until the buffer is copied to the MPI internal memory; after that, the MPI library takes care of sending the data. See a code snippet in Carreira-Perpiñán and Alizadeh (2016).
Distributed-memory cluster. We used General Computing Nodes from the UCSD Triton Shared Computing Cluster (TSCC), available to the public for a fee. Each node contains 2 8-core Intel Xeon E5-2670 processors (16 cores in total), 64GB RAM (4GB/processor) and a 500GB hard drive. The nodes are connected through a 10GbE network. We used up to P = 128 processors. Carreira-Perpiñán and Alizadeh (2016) give detailed specs as well as experiments in a sharedmemory machine.
Datasets. We have used 3 well-known colour image retrieval benchmarks. (1) CIFAR (Krizhevsky, 2009) contains 60 000 images (N = 50 000 training and 10 000 test), represented byD = 320GIST features. (2) SIFT-1M (Jégou et al., 2011a) contains N = 106 training and 104 test images, each
represented by D = 128 SIFT features. (3) SIFT-1B (Jégou et al., 2011a) has three subsets: 109 base vectors where the search is performed, N = 108 learning vectors used to train the model and 104 query vectors.
Performance measures. Regarding the quality of the BA and hash functions learnt, we report the retrieval precision (%) in the test set using as true neighbours the K nearest images in Euclidean distance in the original space, and as retrieved neighbours in the binary space we use the k nearest images in Hamming distance. We set (K, k) = (1 000, 100) for CIFAR and (10 000, 10 000) for SIFT-1M. For SIFT-1B, as suggested by the dataset creators, we report the recall@R: the average number of queries for which the nearest neighbour is ranked within the top R positions (for varying values of R); in case of tied distances, we place the query as top rank. All these measures are computed offline once the BA is trained. Carreira-Perpiñán and Alizadeh (2016) give additional measures and experiments.
Models and their parameters. We use BAs with linear encoders (linear SVM) except with SIFT-1B, where we also use kernel SVMs. The decoder is always linear. We set L = 16 bits (hash functions) for CIFAR and SIFT-1M and L = 64 bits for SIFT-1B. We initialise the binary codes from truncated PCA ran on a subset of the training set (small enough that it fits in one processor). To train the encoder (L SVMs) and decoder (D linear mappings) with stochastic optimisation, we used the SGD code from (Bottou and Bousquet, 2008), using its default parameter settings. The SGD step size is tuned automatically in each iteration by examining the first 1 000 datapoints. We use a multiplicative µ schedule µi = µ0a
i where the initial value µ0 and the factor a > 1 are tuned offline in a trial run using a small subset of the data. For CIFAR we use µ0 = 0.005 and a = 1.2 over 26 iterations (i = 0, . . . , 25). For SIFT-1M and SIFT-1B we use µ0 = 10 −4 and a = 2 over 10 iterations.
Effect of stochastic steps in the W step Fig. 3 shows the effect on the precision on CIFAR of varying the number of epochs within the W step and shuffling the data as a function of the number of processors P . As the number of epochs increases, the W step is solved more exactly (8 epochs is practically exact in this data). Fewer epochs, even just one, cause only a small degradation. The reason is that, although these are relatively small datasets, they contain sufficient redundance that few epochs are sufficient to decrease the error considerably.
This is also helped by the accumulated effect of epochs over MAC iterations. Running more epochs increases the runtime and lowers the parallel speedup in this particular model, because we use few bits (L = 16) and therefore few submodels (M = 2L = 32) compared to the number of machines (up to P = 128), so the W step has less parallelism. The positive effect of data shuffling in the W step is clear: shuffling generally increases the precision with no increase in runtime.
Speedup The fundamental advantage of ParMAC and distributed optimisation in general is the ability to train on datasets that do not fit in a single machine, and the reduction in runtime because of parallel processing. Fig. 4 shows the “strong scaling” speedups achieved, as a function of the number of machines P for fixed problem size (dataset and model), in CIFAR and SIFT-1M (N = 50K and 1M training points, respectively). Even though these datasets and especially the number of independent submodels (M = 2L = 32 effective submodels of the same size, as discussed earlier) are relatively small, the speedups we achieve are nearly perfect for P ≤ M and hold very well for P > M up to the maximum number of machines we used (P = 128 in the distributed system). The speedups flatten as the number ofW-step epochs (and consequently the amount of communication) increases, because for this experiment the bottleneck is the W step, whose parallelisation ability (i.e., the number of concurrent processes) is limited by M = 2L (the Z step has N independent processes and is never a bottleneck, since N is very large). However, as noted earlier, using 1 to 2 epochs gives a good enough result, very close to doing an exactW step. The runtime for SIFT-1M on P = 128 machines with 8 epochs was 12 minutes and its speedup 100×. This is particularly remarkable given that the original, nested model did not have model parallelism.
Fig. 4 also shows the speedups predicted by our theoretical model. We set the parameters e and N to their known values, and M = 2L = 32 for CIFAR and SIFT-1M and M = 2L = 128 for SIFT-1B. For the time parameters, we set tWr = 1 to fix the time units, and we set t W c and t Z r by trial and error to achieve a reasonably good fit to the experimental speedups: tWc = 10 4 for both datasets, and tZr = 200 for CIFAR and 40 for SIFT-1M. Although these are fudge factors, they are in rough agreement with the fact that communicating a weight vector over the network is orders of magnitude slower than updating it with a gradient step, and that the Z step is quite slower than the W step because of the binary optimisation it involves.
Large-scale experiment SIFT-1B is one of the largest datasets, if not the largest one, that are publicly available for comparing nearest-neighbour search algorithmswith known ground-truth (i.e., precomputed exact Euclidean distances for each query to its k nearest vectors in the base set). The training set contains N = 100M vectors, each consisting of 128 SIFT features. We used L = 64 hash functions (M = 128 submodels): linear SVMs as before, and kernel SVMs. These have fixed Gaussian radial basis functions (2 000 centres picked at random from the training set and bandwidth σ = 160), so the only trainable parameters are the weights, and the MAC algorithm does not change except that it operates on a 2 000-dimensional input vector of kernel values, instead of the 128 SIFT features. We use e = 2 epochs with shuffling. All these decisions were based on trials on a subset of the training dataset. We initialised the binary codes from truncated PCA trained on a subset of size 1M (recall@R=100: 55.2%), which gave results comparable to the baseline in (Jégou et al., 2011b).
We ran ParMAC on the whole training set in the distributed system with 128 processors for 6 iterations and achieved a recall@R=100 of 61.5% in 29 hours (linear SVM) and 66.1% in 83 hours (kernel SVM). Using a scaled-down model and training set, we estimated that training in one machine (with enough RAM to hold the data and parameters) would take months. The theoretical speedup (fig. 4 right plot, using the same parameters as in SIFT-1M), is nearly perfect (note the plot goes up to P = 1 024 machines, even though our experiments are limited to P = 128). This is becauseM is quite larger andN is much larger than in the previous datasets.
5 DISCUSSION
Developing parallel, distributed optimisation algorithms for nonconvex problems in machine learning is challenging, as shown by recent efforts by large teams of researchers (Le et al., 2012; Dean et al., 2012). One important advantage of ParMAC is its simplicity. Data and model paral-
lelism arise naturally thanks to the introduction of auxiliary coordinates. The corresponding optimisation subproblems can often be solved reusing existing code as a black box (as with the SGD training of SVMs and linear mappings in the BA). A circular topology is sufficient to achieve a low communication between machines. There is no close coupling between the model structure and the distributed system architecture. This makes ParMAC suitable for architectures as different as supercomputers and data centres.
Further improvements can be made in specific problems. For example, we may have more parallelisation or less dependencies (e.g. the weights of hidden units in layer k of a neural net depend only on auxiliary coordinates in layers k and k + 1). This may reduce the communication in theW step, by sending to a given machine only the model portion it needs, or by allocating cores within a multicore machine accordingly. TheW and Z step optimisations can make use of further parallelisation by GPUs or by distributed convex optimisation algorithms. Many more refinements can be done, such as storing or communicating reduced-precision values with little effect of the accuracy. In this paper, we have tried to keep our implementation as simple as possible, because our goal was to understand the parallelisation speedups of ParMAC in a setting as general as possible, rather than trying to achieve the very best performance for a particular dataset, model or distributed system.
6 CONCLUSION
We have proposed ParMAC, a distributed model for the method of auxiliary coordinates for training nested, nonconvex models in general, analysed its parallel speedup and convergence, and demonstrated it with anMPI-based implementation for a particular case, to train binary autoencoders. MAC creates parallelism by introducing auxiliary coordinates for each data point to decouple nested terms in the objective function. ParMAC is able to translate the parallelism inherent in MAC into a distributed system by 1) using data parallelism, so that each machine keeps a portion of the original data and its corresponding auxiliary coordinates; and 2) using model parallelism, so that independent submodels visit every machine in a circular topology, effectively executing epochs of a stochastic optimisation, without the need for a parameter server and therefore no communication bottlenecks. The convergence properties of MAC (to a stationary point of the objective function) remain essentially unaltered in ParMAC. The parallel speedup can be theoretically predicted to be nearly perfect when the number of submodels is comparable or larger than the number of machines, and to eventually saturate as one continues to increase the number of machines, and indeed this was confirmed in our experiments. ParMAC also makes it easy to account for data shuffling, load balancing, streaming and fault tolerance. Hence, we expect that ParMAC could be a basic building block, in combination with other techniques, for the distributed optimisation of nested models in big data settings.
ACKNOWLEDGMENTS
Work supported by a Google Faculty Research Award and by NSF award IIS–1423515. We thank Ramin Raziperchikolaei (UC Merced) for discussions about binary autoencoders, Dong Li (UC Merced) for discussions about MPI and performance evaluation on parallel systems, and Quoc Le (Google) for discussions about Google’s DistBelief system.

UPDATE:
I looked at the arxiv version of the paper. It is much longer and appears more rigorous. Fig 3 there is indeed more insightful.
However, I am reviewing the submission and my overall assessment does not change. This is not a minor incremental contribution, and if you want to compress it into a conference submission of this type, I would recommend choosing message you want to convey, and focus on that. As you say, "...ICLR submission focus on the ParMAC algorithm...", I would focus on this properly - and remove or move to appendix all extensions and theoretical remarks, and have an extra page on explaining the algorithm. Additionally, make sure to clearly explain the relation of the arxiv paper, in particular that the submission was a compressed version.
ORIGINAL REVIEW:
The submission proposes ParMAC, based on MAC (Method of Auxiliary Coordinates), formulating a distributed variant of the idea.
Related Work: In the part on convex ERM and methods, I would recommend citing general communication efficient frameworks, COCOA (Ma et al.) and AIDE (Reddi et al.). I believe these works are most related to the practical objectives authors of this paper set, while number of the papers cited are less relevant.
Section 2, explaining MAC, is quite clearly written, but I do not find part on MAC and EM particularly useful.
Section 3 is much less clearly written. I have trouble following notation, particularly in the speedups part, as different symbols were introduced at different places. Perhaps a quick summary or paragraph on notation in the introduction would be helpful. In paragraph 2, you write as if reader knew how data/anything is distributed, but this was not mentioned yet; it is specified later. It is not clear what is meant by "submodel". Perhaps a more precise example pointing back to eqs (1) & (2) would be useful. As far as I understand from what is written, there are P independent sets of submodels, that traverse the machines in circular fashion. I don't understand how are they initialized (identically?), and more importantly I don't understand what would be a single output of the algorithm (averaging? does not seem to make sense). Since this is not addressed, I suppose I get it wrong, leaving me to guess what was actually meant.
The fact that I am not able to understand what is actually happening, I see as major issue.
I don't like the later paragraphs on extensions, model for speedup, convergence and topologies. I don't understand whether these are novel contributions or not, as the authors refer to other work for details. If these are novel, the explanation is not sufficient, particularly speedup part, which contains undefined quantities, e.g. T(P) (or I can't find it). If this is not novel, It does not provide enough explanation to understand anything more, compared with a its version compressed to 1/4 of its size and referring to the other work. The statement that we can recover the original convergence guarantees seems strong and I don't see why it should be trivial to show (but author point to other work which I did not look at). In topologies part, claiming that something does "true SGD", without explaining what is "true SGD" seems very strange. Other statements in this section seem also very vague and unjustified/unexplained.
Experimental section seems to suggest that the method is interesting for binary autoencoders, but I don't see how would I conclude anything about any other models. ParMAC is also not compared to alternative methods, only with itself, focusing on scaling properties.
Conclusion contains statements that are too strong or misleading based on what I saw. In particular, "we analysed its parallel speedup and convergence" seems ungrounded. Further, the claim "The convergence properties of MAC remain essentially unaltered in ParMAC" is unsupported, regardless of the meaning of "essentially unchanged".
In summary, the method seems relevant for particular model class, binary autoencoders, but clarity of presentation is insufficient - I wouldn't be able to recreate the algorithm used in experiments - and the paper contains a number of questionable claims.


This paper explores the possibility of learning chess game concepts under weak supervision with convolutional neural networks, which is a topic that has not been visited to the best of our knowledge. We put this task in three different backgrounds: (1) deep reinforcement learning has shown an amazing capability to learn a mapping from visual inputs to most rewarding actions, without knowing the concepts of a video game. But how could we confirm that the network understands these concepts or it just does not? (2) cross-modal supervision for visual representation learning has drawn much attention recently. Is this methodology still applicable when it comes to the domain of game concepts and actions? (3) class activation mapping is widely recognized as a visualization technique to help us understand what a network has learnt. Is it possible for it to activate at non-salient regions? With the simplest chess game tic-tac-toe, we report interesting results as answers to those three questions mentioned above. All codes, pre-processed datasets and pre-trained models will be released.
1 INTRODUCTION
1.1 APPLICATION BACKGROUND
Deep reinforcement learning (DRL) has drawn quite much attention since the publication of influential work Mnih et al. (2015). A convolutional neural network (CNN) is used to bridge the gap between video game screen frames and the most rewarding actions. An amazing feature of this kind of systems is that they do not need to know the concepts of these games (e.g. DRL learns to play Breakout without knowing there is a paddle or a ball in Fig 1a). However, how could we confirm that this network really understands these concepts or it just learns a mapping from patterns in the visual inputs to the best actions? This is the first question we are trying to answer here.
Mnih et al. (2015) provides some unsupervised analysis results for visualization, showing that perceptually dissimilar frames may produce close rewards, yet this does not answer the question. We choose another visualization technique called class activation mapping as described in Zhou et al. (2016), which can reveal where the CNN’s attention is. However, directly applying it in tasks like Breakout still cannot answer the question. Imagine one modifies the network described in Mnih et al. (2015) into another version as Zhou et al. (2016) does. The CNN’s attention may be fixed on the ball but it is still not enough to support that the network understands the concept of a ball.
∗This work was done when Hao Zhao was an intern at Intel Labs China, supervised by Anbang Yao who is responsible for correspondence.
We propose to use a simple chess game called tic-tac-toe for case study. In order to answer the question, we propose a protocol as this: to place a piece where the CNN’s attention is, and examine whether it is the right move. Of course, the training has to be done under weak supervision, or say, without telling the network what exactly a right move is. We think if this experiment succeeds we can claim that the network figures out the concepts of: (1) a chess board grid; (2) the winning rule; (3) two sides. Detailed analysis about these three concepts are provided later.
1.2 METHODOLOGY BACKGROUND
There have been some works about representation learning with cross-modal supervision recently. Owens et al. (2016) clusters sound statistics into several categories, and uses them as labels to learn visual representation from images corresponding to these sounds. It quantitatively shows that visual representation learnt in this way is capable of handling challenging computer vision tasks and qualitatively shows that visual and sound representations are consistent (e.g. babies’ faces correspond to baby cry sound samples). Castrejón et al. (2016) goes even further by learning representations across five modalities: RGB images, clip art pictures, sketches, texts and spatial texts. Gupta et al. (2016) learns depth image representation with mid-level features extracted from RGB images as supervision, and reports improved RGB-D object detection performance.
What is the common point among these works? They generate weak supervision from one modality and use it to learn representation from another (e.g. to learn what a train looks like from what a train sounds like or to learn what a chair looks like in depth images from what a chair looks like in RGB images). During training phase, no concepts about a train or a chair are explicitly modeled. Although there are many other modalities not visited by this methodology, we think the basic ideas behind these works are same: an abstract concept like a train can be observed in different modalities and different representations can be connected.
Here comes the question: is this methodology still applicable when it goes beyond the problem of learning representations from different observations of a same concept? Albanie & Vedaldi (2016) is an example, which tries to relate facial expressions with what happened in a TV show (e.g. if a character earns a lot of money, she will be very happy). Although in Albanie & Vedaldi (2016) what happened is explicitly defined, it still can be regarded as a weak supervision for what this expression is.
Although with the same methodology, the problem studied in this paper addresses even higher semantics: to learn what to do under the weak supervision of what will happen (Fig 1b). This is substantially different from cross-modal supervision works mentioned above because there is no longer a certain abstract concept of object or attribute observed in different modalities. Instead, figuring out the relationship between what to do and what will happen needs a higher level of intelligence.
1.3 TECHNIQUE BACKGROUND
The core technique used in this paper is class activation mapping (CAM) as described in Zhou et al. (2016). So leaving out all the backgrounds about playing a chess game or cross-modal supervision, what do our experiments say more than its inventors’? We think we show that CAM can also activate at non-salient regions. CAM helps us to understand where contributes the most to a classification result. As Fig 1c shows, the heatmap reveals that the face contributes the most to the result that the network claims it as a person.
As has already been shown by Krizhevsky et al. (2012), kernels of lower layers of a CNN capture gradients in an image. Existing CAM experiments tend to activate at salient regions, and this is very reasonable because there are more gradients and therefore more information (e.g. the face in Fig 1c). Here comes the question: could CAM activate at non-salient regions like the empty spaces on a chess board? Our answer is positive as the results (Fig 1d) show that in order to predict what will happen in the future, the CNN’s attention is fixed upon texture-free regions.
Since we render chessboards as visual inputs without adding noise, those empty spaces are completely empty meaning that: (1) if we take out the activated patch in Fig 1d, all pixels in this patch have exactly the same value. (2) If we evaluate this patch with quantitative information metric like entropy, there is no information here. Thus the only reason why these regions are activated is that the network collects enough information from these regions’ receptive fields. We argue that this experiment (CAM can activate at non-salient regions) testifies (again) CNN’s ability to hierarchically collect information from visual inputs.
1.4 WHAT THIS PAPER IS ABOUT
After introducing those three backgrounds, we describe our work briefly as: to classify rendered tic-tac-toe chessboards with weak labels and to visualize that the CNN’s attention automatically reveals where the next piece should be placed. Learnt representation shows that: (1) the network knows some concepts of the game that it is not told of; (2) this level of supervision for representation learning is possible; (3) the technique of class activation mapping can activate at non-salient regions.
2 RELATED WORKS
2.1 CONCEPT LEARNING
Concept learning has different meanings in different contexts, and how to confirm a concept is learnt remains an open question. In Jia et al. (2013), a concept is learnt if a generative model is learnt from a small number of positive samples. In Lake et al. (2015), a concept is learnt if a model learnt from only one instance can generalize to various tasks. Higgins et al. (2016) claims a concept is learnt when a model can predict unseen objects’ sizes and positions. To summarize, they evaluate whether a concept is learnt through a model’s generalization ability. In even earlier works like Zhu et al. (2010);Yang et al. (2010), concept learning means a object/attribute classification task dealing with appearance variations, in which a concept is actually already pre-defined.
Unlike these works, we investigate the concepts of game rules instead of object/attribute. Unlike Jia et al. (2013);Lake et al. (2015);Higgins et al. (2016), we claim a concept is learnt through a novel testing protocol instead of generalization ability. Why generalization ability could show a concept is learnt? We think the reason is that a model understands a concept if it can use it in more cases. To this end, we argue that our protocol could also show a concept is learnt because the learnt representations in our experiments can be used to decide what to do though no rule about what need to be done is provided.
2.2 CROSS-MODAL SUPERVISION
The literature of cross-model supervision and the differences between this paper and existing ones are already covered in last section. Here we re-claim it briefly: Owens et al. (2016);Castrejón et al. (2016);Gupta et al. (2016) learn representations across modalities because actually they are different observations of a same (object or attribute) concept. Whether this methodology is applicable for
higher-level concepts like game rules remains an open question and we provide positive answers to this question.
2.3 CLASS ACTIVATION MAPPING
Before the technique of class activation mapping is introduced by Zhou et al. (2016), pioneering works like Simonyan et al. (2014);Zhou et al. (2015) have already shown CNN’s ability to localize objects with image-level labels. Although with different techniques, Simonyan et al. (2014);Zhou et al. (2015)’s activation visualization results also focus on salient regions. Unlike these works, we show that class activation mapping can activate at non-salient regions, or say more specifically, completely texture-free regions. Since the activated patch itself provides no information, all discriminative information comes from its context. This is another strong evidence to prove CNN’s capability to collect information from receptive fields, as a hierarchical visual model.
3 EXPERIMENT I: GAME ENDS IN NEXT MOVE
A tic-tac-toe chessboard is a 3×3 grid, and there are two players (black and white in our case). Due to duality, we generate all training samples assuming the black side takes the first move. The state space of tic-tac-toe is small consisting of totally 39 = 19683 combinations. Among them, many combinations are illegal such as the one in which all 9 pieces are black. We exhaustively search over the space according to a recursive simulation algorithm, in which: (1) the chessboard state is denoted by an integer smaller than 19683. (2) every state corresponds to a 9-d vector, with each element can take a value from this set {0-illegal, 1-black win, 2-white win, 4-tie, 5-uncertain}. We call this 9-d vector a state transfer vector, denoting what will happen if the next legal piece placement happens at according location. (3) generated transfer vectors can predict the existence of a critical move that will finish the game in advance. We will release this simulation code.
After pruning out illegal states, we collect 4486 possible states in total. Among these samples, we further take out 1029 states that a certain side is going to win in the next move. We then transform these chessboard states into visual representations (gray-scale images at resolution (180, 180)). Each of these 1029 samples is assigned a label according to the state transfer vectors. There are totally 18 different labels illustrating 2 (sides) × 9 (locations). As demonstrated by Fig 2, we randomly pick a sample for each label. As mentioned before black side takes the first move, thus if the numbers of
black and white pieces are equal the next move will be black side’s and if there are one more black piece the next move will be white side’s.
Although the concepts of two sides and nine locations are coded into the labels, this kind of supervision is still weak supervision. Because what we are showing to the algorithm is just 18 abstract categories as Fig 2 shows. Could an algorithm figure out what it needs to do by observing these visual inputs? We think even for a human baby it is difficult because no concepts like this is a game or you need to find out how to win are provided. In the setting of deep reinforcement learning there is at least an objective of getting higher score to pursue.
As mentioned before, the method we exploit is to train a classification network on this rendered dataset (Fig 2) and analyze learnt representations with the technique of class activation mapping. As Zhou et al. (2016) suggests, we add one global average pooling layer after the last convolutional layer of a pre-trained AlexNet model. All fully connected layers of the AlexNet model are discarded, and a new fully connected layer is added after the global average pooling layer. After the new classification network is fine-tuned on our dataset, a CAM visualization is generated by weighting the outputs of the last convolutional layer with parameters from the added fully connected layer. Our CAM implementation is built upon Marvin and it will be released.
Due to the simplicity of this classification task, the top one classification accuracy is 100% (not surprisingly). Class activation mapping results are provided in Fig 3 and here we present the reasons why we claim concepts are learnt: (1) We provide 18 abstract categories, but in order to classify visual inputs into these 18 categories the network’s attention is roughly fixed upon chessboard grids.
This means the concept of grid emerges in the learnt representation. (2) If we place a piece at the most activated location in Fig 3, that will be the right (and legal) move to finish the game. On one hand, this means the concept of winning rule emerges in the learnt representation. On the other hand, this means this learnt concept can be used to deal with un-taught task (analogous to Jia et al. (2013);Lake et al. (2015);Higgins et al. (2016) who use generalization ability to illustrate that concepts are learnt). (3) As Fig 3cehijnpq show, both sides can win in the next move if we violate the take-turns rule. However, the network pays attention to the right location that is consistent to the rule. For example, in Fig 3j, it seems that placing a black piece at the left-top location will also end the game. However, this move will violate the rule because there are already more black pieces than white pieces meaning that this is the white side’s turn. This means that the concept of two sides emerges in learnt representation.
Except for learnt concepts, we analyze what this experiment provides for the remaining two questions. To the second question: results in Fig 3 show that the methodology of generating labels from one modality (state transfer vectors in our case) to supervise another modality is still applicable. More importantly, we use images as inputs yet the learnt visual representations contain not only visual saliency information but also untold chess game concepts. To the third question: as Fig 3 shows, most activated regions are empty spaces on the chessboard.
4 EXPERIMENT II: ADDING GRID LINES
Since we claim complicated concepts emerge in learnt visual representations, a natural question will be: if the chessboard’s and pieces’ appearances are changed does this experiment still work? Thus we design this experiment by adding grid lines to the chessboards when rendering synthetic data (Fig 4). The intentions behind this design is three-folded: (1) in this case, the chessboard’s appearance is changed. (2) after these lines are added, the concept that there is a chessboard grid is actually implied. Still, we do not think these lines directly provide the concept of chessboard grid thus we use the word imply. Whether the network can figure out what these lines mean still remain
uncertain. (3) those locations that are completely empty in Experiment I are no longer empty from the perspective of information (still empty from the perspective of game rule).
We train the same network on the newly rendered dataset with grid lines and calculate CAM results in the same way. The results are demonstrated by Fig 4. Generally speaking, the grid lines allow the network to better activate at the location of right move, making them stands out more on the heatmap. What does this mean to the three intentions mentioned in last paragraph? (1) Firstly, it shows that our experiment is robust to chess board appearance variance. (2) Secondly, after implying the concept that there is a chessboard grid, the network performs better at paying attention to the location of right move. Again we compare this phenomenon against how a human baby learns. Although not supported by phycological experiment, we think with a chessboard grid a human baby is more easy to figure out the game rule than without. (3) Thirdly, heatmap changes in Fig 4 is not surprising, because after adding those lines, the empty (from the perspective of game rule) regions contain more gradients for lower layers of a CNN to collect. However, again it supports that activating at non-salient regions is NOT trivial.
5 EXPERIMENT III: PIECE APPEARANCE CHANGE
In this experiment we change the appearance of the piece by: (1) replacing black boxes with white circles; (2) replacing white boxes with black crosses. Note that in this case the white side moves first. Again we train the same network and visualize with CAM. The results comparison is provided in Fig 6. Further we add grid lines to the cross/circle chessboard.
6 EXPERIMENT IV: MODEL BEHAVIOR OVER TIME
In order to further demonstrate the non-triviality of the model behaviors, we design this experiment. We train on the dataset in Experiment I with 1000 iterations and snap-shotted the parameters at 500th iteration. The classification accuracy is 100% at 1000th iteration and 53.13% at 500th iteration. The
CAM results are shown by Fig 5 in which all samples are true positives. We think it shows that there are two ways to achieve this classification task: (1) by paying attention to the visual patterns formed by the existing pieces; (2) by paying attention to where the next piece should be placed. This experiment shows that at an earlier stage of learning the model’s behavior is consistent to the first hypothesis and after the training is completely done the network can finally fire at correct location.
7 QUANTITATIVE EVALUATION
We propose two different quantitative evaluation protocols. The first one is representation accuracy (RAC), for which we select the most activated patch and examine whether it is the correct location to end the game. The second one is representation consistency (RCO), which correlates the normalized representation and a normalized ideal activation map. The quantitative comparisons are shown in Table 1, in which NAC stands for network classification accuracy. These results quantitatively support that: (1) learnt representation can be used to predict the right move at an over 70% accuracy. (2) adding grid lines (implying the concept of a chessboard) dramatically improves localization.
8 CONCLUSION
The core experiment in this paper is to train a classification CNN on rendered chessboard images under weak labels. After class activation mapping visualization, we analyse and interpret the results
in three different backgrounds. Although simple, we argue that our results are enough to show that: (1) a CNN can automatically figure out complicated game rule concepts in this case. (2) cross-modal supervision for representation learning is still applicable in this case of higher-level semantics. (3) the technique of CAM can activate at non-salient regions, testifying CNN’s capability to collect information from context in an extreme case (only context has information).

Summary
===
This paper presents tic-tac-toe as toy problem for investigating CNNs.
A dataset is created containing tic-tac-toe boards where one player is one
move away from winning and a CNN is trained to label boards according
to (1) the player who can win (2 choices) and (2) the position they may move
to win (9 choices), resulting in 18 labels. The CNN evaluated in this paper
performs perfectly at the task and the paper's goal is to inspect how the
CNN works.
The fundamental mechanism for this inspection is Class Activation
Mapping (CAM) (Zhou et. al. 2016), which identifies regions of implicit attention
in the CNN. These implicit attention maps (localization heat maps) are used to
derive actions (which square each player should move). The attention maps
(1) attend to squares in the tic-tac-toe board rather than arbitrary
blobs, despite the fact that one square in a board has uniform color, and
(2) they can be used to pick correct (winning) actions.
This experiment are used to support assertions that the network understands
(1) chess (tic-tac-toe) boards
(2) a rule for winning tic-tac-toe
(3) that there are two players.
Some follow up experiments indicate similar results under various renderings
of the tic-tac-toe boards and an incomplete training regime.
More Clarifying Questions
===
* I am not quite sure precisely how CAM is implemented here. In the original CAM
one must identify a class of interest to visualize (e.g., cat or dog). I don't
think this paper identifies such a choice. How is one of the 18 possible classes
chosen for creating the CAM visualization and through that visualization
choosing an action?
* How was the test set for this dataset for the table 1 results created?
How many of the final 1029 states were used for test and was the
distribution of labels the same in train and test?
* How is RCO computed? Is rank correlation or Pearson correlation used?
If Pearson correlation is used then it may be good to consider rank correlation,
as argued in "Human Attention in Visual Question Answering: Do Humans and
Deep Networks Look at the Same Regions?" by Das et. al. in EMNLP 2016.
In table 1, what does the 10^3 next to RCO mean?
Pros
===
* The proposed method, deriving an action to take from the result of a
visualization technique, is very novel.
* This paper provides an experiment that clearly shows a CNN relying on context
to make accurate predictions.
* The use of a toy tic-tac-toe domain to study attention in CNNs
(implicit or otherwise) is a potentially fruitful setting that may
lead to better understanding of implicit and maybe explicit attention mechanisms.
Cons
===
* This work distinguishes between predictions about "what will happen"
(will the white player win?) and "what to do" (where should the white
player move to win?). The central idea is generalization from "what will happen"
to "what to do" indicates concept learning (sec. 2.1). Why should an ability to
act be any more indicative of a learned concept than an ability to predict
future states. I see a further issue with the presentation of this approach and
a potential correctness problem:
1. (correctness)
In the specific setting proposed I see no difference between "what to do"
and "what will happen."
Suppose one created labels dictating "what to do" for each example in the
proposed dataset. How would these differ from the labels of "what will happen"
in the proposed dataset? In this case "what will happen" labels include
both player identity (who wins) and board position (which position they move
to win). Wouldn't the "what to do" labels need to indicate board position?
They could also chosen to indicate player identity, which would make them
identical to the "what will happen" labels (both 18-way softmaxes).
2. (presentation)
I think this distinction would usually be handled by the Reinforcement Learning
framework, but the proposed method is not presented in that framework or
related to an RL based approach. In RL "what will happen" is the reward an
agent will receive for making a particular action and "what to do" is the
action an agent should take. From this point of view, generalization from
"what will happen" to "what to do" is not a novel thing to study.
Alternate models include:
    * A deep Q network (Mnih. et. al. 2015) could predict the value of
      every possible action where an action is a (player, board position) tuple.
    * The argmax of the current model's softmax could be used as an action
      prediction.
The deep Q network approach need not be implemented, but differences between
methods should be explained because of the uniqueness of the proposed approach.
* Comparison to work that uses visualization to investigate deep RL networks
is missing. In particular, other work in RL has used Simonyan et. al.
(arXiv 2013) style saliency maps to investigate network behavior. For example,
"Dueling Network Architectures for Deep Reinforcement Learning" by Wang et. al.
in (ICML 2016) uses saliency maps to identify differences between their
state-value and advantage networks. In "Graying the black box:
Understanding DQNs" by Zahavy et. al. (ICML 2016) these saliency maps are
also used to analyze network behavior.
* In section 2.3, saliency maps of Simonyan et. al. are said to not be able to
activate on grid squares because they have constant intensity, yet no empirical
or theoretical evidence is provided for this claim.
On a related note, what precisely is the notion of information referenced in
section 2.3 and why is it relevant? Is it entropy of the distribution of pixel
intensities in a patch? To me it seems that any measure which depends only
on one patch is irrelevant because the methods discussed (e.g., saliency maps)
depend on context as well as the intensities within a patch.
* The presentation in the paper would be improved if the results in section 7
were presented along with relevant discussion in preceding sections.
Overall Evaluation
===
The experiments presented here are novel, but I am not sure they are very
significant or offer clear conclusions. The methods and goals are not presented
clearly and lack the broader relevant context mentioned above. Furthermore, I
find the lines of thought mentioned in the Cons section possibly incorrect
or incomplete. As detailed with further clarifying questions, upon closer
inspection I do not see how some aspects of the proposed approach were
implemented, so my opinion may change with further details.


MULTIPLICATIVE LSTM FOR SEQUENCE MODELLING
We introduce multiplicative LSTM (mLSTM), a novel recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level modelling tasks, and that this improvement increases with the complexity of the task. This model achieves a test error of 1.19 bits/character on the last 4 million characters of the Hutter prize dataset when combined with dynamic evaluation.
1 INTRODUCTION
Recurrent neural networks (RNNs) are powerful sequence density estimators that can use long contexts to make predictions. They have achieved tremendous success in (conditional) sequence modelling tasks such as language modelling, machine translation and speech recognition. Generative models of sequences can apply factorization via the product rule to perform density estimation of the sequence x1:T = {x1, . . . , xT },
P (x1, . . . , xT ) = P (x1)P (x2|x1)P (x3|x2, x1) · · ·P (xT |x1 . . . xT−1). (1)
RNNs can model sequences with the above factorization by using a hidden state to summarize past inputs. The hidden state vector ht is updated recursively using the previous hidden state vector ht−1 and the current input xt as ht = F(ht−1, xt), (2) where F is a differentiable function with learnable parameters. In a vanilla RNN, F multiplies its inputs by a matrix and squashes the result with a non-linear function such as a hyperbolic tangent (tanh). The updated hidden state vector is then used to predict a probability distribution over the next sequence element, using function G. In the case where x1:T consists of mutually exclusive discrete outcomes, G may apply a matrix multiplication followed by a softmax function:
P (xt+1) = G(ht). (3)
Generative RNNs can evaluate log-likelihoods of sequences exactly, and are differentiable with respect to these log-likelihoods. RNNs can be difficult to train due to the vanishing gradient problem (Bengio et al., 1994), but advances such as the long short-term memory architecture (LSTM) (Hochreiter & Schmidhuber, 1997) have allowed RNNs to be successful. Despite their success, generative RNNs (as well as other conditional generative models) are known to have problems with recovering from mistakes (Graves, 2013). Each time the recursive function of the RNN is applied and the hidden state is updated, the RNN must decide which information from the previous hidden state to store, due to its limited capacity. If the RNN’s hidden representation remembers the wrong information and reaches a bad numerical state for predicting future sequence elements, for instance as a result of an unexpected input, it may take many time-steps to recover.
We argue that RNN architectures with hidden-to-hidden transition functions that are input-dependent are better suited to recover from surprising inputs. Our approach to generative RNNs combines LSTM units with multiplicative RNN (mRNN) factorized hidden weights, allowing flexible input-dependent
transitions that are easier to control due to the gating units of LSTM . We compare this multiplicative LSTM hybrid architecture with other variants of LSTM on a range of character level language modelling tasks. Multiplicative LSTM is most appropriate when it can learn parameters specifically for each possible input at a given timestep. Therefore, its main application is to sequences of discrete mutually exclusive elements, such as language modelling and related problems.
1.1 INPUT-DEPENDENT TRANSITION FUNCTIONS
RNNs learn a mapping from previous hidden state ht−1 and input xt to hidden state ht. Let ĥt denote the input to the next hidden state before any non-linear operation:
ĥ(t) =Whhht−1 +Whxxt, (4)
where Whh is the hidden-to-hidden weight matrix, and Whx is the input-to-hidden weight matrix. For problems such as language modelling, xt is a one-hot vector, meaning that the output of Whxxt is a column in Whx, corresponding to the unit element in xt.
The possible future hidden states in an RNN can be viewed as a tree structure, as shown in Figure 1. For an alphabet of N inputs and a fixed ht−1, there will be N possible transition functions between ht−1 and ĥt. The relative magnitude of Whhht−1 to Whxxt will need to be large for the RNN to be able to use long range dependencies, and the resulting possible hidden state vectors will therefore be highly correlated across the possible inputs, limiting the width of the tree and making it harder for the RNN to form distinct hidden representations for different sequences of inputs. However, if the RNN has flexible input-dependent transition functions, the tree will be able to grow wider more quickly, giving the RNN the flexibility to represent more probability distributions.
In a vanilla RNN, it is difficult to allow inputs to greatly affect the hidden state vector without erasing information from the past hidden state. However, an RNN with a transition function mapping ĥt ← ht−1 dependent on the input would allow the relative values of ht to vary with each possible input xt, without overwriting the contribution from the previous hidden state, allowing for more long term information to be stored. This ability to adjust to new inputs quickly while limiting the overwriting of information should make an RNN more robust to mistakes when it encounters surprising inputs, as the hidden vector is less likely to get trapped in a bad numerical state for making future predictions.
1.2 MULTIPLICATIVE RNN
The multiplicative RNN (mRNN) (Sutskever et al., 2011) is an architecture designed specifically to allow flexible input-dependent transitions. Its formulation was inspired by the tensor RNN, an RNN architecture that allows for a different transition matrix for each possible input. The tensor RNN
features a 3-way tensor W 1:Nhh , which contains a separately learned transition matrix Whh for each input dimension. The 3-way tensor can be stored as an array of matrices
W (1:N) hh = {W (1) hh , ...,W (N) hh }, (5)
where superscript is used to denote the index in the array, and N is the dimensionality of xt. The specific hidden-to-hidden weight matrix W (xt)hh used for a given input xt is then
W (xt) hh = N∑ n=1 W (n) hh x (n) t . (6)
For language modelling problems, only one unit of xt will be on, and W (xt) hh will be the matrix in W (1:N) hh corresponding to that unit. Hidden-to-hidden propagation in the tensor RNN is then given by
ĥ(t) =W (xt) hh ht−1 +Whxxt. (7)
The large number of parameters in the tensor RNN make it impractical for most problems. mRNNs can be thought of as a tied-parameter approximation to the tensor RNN that use a factorized hiddento-hidden transition matrix in place of the normal RNN hidden-to-hidden matrix Whh, with an input-dependent intermediate diagonal matrix diag(Wmxxt). The input-dependent hidden-to-hidden weight matrix, W (xt)hh is then
W (xt) hh =Whmdiag(Wmxxt)Wmh. (8)
An mRNN is thus equivalent to a tensor RNN using the above form for W (xt)hh . For readability, an mRNN can also be described using intermediate state mt as follows:
mt = (Wmxxt) (Wmhht−1) (9) ĥt =Whmmt +Whxxt. (10)
mRNNs have improved on vanilla RNNs at character level language modelling tasks (Sutskever et al., 2011; Mikolov et al., 2012), but have fallen short of the more popular LSTM architecture, for instance as shown with LSTM baselines from (Cooijmans et al., 2016). The standard RNN units in an mRNN do not provide an easy way for information to bypass its complex transitions, resulting in the potential for difficulty in retaining long term information.
1.3 LONG SHORT-TERM MEMORY
LSTM is a commonly used RNN architecture that uses a series of multiplicative gates to control how information flows in and out of internal states of the network (Hochreiter & Schmidhuber, 1997). There are several slightly different variants of LSTM, and we present the variant used in our experiments.
The LSTM hidden state receives inputs from the input layer xt and the previous hidden state ht−1:
ĥt =Whxxt +Whhht−1. (11)
The LSTM network also has 3 gating units – input gate i, output gate o, and forget gate f – that have both recurrent and feed-forward connections:
it = σ(Wixxt +Wihht−1) (12) ot = σ(Woxxt +Wohht−1) (13) ft = σ(Wfxxt +Wfhht−1), (14)
where σ is the logistic sigmoid function. The input gate controls how much of the input to each hidden unit is written to the internal state vector ct, and the forget gate determines how much of the previous internal state ct−1 is preserved. This combination of write and forget gates allows the network to control what information should be stored and overwritten across each time-step. The internal state is updated by
ct = ft ct−1 + it ĥt. (15)
The output gate controls how much of each unit’s activation is preserved. It allows the LSTM cell to keep information that is not relevant to the current output, but may be relevant later. The final output of the hidden state is given by ht = tanh(ct ot). (16) This is slightly different from the typical LSTM variant, where the output gate is applied after the tanh. LSTM’s ability to control how information is stored in each unit has proven generally useful.
1.4 COMPARING LSTM WITH MRNN
The LSTM and mRNN architectures both feature multiplicative units, but these units serve different purposes. LSTM’s gates are designed to control the flow of information through the network, whereas mRNN’s gates are designed to allow transition functions to vary across inputs. LSTM gates receive input from both the input units and hidden units, allowing multiplicative interactions between hidden units, but also potentially limiting the extent of input-hidden multiplicative interaction. LSTM gates are also squashed with a sigmoid, forcing them to take values between 0 and 1, which makes them easier to control, but less expressive than mRNN’s linear gates. For language modelling problems, mRNN’s linear gates do not need to be controlled by the network because they are explicitly learned for each input. They are also placed in between a product of 2 dense matrices, giving more flexibility to the possible values of the final product of matrices.
2 MULTIPLICATIVE LSTM
Since the LSTM and mRNN architectures are complimentary, we propose the multiplicative LSTM (mLSTM), a hybrid architecture that combines the factorized hidden-to-hidden transition of mRNNs with the gating framework from LSTMs. The mRNN and LSTM architectures can be combined by adding connections from the mRNN’s intermediate state mt (which is redefined below for convenience) to each gating units in the LSTM, resulting in the following system:
mt = (Wmxxt) (Wmhht−1) (17) ĥt =Whxxt +Whmmt (18) it = σ(Wixxt +Wimmt) (19) ot = σ(Woxxt +Wommt) (20) ft = σ(Wfxxt +Wfmmt). (21)
We set the dimensionality of mt and ht equal for all our experiments. We also chose to share mt across all LSTM unit types, resulting in a model with 1.25 times the number of recurrent weights as LSTM for the same number of hidden units.
The goal of this architecture is to combine the flexible input-dependent transitions of mRNNs with the long time lag and information control of LSTMs. The gated units of LSTMs could make it easier to control (or bypass) the complex transitions in that result from the factorized hidden weight matrix. The additional sigmoid input and forget gates featured in LSTM units allow even more flexible input-dependent transition functions than in regular mRNNs.
3 RELATED APPROACHES
Many recently proposed RNN architectures use recurrent depth, which is depth between recurrent steps. Recurrent depth allows more non-linearity in the combination of inputs and previous hidden states from every time step, which in turn allows for more flexible input-dependent transitions. Recurrent depth has been found to perform better than other kinds of non-recurrent depth for sequence modelling (Zhang et al., 2016). Recurrent highway networks (Zilly et al., 2016) use a more sophisticated recurrent depth that carefully controls propagation through layers using gating units. The gating units also allow for a greater deal of multiplicative interaction between the inputs and hidden units. While adding recurrent depth could improve our model, we believe that maximizing the input-dependent flexibility of the transition function is more important for expressive sequence modelling. Recurrent depth can do this through non-linear layers combining hidden and input contributions, but our method can do this independently of depth.
Another approach, multiplicative integration RNNs (Wu et al., 2016), use Hadamard products instead of addition when combining contributions from input and hidden units. When applying this to LSTM, this architecture achieves impressive sequence modelling results. The main difference between multiplicative integration LSTM and mLSTM is that mLSTM applies the Hadamard product between the multiplication of two matrices. In the case of LSTM, this allows for the potential for greater expressiveness, without significantly increasing the size of the model.
4 EXPERIMENTS
4.1 SYSTEM SETUP
Our experiments compared the performance of mLSTM with regular LSTM for different characterlevel language modelling tasks of varying complexity1. Gradient computation in these experiments used truncated backpropagation through time on sequences of length 100, only resetting the hidden state every 10 000 timesteps to allow networks access to information far in the past. All experiments used a variant of RMSprop, (Tieleman & Hinton, 2012), with normalized updates in place of a learning rate. All unnormalized update directions v∗, computed by RMSprop, were normalized to have length `, where ` was decayed exponentially over training:
v ← `√ vT∗ v∗ v∗. (22)
We found that this allowed for fast convergence with larger batch sizes, allowing for greater parallelization during training without hurting performance.
We compared mLSTM to previously reported regular LSTM, stacked LSTM, and RNN character-level language models. The stacked LSTMs were all 2-layer, and both LSTM layers contained direct connections from the inputs and to the outputs. We used the Penn Treebank dataset (Marcus et al., 1993) to test small scale language modelling, the processed and raw versions of the Wikipedia text8 dataset (Hutter, 2012) to test large scale language modelling and byte level language modelling respectively, and the European parliament dataset (Koehn, 2005) to investigate multilingual fitting.
4.2 PENN TREEBANK
The Penn treebank dataset is relatively small, and consists of only case insensitive English characters, with no punctuation. It is one of the most widely used language modelling bench mark tasks. Due to its small size, the main bottleneck for performance is overfitting.
We fitted an mLSTM with 700 hidden units to the Penn Treebank dataset, with no regularization other than early stopping. We used a slightly different version of this dataset, where the frequently occurring token <unk> was replaced by a single character, shortening the file by about 4%. To make our results comparable to other results on this dataset, we computed the total cross entropy of the test set file and divided this by the number of characters in the original test set. The results are shown in
1Code to replicative our large scale experiments on the Hutter prize dataset is available at https:// github.com/benkrause/mLSTM.
Table 1, where it can be seen that mLSTM achieved 1.35 bits/char test set error, compared with 1.38 bits/char for an unregularized LSTM (Cooijmans et al., 2016).
4.3 TEXT8 DATASET
Text8 contains 100 million characters of English text taken from Wikipedia in 2006, consisting of just the 26 characters of the English alphabet plus spaces. This dataset can be found at http: //mattmahoney.net/dc/textdata. This corpus has been widely used to benchmark RNN character level language models, with the first 90 million characters used for training, the next 5 million used for validation, and the final 5 million used for testing. The results of these experiments are shown in Table 2.
The first set of experiments we performed were designed to be comparable to those of Zhang et al. (2016), who benchmarked several deep LSTMs against shallow LSTMs on this dataset. The shallow LSTM had a hidden state dimensionality of 512, and the deep versions had reduced dimensionality to give them roughly the same number of parameters. Our experiment used an mLSTM with a hidden dimensionality of 450, giving it slightly fewer parameters than the past work, and our own LSTM baseline with hidden dimensionality 512. mLSTM showed an improvement over our baseline and the previously reported best deep LSTM variant.
We ran additional experiments to compare a large mLSTM with other reported experiments. We trained an mLSTM with hidden dimensionality of 1900 on the text8 dataset. mLSTM was able to fit the training data well and achieved a competitive performance; however it was outperformed by other architectures that are less prone to over-fitting.
4.4 HUTTER PRIZE DATASET
We performed experiments using the raw version of the Wikipedia dataset, originally used for the Hutter Prize compression benchmark (Hutter, 2012). This dataset consists mostly of English language text and mark-up language text, but also contains text in other languages, including non-Latin languages. The dataset is modelled using a UTF-8 encoding, and contains 205 unique bytes.
We compared mLSTMs and 2-layer stacked LSTMs for varying network sizes, ranging from about 3–20 million parameters. These results all used RMS prop with normalized updates, stopping after 4 epochs, with test performance measured on the last 5 million bytes. Hyperparameters for each mLSTM and stacked LSTM were kept constant across all sizes, and were reused from the previous experiment using the text8 dataset. The results, shown in Figure 2, show that mLSTM gives a modest improvement across all network sizes.
We hypothesized that mLSTM’s superior performance over stacked LSTM was in part due to its ability to recover from surprising inputs. To test this we looked at each network’s performance after viewing surprising inputs that occurred naturally in the test set by creating a set of the 10% characters
with the largest average loss taken by mLSTM and stacked LSTM. Both networks perform roughly equally on this set of surprising characters, with mLSTM and stacked LSTM taking losses of 6.27 bits/character and 6.29 bits/character respectively. However, stacked LSTM tended to take much larger losses than mLSTM in the timesteps immediately following surprising inputs. One to four time-steps after a surprising input occurred, mLSTM and stacked LSTM took average losses of (2.26, 2.04, 1.61, 1.51) and (2.48, 2.25, 1.79, 1.67) bits per character respectively, as shown in Figure 3. mLSTM’s overall advantage over stacked LSTM was 1.42 bits/char to 1.53 bits/char; mLSTM’s advantage over stacked LSTM was greater after a surprising input than it is in general.
We also tested our largest mLSTM and stacked LSTM models using dynamic evaluation, where the network’s weights are adapted to fit recent sequences (Graves, 2013). Dynamic evaluation can be thought of as a type of fast weights memory structure (Ba et al., 2016) that draws the network towards regularities that recently occurred in the sequence, causing the network to assign higher probabilities to these regularities occurring again soon. Unlike other approaches to fast weights where the network learns to control the weights, dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted. Rather than performing fully online dynamic evaluation, we adapted incrementally to short sequences, allowing for gradients to be passed back over longer time scales. Thus, we divided the test set into sequences of length 50 in order of occurrence. After predicting a sequence and incurring a loss, we trained the RNN for a single iteration on that sequence, using RMSprop (with a learning rate, as normalized updates no longer make sense in a stochastic setting), and weight decay. After updating the RNN, we recomputed the forward pass through this sequence to update the final hidden state. The updated RNN was then used to predict the next sequence of 50 elements, and this process was repeated. Dynamic evaluation was applied to a test set of the last 4 million bytes (instead of last 5 million) to be comparable with the only other dynamic evaluation result (Graves, 2013).
The best results for stacked LSTM and mLSTM are given in Table 3, alongside results from the literature. mLSTM performs at near state-of-the-art level when evaluated statically, and greatly outperforms the best static models and other dynamic models when evaluated dynamically.
4.5 MULTILINGUAL LEARNING
Character-level RNNs are able to simultaneously model multiple languages using shared parameters. These experiments compared the relative ability of LSTM and mLSTM at fitting a dataset in a single language and a combined dataset with two separate languages. We used the first 100 million characters of the English and Spanish translations of the European parliament dataset to make an English dataset and a Spanish dataset. Each dataset was split 90-5-5 for training, validation, and testing. We created a third Spanish-English hybrid dataset by combining the Spanish and English datasets, resulting in a dataset twice as large. These datasets were left in their raw form, containing punctuation and both
upper-case and lower-case letters. The LSTMs in these experiments had a hidden dimensionality of 2200, and the mLSTMs had a hidden dimensionality of 1900. All experiments were run for 4 epochs.
mLSTM generally outperformed LSTM at this task, as shown in Table 4. However, there also seemed to be an interaction effect with the number of languages. Increasing the complexity of the task by forcing the RNN to learn 2 languages instead of 1 presented a larger fitting difficulty for LSTM than for mLSTM.
5 DISCUSSION
This work combined the mRNN’s factorized hidden weights with the LSTM’s hidden units for generative modelling of discrete multinomial sequences. This mLSTM architecture was motivated by its ability to have both controlled and flexible input-dependent transitions, to allow for fast changes to the distributed hidden representation without erasing information. In a series of character-level language modelling experiments, mLSTM showed improvements over LSTM and its deep variants. This relative improvement increased with the complexity of the task, and provides evidence that mLSTM has a more powerful fitting ability for character-level language modelling than regular LSTM and its common deep variants. mLSTM performed competitively at large scale characterlevel language modelling, and achieved dramatic improvement over the state of the art with 1.19 bits/character on the Hutter prize dataset when combined with dynamic evaluation, motivating further investigation of dynamic evaluation for RNN sequence modelling.
While these results are promising, it remains to be seen how mLSTM performs at word-level language modelling and other discrete multinomial generative modelling tasks, and whether mLSTM can be formulated to apply more broadly to tasks with continuous or non-sparse input units. We also hope this work will motivate further exploration in generative RNN architectures with flexible input-dependent transition functions.

This paper proposes an extension of the multiplicative RNN [1] where the authors apply the same reparametrization trick to the weight matrices of the LSTM.
The paper proposes some interesting tricks, but none of them seems to be very crucial. For instance, in Eq. (16), the authors propose to multiply the output gate inside the activation function in order to alleviate the saturation problem in logistic sigmoid or hyperbolic tangent. Also, the authors share m_t across the inference of different gating units and cell-state candidates, at the end this brings only 1.25 times increase on the number of model parameters. Lastly, the authors use a variant of RMSProp where they add an additional hyper-parameter $\ell$ and schedule it across the training time. It would be nicer to apply the same tricks to other baseline models and show the improvement with regard to each trick.
With the new architectural modification to the LSTM and all the tricks combined, the performance is not as great as we would expect. Why didn’t the authors apply batch normalization, layer normalization or zoneout to their models? Was there any issue with applying one of those regularization or optimization techniques?
At the fourth paragraph of Section 4.4, where the authors connect dynamic evaluation with fast weights is misleading. I find it a bit hard to connect dynamic evaluation as a variant of fast weights. Fast weights do not use test error signal. In the paper, the authors claim that “dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted”, and I am afraid to tell that this assumption is very misleading. We should never assume that test label information is given at the inference time. The test label information is there to evaluate the generalization performance of the model. In some applications, we may get the label information at test time, e.g., stock prediction, weather forecasting, however, in many other applications, we don’t. For instance, in machine translation, we don't know what's the best translation at the end, unlike weather forecasting. Also, it would be fair to apply dynamic evaluation to all the other baseline models as well to compare with the BPC score 1.19 achieved by the proposed mLSTM.
The quality of the work is not that bad, but the novelty of the paper is not that good either. The performance of the proposed model is oftentime worse than other methods, and it is only better when dynamic evaluation is coupled together. However, dynamic evaluation can improve the other methods as well.
[1] Ilya et al., “Generating Text with Recurrent Neural Networks”, ICML’11