DEEP CONVOLUTIONAL NEURAL NETWORK DESIGN PATTERNS
Recent research in the deep learning field has produced a plethora of new architectures. At the same time, a growing number of groups are applying deep learning to new applications. Some of these groups are likely to be composed of inexperienced deep learning practitioners who are baffled by the dizzying array of architecture choices and therefore opt to use an older architecture (i.e., Alexnet). Here we attempt to bridge this gap by mining the collective knowledge contained in recent deep learning research to discover underlying principles for designing neural network architectures. In addition, we describe several architectural innovations, including Fractal of FractalNet network, Stagewise Boosting Networks, and Taylor Series Networks (our Caffe code and prototxt files is available at https://github.com/iPhysicist/CNNDesignPatterns). We hope others are inspired to build on our preliminary work.
1 INTRODUCTION
Many recent articles discuss new architectures for neural networking, especially regarding Residual Networks (He et al. (2015; 2016); Larsson et al. (2016); Zhang et al. (2016); Huang et al. (2016b)). Although the literature covers a wide range of network architectures, we take a high-level view of the architectures as the basis for discovering universal principles of the design of network architecture. We discuss 14 original design patterns that could benefit inexperienced practitioners who seek to incorporate deep learning in various new applications. This paper addresses the current lack of guidance on design, a deficiency that may prompt the novice to rely on standard architecture, e.g., Alexnet, regardless of the architecture’s suitability to the application at hand.
This abundance of research is also an opportunity to determine which elements provide benefits in what specific contexts. We ask: Do universal principles of deep network design exist? Can these principles be mined from the collective knowledge on deep learning? Which architectural choices work best in any given context? Which architectures or parts of architectures seem elegant?
Design patterns were first described by Christopher Alexander (Alexander (1979)) in regards to the architectures of buildings and towns. Alexander wrote of a timeless quality in architecture that “lives” and this quality is enabled by building based on universal principles. The basis of design patterns is that they resolve a conflict of forces in a given context and lead to an equilibrium analogous to the ecological balance in nature. Design patterns are both highly specific, making them clear to follow, and flexible so they can be adapted to different environments and situations. Inspired by Alexander’s work, the “gang of four” (Gamma et al. (1995)) applied the concept of design patterns to the architecture of object-oriented software. This classic computer science book describes 23 patterns that resolve issues prevalent in software design, such as “requirements always change”. We were inspired by these previous works on architectures to articulate possible design patterns for convolutional neural network (CNN) architectures.
Design patterns provide universal guiding principles, and here we take the first steps to defining network design patterns. Overall, it is an enormous task to define design principles for all neural networks and all applications, so we limit this paper to CNNs and their canonical application of image classification. However, we recognize that architectures must depend upon the application by defining our first design pattern; Design Pattern 1: Architectural Structure follows the Application
(we leave the details of this pattern to future work). In addition, these principles allowed us to discover some gaps in the existing research and to articulate novel networks (i.e, Fractal of FractalNets, Stagewise Boosting and Taylor Series networks) and units (i.e., freeze-drop-path). We hope the rules of thumb articulated here are valuable for both the experienced and novice practitioners and that our preliminary work serves as a stepping stone for others to discover and share additional deep learning design patterns.
2 RELATED WORK
To the best of our knowledge, there has been little written recently to provide guidance and understanding on appropriate architectural choices1. The book ”Neural Networks: Tricks of the Trade” (Orr & Müller, 2003) contains recommendations for network models but without reference to the vast amount of research in the past few years. Perhaps the closest to our work is Szegedy et al. (2015b) where those authors describe a few design principles based on their experiences.
Much research has studied neural network architectures, but we are unaware of a recent survey of the field. Unfortunately, we cannot do justice to this entire body of work, so we focus on recent innovations in convolutional neural networks architectures and, in particular, on Residual Networks (He et al., 2015) and its recent family of variants. We start with Network In Networks (Lin et al., 2013), which describes a hierarchical network with a small network design repeatedly embedded in the overall architecture. Szegedy et al. (2015a) incorporated this idea into their Inception architecture. Later, these authors proposed modifications to the original Inception design (Szegedy et al., 2015b). A similar concept was contained in the multi-scale convolution architecture (Liao & Carneiro, 2015). In the meantime, Batch Normalization (Ioffe & Szegedy, 2015) was presented as a unit within the network that makes training faster and easier.
Before the introduction of Residual Networks, a few papers suggested skip connections. Skip connections were proposed by Raiko et al. (2012). Highway Networks (Srivastava et al., 2015) use a gating mechanism to decide whether to combine the input with the layer’s output and showed how these networks allowed the training of very deep networks. The DropIn technique (Smith et al., 2015; 2016) also trains very deep networks by allowing a layer’s input to skip the layer. The concept of stochastic depth via a drop-path method was introduced by Huang et al. (2016b).
Residual Networks were introduced by He et al. (2015), where the authors describe their network that won the 2015 ImageNet Challenge. They were able to extend the depth of a network from tens to hundreds of layers and in doing so, improve the network’s performance. The authors followed up with another paper (He et al., 2016) where they investigate why identity mappings help and report results for a network with more than a thousand layers. The research community took notice of this architecture and many modifications to the original design were soon proposed.
The Inception-v4 paper (Szegedy et al., 2016) describes the impact of residual connections on their Inception architecture and compared these results with the results from an updated Inception design. The Resnet in Resnet paper (Targ et al., 2016) suggests a duel stream architecture. Veit et al. (2016) provided an understanding of Residual Networks as an ensemble of relatively shallow networks. These authors illustrated how these residual connections allow the input to follow an exponential number of paths through the architecture. At the same time, the FractalNet paper (Larsson et al., 2016) demonstrated training deep networks with a symmetrically repeating architectural pattern. As described later, we found the symmetry introduced in their paper intriguing. In a similar vein, Convolutional Neural Fabrics (Saxena & Verbeek, 2016) introduces a three dimensional network, where the usual depth through the network is the first dimension.
Wide Residual Networks (Zagoruyko & Komodakis, 2016) demonstrate that simultaneously increasing both depth and width leads to improved performance. In Swapout (Singh et al., 2016), each layer can be dropped, skipped, used normally, or combined with a residual. Deeply Fused Nets (Wang et al., 2016) proposes networks with multiple paths. In the Weighted Residual Networks paper (Shen & Zeng, 2016), the authors recommend a weighting factor for the output from the convolutional layers, which gradually introduces the trainable layers. Convolutional Residual Memory Networks (Moniz & Pal, 2016) proposes an architecture that combines a convolutional Residual Network with
1After submission we became aware of an online book being written on deep learning design patterns at http://www.deeplearningpatterns.com
an LSTM memory mechanism. For Residual of Residual Networks (Zhang et al., 2016), the authors propose adding a hierarchy of skip connections where the input can skip a layer, a module, or any number of modules. DenseNets (Huang et al., 2016a) introduces a network where each module is densely connected; that is, the output from a layer is input to all of the other layers in the module. In the Multi-Residual paper (Abdi & Nahavandi, 2016), the authors propose expanding a residual block width-wise to contain multiple convolutional paths. Our Appendix A describes the close relationship between many of these Residual Network variants.
3 DESIGN PATTERNS
We reviewed the literature specifically to extract commonalities and reduce their designs down to fundamental elements that might be considered design patterns. It seemed clear to us that in reviewing the literature some design choices are elegant while others are less so. In particular, the patterns described in this paper are the following:
1. Architectural Structure follows the Application
2. Proliferate Paths
3. Strive for Simplicity
4. Increase Symmetry
5. Pyramid Shape
6. Over-train
7. Cover the Problem Space
8. Incremental Feature Construction
9. Normalize Layer Inputs
10. Input Transition
11. Available Resources Guide Layer Widths
12. Summation Joining
13. Down-sampling Transition
14. Maxout for Competition
3.1 HIGH LEVEL ARCHITECTURE DESIGN
Several researchers have pointed out that the winners of the ImageNet Challenge (Russakovsky et al., 2015) have successively used deeper networks (as seen in, Krizhevsky et al. (2012), Szegedy et al. (2015a), Simonyan & Zisserman (2014), He et al. (2015)). It is also apparent to us from the ImageNet Challenge that multiplying the number of paths through the network is a recent trend that is illustrated in the progression from Alexnet to Inception to ResNets. For example, Veit et al. (2016) show that ResNets can be considered to be an exponential ensemble of networks with different lengths. Design Pattern 2: Proliferate Paths is based on the idea that ResNets can be an exponential ensemble of networks with different lengths. One proliferates paths by including a multiplicity of branches in the architecture. Recent examples include FractalNet (Larsson et al. 2016), Xception (Chollet 2016), and Decision Forest Convolutional Networks (Ioannou et al. 2016).
Scientists have embraced simplicity/parsimony for centuries. Simplicity was exemplified in the paper ”Striving for Simplicity” (Springenberg et al. 2014) by achieving state-of-the-art results with fewer types of units. Design Pattern 3: Strive for Simplicity suggests using fewer types of units and keeping the network as simple as possible. We also noted a special degree of elegance in the FractalNet (Larsson et al. 2016) design, which we attributed to the symmetry of its structure. Design Pattern 4: Increase Symmetry is derived from the fact that architectural symmetry is typically considered a sign of beauty and quality. In addition to its symmetry, FractalNets also adheres to the Proliferate Paths design pattern so we used it as the baseline of our experiments in Section 4.
An essential element of design patterns is the examination of trade-offs in an effort to understand the relevant forces. One fundamental trade-off is the maximization of representational power versus
elimination of redundant and non-discriminating information. It is universal in all convolutional neural networks that the activations are downsampled and the number of channels increased from the input to the final layer, which is exemplified in Deep Pyramidal Residual Networks (Han et al. (2016)). Design Pattern 5: Pyramid Shape says there should be an overall smooth downsampling combined with an increase in the number of channels throughout the architecture.
Another trade-off in deep learning is training accuracy versus the ability of the network to generalize to non-seen cases. The ability to generalize is an important virtue of deep neural networks. Regularization is commonly used to improve generalization, which includes methods such as dropout (Srivastava et al. 2014a) and drop-path (Huang et al. 2016b). As noted by Srivastava et al. 2014b, dropout improves generalization by injecting noise in the architecture. We believe regularization techniques and prudent noise injection during training improves generalization (Srivastava et al. 2014b, Gulcehre et al. 2016). Design Pattern 6: Over-train includes any training method where the network is trained on a harder problem than necessary to improve generalization performance of inference. Design Pattern 7: Cover the Problem Space with the training data is another way to improve generalization (e.g., Ratner et al. 2016, Hu et al. 2016, Wong et al. 2016, Johnson-Roberson et al. 2016). Related to regularization methods, cover the problem space includes the use of noise (Rasmus et al. 2015, Krause et al. 2015, Pezeshki et al. 2015), synthetic data, and data augmentation, such as random cropping, flipping, and varying brightness, contrast, and the like.
3.2 DETAILED ARCHITECTURE DESIGN
A common thread throughout many of the more successful architectures is to make each layer’s “job” easier. Use of very deep networks is an example because any single layer only needs to incrementally modify the input, and this partially explains the success of Residual Networks, since in very deep networks, a layer’s output is likely similar to the input; hence adding the input to the layer’s output makes the layer’s job incremental. Also, this concept is part of the motivation behind design pattern 2 but it extends beyond that. Design Pattern 8: Incremental Feature Construction recommends using short skip lengths in ResNets. A recent paper (Alain & Bengio (2016)) showed in an experiment that using an identity skip length of 64 in a network of depth 128 led to the first portion of the network not being trained.
Design Pattern 9: Normalize Layer Inputs is another way to make a layer’s job easier. Normalization of layer inputs has been shown to improve training and accuracy but the underlying reasons are not clear (Ioffe & Szegedy 2015, Ba et al. 2016, Salimans & Kingma 2016). The Batch Normalization paper (Ioffe & Szegedy 2015) attributes the benefits to handling internal covariate shift, while the authors of streaming normalization (Liao et al. 2016) express that it might be otherwise. We feel that normalization puts all the layer’s input samples on more equal footing (analogous to a units conversion scaling), which allows back-propagation to train more effectively.
Some research, such as Wide ResNets (Zagoruyko & Komodakis 2016), has shown that increasing the number of channels improves performance but there are additional costs with extra channels. The input data for many of the benchmark datasets have 3 channels (i.e., RGB). Design Pattern 10: Input Transition is based on the common occurrence that the output from the first layer of a CNN significantly increases the number of channels from 3. A few examples of this increase in channels/outputs at the first layer for ImageNet are AlexNet (96 channels), Inception (32), VGG (224), and ResNets (64). Intuitively it makes sense to increase the number of channels from 3 in the first layer as it allows the input data to be examined many ways but it is not clear how many outputs are best. Here, the trade-off is that of cost versus accuracy. Costs include the number of parameters in the network, which directly affects the computational and storage costs of training and inference. Design Pattern 11: Available Resources Guide Layer Widths is based on balancing costs against an application’s requirements. Choose the number of outputs of the first layer based on memory and computational resources and desired accuracy.
3.2.1 JOINING BRANCHES: CONCATENATION, SUMMATION/MEAN, MAXOUT
When there are multiple branches, three methods have been used to combine the outputs; concatenation, summation (or mean), or Maxout. It seems that different researchers have their favorites and there hasn’t been any motivation for using one in preference to another. In this Section, we propose some rules for deciding how to combine branches.
Summation is one of the most common ways of combining branches. Design Pattern 12: Summation Joining is where the joining is performed by summation/mean. Summation is the preferred joining mechanism for Residual Networks because it allows each branch to compute corrective terms (i.e., residuals) rather than the entire approximation. The difference between summation and mean (i.e., fractal-join) is best understood by considering drop-path (Huang et al. 2016b). In a Residual Network where the input skip connection is always present, summation causes the layers to learn the residual (the difference from the input). On the other hand, in networks with several branches, where any branch can be dropped (e.g., FractalNet (Larsson et al. (2016))), using the mean is preferable as it keeps the output smooth as branches are randomly dropped.
Some researchers seem to prefer concatenation (e.g., Szegedy et al. (2015a)). Design Pattern 13: Down-sampling Transition recommends using concatenation joining for increasing the number of outputs when pooling. That is, when down-sampling by pooling or using a stride greater than 1, a good way to combine branches is to concatenate the output channels, hence smoothly accomplishing both joining and an increase in the number of channels that typically accompanies down-sampling.
Maxout has been used for competition, as in locally competitive networks (Srivastava et al. 2014b) and competitive multi-scale networks Liao & Carneiro (2015). Design Pattern 14: Maxout for Competition is based on Maxout choosing only one of the activations, which is in contrast to summation or mean where the activations are “cooperating”; here, there is a “competition” with only one “winner”. For example, when each branch is composed of different sized kernels, Maxout is useful for incorporating scale invariance in an analogous way to how max pooling enables translation invariance.
4 EXPERIMENTS
4.1 ARCHITECTURAL INNOVATIONS
In elucidating these fundamental design principles, we also discovered a few architectural innovations. In this section we will describe these innovations.
First, we recommended combining summation/mean, concatenation, and maxout joining mechanisms with differing roles within a single architecture, rather than the typical situation where only one is used throughout. Next, Design Pattern 2: Proliferate Branches led us to modify the overall sequential pattern of modules in the FractalNet architecture. Instead of lining up the modules for maximum depth, we arranged the modules in a fractal pattern as shown in 1b, which we named a Fractal of FractalNet (FoF) network, where we exchange depth for a greater number of paths.
4.1.1 FREEZE-DROP-PATH AND STAGEWISE BOOSTING NETWORKS (SBN)
Drop-path was introduced by Huang et al. (2016b), which works by randomly removing branches during an iteration of training, as though that path doesn’t exist in the network. Symmetry considerations led us to an opposite method that we named freeze-path. Instead of removing a branch from the network during training, we freeze the weights, as though the learning rate was set to zero. A similar idea has been proposed for recurrent neural networks (Krueger et al. 2016).
The potential usefulness of combining drop-path and freeze-path, which we named freeze-drop-path, is best explained in the non-stochastic case. Figure 1 shows an example of a fractal of FractalNet architecture. Let’s say we start training only the leftmost branch in Figure 1b and use drop-path on all of the other branches. This branch should train quickly since it has only a relatively few parameters compared to the entire network. Next we freeze the weights in that branch and allow the next branch to the right to be active. If the leftmost branch is providing a good function approximation, the next branch works to produce a “small” corrective term. Since the next branch contains more layers than the previous branch and the corrective term should be easier to approximate than the original function, the network should attain greater accuracy. One can continue this process from left to right to train the entire network. We used freeze-drop-path as the final/bottom join in the FoF architecture in Figure 1b and named this the Stagewise Boosting Networks (SBN) because they are analogous to stagewise boosting (Friedman et al. 2001). The idea of boosting neural networks is not new (Schwenk & Bengio 2000) but this architecture is new. In Appendix B we discuss the implementation we tested.
4.1.2 TAYLOR SERIES NETWORKS (TSN)
Taylor series expansions are classic and well known as a function approximation method, which is: f(x+ h) = f(x) + hf ′(x) + h2f ′′(x)/2 + ... (1)
Since neural networks are also function approximators, it is a short leap from FoFs and SBNs to consider the branches of that network as terms in a Taylor series expansion. Hence, the Taylor series implies squaring the second branch before the summation joining unit, analogous to the second order term in the expansion. Similarly, the third branch would be cubed. We call this “Taylor Series Networks” (TSN) and there is precedence for this idea in the literature with polynomial networks (Livni et al. 2014) and multiplication in networks (e.g. Lin et al. 2015. The implementation details of this TSN are discussed in the Appendix.
4.2 RESULTS
The experiments in this section are primarily to empirically validate the architectural innovations described above but not to fully test them. We leave a more complete evaluation to future work.
Table 1 and Figures 2 and 3 compare the final test accuracy results for CIFAR-10 and CIFAR-100 in a number of experiments. An accuracy value in Table 1 is computed as the mean of the last 6 test
accuracies computed over the last 3,000 iterations (out of 100,000) of the training. The results from the original FractalNet (Larsson et al. 2016) are given in the first row of the table and we use this as our baseline. The first four rows of Table 1 and Figure 2 compare the test accuracy of the original FractalNet architectures to architectures with a few modifications advocated by design patterns. The first modification is to use concatenation instead of fractal-joins at all the downsampling locations in the networks. The results for both CIFAR-10 (2a) and CIFAR-100 (2b indicate that the results are equivalent when concatenation is used instead of fractal-joins at all the downsampling locations in the networks. The second experiment was to change the kernel sizes in the first module from 3x3 such that the left most column used a kernel size of 7x7, the second column 5x5, and the third used 3x3. The fractal-join for module one was replaced with Maxout. The results in Figure 2 are a bit worse, indicating that the more cooperative fractal-join (i.e., mean/summation) with 3x3 kernels has better performance than the competitive Maxout with multiple scales. Figure 2 also illustrates how an experiment replacing max pooling with average pooling throughout the architecture changes the training profile. For CIFAR-10, the training accuracy rises quickly, plateaus, then lags behind the original FractalNet but ends with a better final performance, which implies that average pooling might significantly reduce the length of the training (we plan to examine this in future work). This behavior provides some evidence that “cooperative” average pooling might be preferable to “competitive” max pooling.
Table 1 and Figure 3 compare the test accuracy results for the architectural innovations described in Section 4.1. The FoF architecture ends with a similar final test accuracy as FractalNet but the SBN and TSN architectures (which use freeze-drop-path) lag behind when the learning rate is dropped. However, it is clear from both Figures 3a and 3b that the new architectures train more quickly than FractalNet. The FoF network is best as it trains more quickly than FractalNet and achieves similar accuracy. The use of freeze-drop-path in SBN and TSN is questionable since the final performance lags behind FractalNet, but we are leaving the exploration for more suitable applications of these new architectures for future work.
5 CONCLUSION
In this paper we describe convolutional neural network architectural design patterns that we discovered by studying the plethora of new architectures in recent deep learning papers. We hope these design patterns will be useful to both experienced practitioners looking to push the state-of-the-art and novice practitioners looking to apply deep learning to new applications. There exists a large expanse of potential follow up work (some of which we have indicated here as future work). Our effort here is primarily focused on Residual Networks for classification but we hope this preliminary work will inspire others to follow up with new architectural design patterns for Recurrent Neural Networks, Deep Reinforcement Learning architectures, and beyond.
ACKNOWLEDGMENTS
The authors want to thank the numerous researchers upon whose work these design patterns are based and especially Larsson et al. 2016 for making their code publicly available. This work was supported by the US Naval Research Laboratory base program.
A RELATIONSHIPS BETWEEN RESIDUAL ARCHITECTURES
The architectures mentioned in Section 2 commonly combine outputs from two or more layers using concatenation along the depth axis, element-wise summation, and element-wise average. We show here that the latter two are special cases of the former with weight-sharing enforced. Likewise, we show that skip connections can be considered as introducing additional layers into a network that share parameters with existing layers. In this way, any of the Residual Network variants can be reformulated into a standard form where many of the variants are equivalent.
A filter has three dimensions: two spatial dimensions, along which convolution occurs, and a third dimension, depth. Each input channel corresponds to a different depth for each filter of a layer. As a result, a filter can be considered to consist of “slices,” each of which is convolved over one input channel. The results of these convolutions are then added together, along with a bias, to produce a single output channel. The output channels of multiple filters are concatenated to produce the output of a single layer. When the outputs of several layers are concatenated, the behavior is similar to that of a single layer. However, instead of each filter having the same spatial dimensions, stride, and padding, each filter may have a different structure. As far as the function within a network, though, the two cases are the same. In fact, a standard layer, one where all filters have the same shape, can be considered a special case of concatenating outputs of multiple layer types.
If summation is used instead of concatenation, the network can be considered to perform concatenation but enforce weight-sharing in the following layer. The results of first summing several channels element-wise and then convolving a filter slice over the output yields the same result as convolving the slice over the channels and then performing an element-wise summation afterwards. Therefore, enforcing weight-sharing such that the filter slices applied to the nth channel of all inputs share weight results in behavior identical to summation, but in a form similar to concatenation, which highlights the relationship between the two. When Batch Normalization (BN) (Ioffe & Szegedy 2015 is used, as is the current standard practice, performing an average is essentially identical to performing a summation, since BN scales the output. Therefore, scaling the input by a constant (i.e., averaging instead of a summation) is rendered irrelevant. The details of architecture-specific manipulations of summations and averages is described further in Section 3.2.1.
Due to the ability to express depth-wise concatenation, element-wise sum, and element-wise mean as variants of each other, architectural features of recent works can be combined within a single network, regardless of choice of combining operation. However, this is not to say that concatenation has the most expressivity and is therefore strictly better than the others. Summation allows networks to divide up the network’s task. Also, there is a trade-off between the number of parameters and the expressivity of a layer; summation uses weight-sharing to significantly reduce the number of parameters within a layer at the expense of some amount of expressivity.
Different architectures can further be expressed in a similar fashion through changes in the connections themselves. A densely connected series of layers can be “pruned” to resemble any desired architecture with skip connections through zeroing specific filter slices. This operation removes the dependency of the output on a specific input channel; if this is done for all channels from a given layer, the connection between the two layers is severed. Likewise, densely connected layers can be turned into linearly connected layers while preserving the layer dependencies; a skip connection can be passed through the intermediate layers. A new filter can be introduced for each input channel passing through, where the filter performs the identity operation for the given input channel. All existing filters in the intermediate layers can have zeroed slices for this input so as to not introduce new dependencies. In this way, arbitrarily connected layers can be turned into a standard form.
We certainly do not recommend this representation for actual experimentation as it introduces fixed parameters. We merely describe it to illustrate the relationship between different architectures. This representation illustrates how skip connections effectively enforce specific weights in intermediate layers. Though this restriction reduces expressivity, the number of stored weights is reduced, the number of computations performed is decreased, and the network might be more easily trainable.
B IMPLEMENTATION DETAILS
Our implementations are in Caffe (Jia et al. 2014; downloaded October 9, 2016) using CUDA 8.0. These experiments were run on a 64 node cluster with 8 Nvidia Titan Black GPUs, 128 GB memory, and dual Intel Xenon E5-2620 v2 CPUs per node. We used the CIFAR10 and CIFAR-100 datasets (Krizhevsky & Hinton 2009 for our classification tests. These datasets consist of 60,000 32x32 colour images (50,000 for training and 10,000 for testing) in 10 or 100 classes, respectively. Our Caffe code and prototxt files are publicly available at https://github.com/iPhysicist/CNNDesignPatterns.
B.1 ARCHITECTURES
We started with the FractalNet implementation 2 as our baseline and it is described in Larsson et al. 2016. We used the three column module as shown in Figure 1a. In some of our experiments, we replaced the fractal-join with concatenation at the downsampling locations. In other experiments, we modified the kernel sizes in module one and combined the branches with Maxout. A FractalNet module is shown in Figure 1a and the architecture consists of five sequential modules.
Our fractal of FractalNet (FoF) architecture uses the same module but has an overall fractal design as in Figure 1b rather than the original sequential one. We limited our investigation to this one realization and left the study of other (possibly more complex) designs for future work. We followed the FractalNet implementation in regards to dropout where the dropout rate for a module were 0%, 10%, 20%, or 30%, depending on the depth of the module in the architecture. This choice for dropout rates were not found by experimentation and better values are possible. The local drop-path rate in the fractal-joins were fixed at 15%, which is identical to the FractalNet implementation.
Freeze-drop-path introduces four new parameters. The first is whether the active branch is chosen stochastically or deterministically. If it is chosen stochastically, a random number is generated and the active branch is assigned based on which interval it falls in (intervals will be described shortly). If it is deterministically, a parameter is set by the user as to the number of iterations in one cycle through all the branches (we called this parameter num iter per cycle). In our Caffe implementation of the freeze-drop-path unit, the bottom input specified first is assigned as branch 1, the next is branch 2, then branch 3, etc. The next parameter indicates the proportion of iterations each branch should be active relative to all the other branches. The first type of interval uses the square of the branch number (i.e., 1, 4, 9, 16, ...) to assign the interval length for that branch to be active, which gives the more update iterations to the higher numbered branches. The next type gives the same amount of iterations to each branch. Our experiments showed that the first interval type works better (as we expected) and was used to obtained the results in Section 4.2. In addition, our experiments showed that the stochastic option works better than the deterministic option (unexpected) and was used for Section 4.2 results.
2https://github.com/gustavla/fractalnet/tree/master/caffe
The Stagewise Boosting Network’s (SBN) architecture is the same as the FoF architecture except that branches 2 and 3 are combined with a fractal-join and then combined with branch 1 in a freezedrop-path join. The reason for combining branches 2 and 3 came out of our first experiments; if branches 2 and 3 were separate, the performance deteriorated when branch 2 was frozen and branch 3 was active. In hindsight, this is due to the weights in the branch 2 path that are also in branch 3’s path being modified by the training of branch 3. The Taylor series network has the same architecture as SBN with the addition of squaring the branch 2 and 3 combined activations before the freeze-drop-path join.
For all of our experiments, we trained for 400 epochs. Since the training used 8 GPUs and each GPU had a batchsize of 25, 400 epochs amounted to 100,000 iterations. We adopted the same learning rate as the FractalNet implementation, which started at 0.002 and dropped the learning rate by a factor of 10 at epochs 200, 300, and 350.

The authors take on the task of figuring out a set of design patterns for current deep architectures - namely themes that are recurring in the literature.  If one may say so, a distributed representation of deep architectures.
There are two aspects of the paper that I particularly valued: firstly, the excellent review of recent works, which made me realize how many things I have been missing myself.  Secondly, the "community service" aspect of helping someone who starts figure out the "coordinate system" for deep architectures - this could potentially be more important than introducing yet-another trick of the trade, as most other submissions may do.
However I think this work is still half-done, and even though working on this project is a great idea, the authors do not yet do it properly.
Firstly, I am not too sure how the choice of these 14 patterns was made. Maxout for instance (pattern 14) is one of the many nonlinearities (PreLU, ReLU, ...) and I do not see how it stands on the same grounds as something as general as "3 Strive for simplicity".
Similarly some of the patterns are as vague as "Increase symmetry" and are backed up by statements such as "we noted a special degree of elegance in the FractalNet". I do not see how this leads to a design pattern that can be applied to a new architecture - or if it applies to anything other than the FractalNet.
Some other patterns are phrased with weird names "7 Cover the problem space" - which I guess stands for dataset augmentation; or "6 over-train" which is not backed up by a single reference. Unless the authors relate it to regularization (text preceding "overtrain"), which then has no connection to the description of "over-train" provided by the authors ("training a network on a harder problem to improve generalization"). If "harder problem" means one where one adds an additional term (i.e. the regularizer), the authors are doing harm to the unexperienced reader, confusing "regularization" with something that sounds like "overfitting" (i.e. the exact opposite).
Furthermore, the extensions proposed in Section 4 seem a bit off tune - in particular I could not figure out
-how the Taylor Series networks stem from any of the design patterns proposed in the rest of the paper.
-whether the text between 4.1 and 4.1.1 is another of the architecture innovations (and if yes, why it is not in the 4.1.2, or 4.1.0)
-and, most importantly, how these design patterns would be deployed in practice to think of a new network.
To be more concrete, the authors mention that they propose the "freeze-drop-path" variant from "symmetry considerations" to "drop-path".
Is this an application of the "increase symmetry" pattern? How would "freeze-drop-path" be more symmetric that "drop-path"?
 Can this be expressed concretely, or is it some intuitive guess? If the second, it is not really part of applying a pattern, in my understanding. If the first, this is missing.
What I would have appreciated more (and would like to see in a revised version) would have been a table of "design patterns" on one axis, "Deep network" on another, and a breakdown of which network applies which design pattern.
A big part of the previous work is also covered in cryptic language - some minimal explanation of what is taking place in the alternative works would be useful.


A CONVOLUTIONAL ENCODER MODEL FOR NEURAL MACHINE TRANSLATION
The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT’16 English-Romanian translation we achieve competitive accuracy to the state-of-theart and we outperform several recently published results on the WMT’15 EnglishGerman task. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT’14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline.
1 INTRODUCTION
Neural machine translation (NMT) is an end-to-end approach to machine translation (Sutskever et al., 2014). The most successful approach to date encodes the source sentence with a bi-directional recurrent neural network (RNN) into a variable length representation and then generates the translation left-to-right with another RNN where both components interface via a soft-attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a; Bradbury & Socher, 2016; Sennrich et al., 2016b). The recurrent networks are typically parameterized as long short term memory networks (LSTM; Hochreiter et al. 1997) or gated recurrent units (GRU; Cho et al. 2014), often with residual or skip connections (Wu et al., 2016; Zhou et al., 2016) to enable stacking of several layers (§2). There have been several attempts to use convolutional encoder models for neural machine translation in the past but they were either only applied to rescoring n-best lists of classical systems (Kalchbrenner & Blunsom, 2013) or were not competitive to recurrent alternatives (Cho et al., 2014a). This is despite several attractive properties of convolutional networks. For example, convolutional networks operate over a fixed-size window of the input sequence which enables the simultaneous computation of all features for a source sentence. This contrasts with RNNs which maintain a hidden state of the entire past that prevents parallel computation within a sequence.
Furthermore, a succession of convolutional layers provides a shorter path to capture relationships between elements of a sequence compared to recurrent networks.1 This also eases learning because the resulting tree-structure applies a fixed number of non-linearities compared to a recurrent neural network. Because processing is bottom-up, all words undergo the same number of transformations, whereas for recurrent networks the first word is over-processed and the last word is transformed only once.
In this paper we show that an architecture based on convolutional layers is very competitive to recurrent encoders. We investigate simple average pooling as well as parameterized convolutions as an alternative for recurrent encoders and enable very deep convolutional encoders by using residual connections (He et al., 2015; §3). We experiment on several standard datasets and compare our approach to variants of recurrent encoders such as uni-directional and bi-directional LSTMs. On WMT’16 English-Romanian transla-
1For kernel width k and sequence length n we require max ( 1, ⌈
n−1 k−1
⌉) forwards on a succession of stacked
convolutional networks compared to n forwards with a recurrent network.
tion we achieve accuracy that is very competitive to the current state-of-the-art single system result. We perform competitively on WMT’15 English-German, and nearly match the performance of the best WMT’14 English-French system based on a deep LSTM setup when comparing on a commonly used subset of the training data (Zhou et al. 2016; §4, §5).
2 RECURRENT NEURAL MACHINE TRANSLATION
The general architecture of the models in this work follows the encoder-decoder approach with soft attention first introduced in Bahdanau et al. (2015). A source sentence x = (x1, . . . , xm) of m words is processed by an encoder which outputs a sequence of states z = (z1. . . . , zm).
The decoder is an RNN network that computes a new hidden state si+1 based on the previous state si, an embedding gi of the previous target language word yi, as well as a conditional input ci derived from the encoder output z. We use LSTMs (Hochreiter & Schmidhuber, 1997) for all decoder networks whose state si comprises of a cell vector and a hidden vector hi which is output by the LSTM at each time step. We input ci into the LSTM by concatenating it to gi.
The translation model computes a distribution over the V possible target words yi+1 by transforming the LSTM output hi via a linear layer with weights Wo and bias bo:
p(yi+1|y1, . . . , yi,x) = softmax(Wohi+1 + bo) ∈ RV
The conditional input ci at time i is computed via a simple dot-product style attention mechanism (Luong et al., 2015a). Specifically, we transform the decoder hidden state hi by a linear layer with weights Wd and bd to match the size of the embedding of the previous target word gi and then sum the two representations to yield di. Conditional input ci is a weighted sum of attention scores ai ∈ Rm and encoder outputs z. The attention scores ai are determined by a dot product between hi with each zj , followed by a softmax over the source sequence:
di = Wdhi + bd + gi aij = exp
( dTi zj )∑m t=1 exp ( dTi zt ) ci = m∑ j=1 aijzj
In preliminary experiments, we did not find the MLP attention of Bahdanau et al. (2015) to perform significantly better in terms of BLEU nor perplexity. However, we found the dot-product attention to be more favorable in terms of training and evaluation speed.
We use bi-directional LSTMs to implement recurrent encoders similar to Zhou et al. (2016) which achieved some of the best WMT14 English-French results reported to date. First, each word of the input sequence x is embedded in distributional space resulting in e = (e1, . . . , em). The embeddings are input to two stacks of uni-directional RNNs where the output of each layer is reversed before being fed into the next layer. The first stack takes the original sequence while the second takes the reversed input sequence; the output of the second stack is reversed so that the final outputs of the stacks align. Finally, the top-level hidden states of the two stacks are concatenated and fed into a linear layer to yield z. We denote this encoder architecture as BiLSTM.
3 NON-RECURRENT ENCODERS
3.1 POOLING ENCODER
A simple baseline for non-recurrent encoders is the pooling model described in Ranzato et al. (2015) which simply averages the embeddings of k consecutive words. Averaging word embeddings does not convey positional information besides that the words in the input are somewhat close to each other. As a remedy, we add position embeddings to encode the absolute position of each source word within a sentence. Each source embedding ej therefore contains a position embedding lj as well as the word embedding wj . Position embeddings have also been found helpful in memory networks for question-answering and language modeling (Sukhbaatar et al., 2015). Similar to the recurrent encoder (§2), the attention scores aij are computed from the pooled representa-
tions zj , however, the conditional input ci is a weighted sum of the embeddings ej , not zj , i.e.,
ej = wj + lj zj = 1
k bk/2c∑ t=−bk/2c ej+t ci = m∑ j=1 aijej
The input sequence is padded prior to pooling such that the encoder output matches the input length |z| = |x|. We set k to 5 in all experiments as Ranzato et al. (2015).
3.2 CONVOLUTIONAL ENCODER
A straightforward extension of pooling is to learn the kernel in a convolutional neural network (CNN). The encoder output zj contains information about a fixed-sized context depending on the kernel width k but the desired context width may vary. This can be addressed by stacking several layers of convolutions followed by non-linearities: additional layers increase the total context size while non-linearities can modulate the effective size of the context as needed. For instance, stacking 5 convolutions with kernel width k = 3 results in an input field of 11 words, i.e., each output depends on 11 input words, and the non-linearities allow the encoder to exploit the full input field, or to concentrate on fewer words as needed.
To ease learning for deep encoders, we add residual connections from the input of each convolution to the output and then apply the non-linear activation function to the output (tanh; He et al., 2015); the non-linearities are therefore not ’bypassed’. Multi-layer CNNs are constructed by stacking several blocks on top of each other. The CNNs do not contain pooling layers which are commonly used for down-sampling, i.e., the full source sequence length will be retained after the network has been applied. Similar to the pooling model, the convolutional encoder uses position embeddings.
The final encoder consists of two stacked convolutional networks (Figure 1): CNN-a produces the encoder output zj to compute the attention scores ai, while the conditional input ci to the decoder is computed by summing the outputs of CNN-c,
zj = CNN-a(e)j ci = m∑ j=1 aij CNN-c(e)j .
In practice, we found that two different CNNs resulted in better perplexity as well as BLEU compared to using a single one (§5.3). We also found this to perform better than directly summing the ei without transformation as for the pooling model.
3.3 RELATED WORK
There are several past attempts to use convolutional encoders for neural machine translation, however, to our knowledge none of them were able to match the performance of recurrent encoders. Kalchbrenner & Blunsom (2013) introduce a convolutional sentence encoder in which a multi-layer CNN generates a fixed sized embedding for a source sentence, or an n-gram representation followed by transposed convolutions for directly generating a per-token decoder input. The latter requires
the length of the translation prior to generation and both models were evaluated by rescoring the output of an existing translation system. Cho et al. (2014a) propose a gated recursive CNN which is repeatedly applied until a fixed-size representation is obtained but the recurrent encoder achieves higher accuracy. In follow-up work, the authors improved the model via a soft-attention mechanism but did not re-consider convolutional encoder models (Bahdanau et al., 2015).
Concurrently to our work, Kalchbrenner et al. (2016) have introduced convolutional translation models without an explicit attention mechanism but their approach does not yet result in state-of-the-art accuracy. Lamb & Xie (2016) also proposed a multi-layer CNN to generate a fixed-size encoder representation but their work lacks quantitative evaluation in terms of BLEU. Meng et al. (2015) and Tu et al. (2015) applied convolutional models to score phrase-pairs of traditional phrase-based and dependency-based translation models. Convolutional architectures have also been successful in language modeling but so far failed to outperform LSTMs (Pham et al., 2016).
4 EXPERIMENTAL SETUP
4.1 DATASETS
We evaluate different encoders and ablate architectural choices on a small dataset from the GermanEnglish machine translation track of IWSLT 2014 (Cettolo et al., 2014) with a similar setting to Ranzato et al. (2015). Unless otherwise stated, we restrict training sentences to have no more than 175 words; test sentences are not filtered. This is a higher threshold compared to other publications but ensures proper training of the position embeddings for non-recurrent encoders; the length threshold did not significantly effect recurrent encoders. Length filtering results in 167K sentence pairs and we test on the concatenation of tst2010, tst2011, tst2012, tst2013 and dev2010 comprising 6948 sentence pairs.2 Our final results are on three major WMT tasks:
WMT’16 English-Romanian. We use the same data and pre-processing as Sennrich et al. (2016b) and train on 2.8M sentence pairs.3 Our model is word-based instead of relying on byte-pair encoding (Sennrich et al., 2016a). We evaluate on newstest2016.
WMT’15 English-German. We use all available parallel training data, namely Europarl v7, Common Crawl and News Commentary v10 and apply the standard Moses tokenization to obtain 3.9M sentence pairs (Koehn et al., 2007). We report results on newstest2015.
WMT’14 English-French. We use a commonly used subset of 12M sentence pairs (Schwenk, 2014), and remove sentences longer than 150 words. This results in 10.7M sentence-pairs for training. Results are reported on ntst14.
A small subset of the training data serves as validation set (5% for IWSLT’14 and 1% for WMT) for early stopping and learning rate annealing (§4.3). For IWSLT’14, we replace words that occur fewer than 3 times with a <unk> symbol, which results in a vocabulary of 24158 English and 35882 German word types. For WMT datasets, we retain 200K source and 80K target words. For EnglishFrench only, we set the target vocabulary to 30K types to be comparable with previous work.
4.2 MODEL PARAMETERS
We use 512 hidden units for both recurrent encoders and decoders. We reset the decoder hidden states to zero between sentences. For the convolutional encoder, 512 hidden units are used for each layer in CNN-a, while layers in CNN-c contain 256 units each. All embeddings, including the output produced by the decoder before the final linear layer, are of 256 dimensions. On the WMT corpora, we find that we can improve the performance of the bi-directional LSTM models (BiLSTM) by using 512-dimensional word embeddings.
Model weights are initialized from a uniform distribution within [−0.05, 0.05]. For convolutional layers, we use a uniform distribution of [ −kd−0.5, kd−0.5 ] , where k is the kernel width (we use 3
2Different to the other datasets, we lowercase the training data and evaluation with case-insensitive BLEU. 3We followed the pre-processing of https://github.com/rsennrich/wmt16-scripts/ blob/master/sample/preprocess.sh and added the back-translated data from http://data. statmt.org/rsennrich/wmt16_backtranslations/en-ro.
throughout this work) and d is the input size for the first layer and the number of hidden units for subsequent layers (Collobert et al., 2011b). For CNN-c, we transform the input and output with a linear layer each to match the smaller embedding size. The model parameters were tuned on IWSLT’14 and cross-validated on the larger WMT corpora.
4.3 OPTIMIZATION
Recurrent models are trained with Adam as we found them to benefit from aggressive optimization. We use a step width of 3.125·10−4 and early stopping based on validation perplexity (Kingma & Ba, 2014). For non-recurrent encoders, we obtain best results with stochastic gradient descent (SGD) and annealing: we use a learning rate of 0.1 and once the validation perplexity stops improving, we reduce the learning rate by an order of magnitude each epoch until it falls below 10−4.
For all models, we use mini-batches of 32 sentences for IWSLT’14 and 64 for WMT. We use truncated back-propagation through time to limit the length of target sequences per mini-batch to 25 words. Gradients are normalized by the mini-batch size. We re-normalize the gradients if their norm exceeds 25 (Pascanu et al., 2013). Gradients of convolutional layers are scaled by sqrt(dim(input))−1 similar to Collobert et al. (2011b). We use dropout on the embeddings and decoder outputs hi with a rate of 0.2 for IWSLT’14 and 0.1 for WMT (Srivastava et al., 2014). All models are implemented in Torch (Collobert et al., 2011a) and trained on a single GPU.
4.4 EVALUATION
We report accuracy of single systems by training several identical models with different random seeds (5 for IWSLT’14, 3 for WMT) and pick the one with the best validation perplexity for final BLEU evaluation. Translations are generated by a beam search and we normalize log-likelihood scores by sentence length. On IWSLT’14 we use a beam width of 10 and for WMT models we tune beam width and word penalty on a separate test set, that is newsdev2016 for WMT’16 EnglishRomanian, newstest2014 for WMT’15 English-German and ntst1213 for WMT’14 English-French.4 The word penalty adds a constant factor to log-likelihoods, except for the end-of-sentence token.
Prior to scoring the generated translations against the respective references, we perform unknown word replacement based on attention scores (Jean et al., 2015). Unknown words are replaced by looking up the source word with the maximum attention score in a pre-computed dictionary. If the dictionary contains no translation, then we simply copy the source word. Dictionaries were extracted from the aligned training data that was aligned with fast align (Dyer et al., 2013). Each source word is mapped to the target word it is most frequently aligned to.
For convolutional encoders with stacked CNN-c layers we noticed for some models that the attention maxima were consistently shifted by one word. We determine this per-model offset on the abovementioned development sets and correct for it. Finally, we compute case-sensitive tokenized BLEU, except for WMT’16 English-Romanian where we use detokenized BLEU to be comparable with Sennrich et al. (2016b).5
5 RESULTS
5.1 RECURRENT VS. NON-RECURRENT ENCODERS
We first compare recurrent and non-recurrent encoders in terms of perplexity and BLEU on IWSLT’14 with and without position embeddings (§3.1) and include a phrase-based system (Koehn et al., 2007). Table 1 shows that a single-layer convolutional model with position embeddings (Convolutional) can outperform both a uni-directional LSTM encoder (LSTM) as well as a bi-directional LSTM encoder (BiLSTM). Next, we increase the depth of the convolutional encoder. We choose a good setting by independently varying the number of layers in CNN-a and CNN-c between 1 and 10 and obtained best validation set perplexity with six layers for CNN-a and three layers for CNN-c.
4Specifically, we select a beam from {5, 10} and a word penalty from {0,−0.5,−1,−1.5} 5https://github.com/moses-smt/mosesdecoder/blob/617e8c8ed1630fb1d1/
scripts/generic/{multi-bleu.perl,mteval-v13a.pl}
This configuration outperforms BiLSTM by 0.7 BLEU (Deep Convolutional 6/3). We investigate depth in the convolutional encoder more in §5.3. Among recurrent encoders, the BiLSTM is 2.3 BLEU better than the uni-directional version. The simple pooling encoder which does not contain any parameters is only 1.3 BLEU lower than a uni-directional LSTM encoder and 3.6 BLEU lower than BiLSTM. The results without position embeddings (words) show that position information is crucial for convolutional encoders. In particular for shallow models (Pooling and Convolutional), whereas deeper models are less effected. Recurrent encoders do not benefit from explicit position information because this information can be naturally extracted through the sequential computation.
When tuning model settings, we generally observe good correlation between perplexity and BLEU. However, for convolutional encoders perplexity gains translate to smaller BLEU improvements compared to recurrent counterparts (Table 1). We observe a similar trend on larger datasets.
5.2 EVALUATION ON WMT CORPORA
Next, we evaluate the BiLSTM encoder and the convolutional encoder architecture on three larger tasks and compare against previously published results. On WMT’16 English-Romanian translation we compare to Sennrich et al. (2016b), the winning single system entry for this language pair. Their model consists of a bi-directional GRU encoder, a GRU decoder and MLP-based attention. They use byte pair encoding (BPE) to achieve open-vocabulary translation and dropout in all components of the neural network to achieve 28.1 BLEU; we use the same pre-processing but no BPE (§4). The results (Table 2) show that a deep convolutional encoder performs competitively to the state of the art on this dataset (Sennrich et al., 2016b). Our bi-directional LSTM encoder baseline is 0.7 BLEU lower than Sennrich et al. (2016b) but uses only 512 hidden units compared to 1024. A single-layer convolutional encoder with embedding size 256 performs very competitively at 27.1 BLEU which is only 0.3 BLEU below the BiLSTM baseline. Increasing the number of convolutional layers to 8 in CNN-a and 4 in CNN-c achieves 27.8 BLEU which performs well above this baseline.
On WMT’15 English to German, we compare to a BiLSTM baseline and prior work: Jean et al. (2015) introduce a large output vocabulary; the decoder of Chung et al. (2016) operates on the character-level; Yang et al. (2016) uses LSTMs instead of GRUs and feeds the conditional input to the output layer as well as to the decoder.
Our single-layer BiLSTM baseline performs competitively compared to prior work and a two-layer BiLSTM performs about 0.4 BLEU better at 23.6 BLEU. Previous work also used multi-layer setups, e.g., Chung et al. (2016) has two layers both in the encoder and the decoder with 1024 hidden units, and Yang et al. (2016) use 1000 hidden units per LSTM. We use 512 hidden units for both LSTM and convolutional encoders. A single-layer CNN encoder (Convolutional) achieves 22.0 BLEU which is significantly lower than the two-layer BiLSTM. However, adding additional layers (Deep Convolutional 8/4) achieves the same accuracy as the two-layer BiLSTM and a 15 layer CNN-a outperforms it by 0.7 BLEU (Deep Convolutional 15/5). The latter performs competitively to the best published results which use decoder improvements that may benefit our setup as well.
Finally, we evaluate on the larger WMT’14 English-French corpus. On this dataset the recurrent architectures benefit from an additional layer both in the encoder and the decoder. For a singlelayer decoder, a deep convolutional encoder matches the BiLSTM accuracy and for a two-layer decoder, our very deep convolutional encoder with up to 20 layers outperforms the BiLSTM by 0.4 BLEU. It has 40% fewer parameters than the BiLSTM due to the smaller embedding sizes. We also outperform several previous systems, including the very deep encoder-decoder model proposed by Luong et al. (2015a). Our best result is just 0.2 BLEU below Zhou et al. (2016) who use a very deep LSTM setup with a 9-layer encoder, a 7-layer decoder, shortcut connections and extensive regularization with dropout and L2 regularization.
5.3 CONVOLUTIONAL ENCODER ARCHITECTURE DETAILS
We next motivate our design of the convolutional encoder (§3.2). We use the smaller IWSLT’14 German-English setup without unknown word replacement to enable fast experimental turn-around. BLEU results are averaged over three training runs initialized with different seeds.
Figure 2 shows accuracy for a different number of layers of both CNNs with and without residual connections. Our first observation is that computing the conditional input ci directly over embeddings e (line ”without CNN-c”) is already working well at 28.3 BLEU with a single CNN-a layer and at 29.1 BLEU for CNN-a with 7 layers (Figure 2a). Increasing the number of CNN-c layers is beneficial up to three layers and beyond this we did not observe further improvements. Similarly, increasing the number of layers in CNN-a beyond six does not increase accuracy on this relatively small dataset. In general, choosing two to three times as many layers in CNN-a as in CNN-c is a good rule of thumb. Without residual connections, the model fails to utilize the increase in modeling power from additional layers, and performance drops significantly for deeper encoders (Figure 2b).
Our convolutional architecture relies on two sets of networks, CNN-a for attention score computation ai and CNN-c for the conditional input ci to be fed to the decoder. We found that using the same network for both tasks, similar to recurrent encoders, resulted in poor accuracy of 22.9 BLEU. This compares to 28.5 BLEU for separate single-layer networks, or 28.3 BLEU when aggregating embeddings for ci. Increasing the number of layers in the single network setup did not help. Figure 2(a) suggests that the attention weights (CNN-a) need to integrate information from a wide context which can be done with a deep stack. At the same time, the vectors which are averaged (CNN-c) seem to benefit from a shallower, more local representation closer to the input words. Two stacks are an easy way to achieve these contradicting requirements.
In Appendix A we visualize attention scores and find that alignments for CNN encoders are less sharp compared to BiLSTMs, however, this does not affect the effectiveness of unknown word replacement once we adjust for shifted maxima. In Appendix B we investigate whether deep convolutional encoders are required for translating long sentences and observe that even relatively shallow encoders perform well on long sentences.
5.4 TRAINING AND GENERATION SPEED
For training, we use the fast CuDNN LSTM implementation for layers without attention and experiment on IWSLT’14 with batch size 32. The single-layer BiLSTM model trains at 4300 target words/second, while as the 6/3 deep convolutional encoder compares at 5500 words/second on an NVidia Tesla M40 GPU. We do not observe shorter overall training time since SGD converges slower than Adam which we use for BiLSTM models.
We measure generation speed on an Intel Haswell CPU clocked at 2.50GHz with a single thread for BLAS operations. We use vocabulary selection which can speed up generation by up to a factor of ten at no cost in accuracy via making the time to compute the final output layer negligible (Mi et al., 2016; L’Hostis et al., 2016). This shifts the focus from the efficiency of the encoder to the efficiency of the decoder. On IWSLT’14 (Table 3a) the convolutional encoder increases the speed of
the overall model by a factor of 1.35 compared to the BiLSTM encoder while improving accuracy by 0.7 BLEU. In this setup both encoders models have the same hidden layer and embedding sizes.
On the larger WMT’15 English-German task (Table 3b) the convolutional encoder speeds up generation by 2.1 times compared to a two-layer BiLSTM. This corresponds to 231 source words/second with beam size 5. Our best model on this dataset generates 203 words/second but at slightly lower accuracy compared to the full vocabulary setting in Table 2. The recurrent encoder uses larger embeddings than the convolutional encoder which were required for the models to match in accuracy.
The smaller embedding size is not the only reason for the speed-up. In Table 3a (a), we compare a Conv 6/3 encoder and a BiLSTM with equal embedding sizes. The convolutional encoder is still 1.34x faster (at 0.7 higher BLEU) although it requires roughly 1.6x as many FLOPs. We believe that this is likely due to better cache locality for convolutional layers on CPUs: an LSTM with fused gates6 requires two big matrix multiplications with different weights as well as additions, multiplications and non-linearities for each source word, while the output of each convolutional layer can be computed as whole with a single matrix multiply.
For comparison, the quantized deep LSTM-based model in Wu et al. (2016) processes 106.4 words/second for English-French on a CPU with 88 cores and 358.8 words/second on a custom TPU chip. The optimized RNNsearch model and C++ decoder described by Junczys-Dowmunt et al. (2016) translates 265.3 words/s on a CPU with a similar vocabulary selection technique, computing 16 sentences in parallel, i.e., 16.6 words/s on a single core.
6 CONCLUSION
We introduced a simple encoder model for neural machine translation based on convolutional networks. This approach is more parallelizable than recurrent networks and provides a shorter path to capture long-range dependencies in the source. We find it essential to use source position embeddings as well as different CNNs for attention score computation and conditional input aggregation.
Our experiments show that convolutional encoders perform on par or better than baselines based on bi-directional LSTM encoders. In comparison to other recent work, our deep convolutional encoder is very competitive to the best published results to date (WMT’16 English-Romanian) which are obtained with significantly more complex models (WMT’14 English-French) or stem from improvements that are orthogonal to our work (WMT’15 English-German). Our architecture also leads to large generation speed improvements: translation models with our convolutional encoder can translate twice as fast as strong baselines with bi-directional recurrent encoders.
Future work includes better training to enable faster convergence with the convolutional encoder to better leverage the higher processing speed. Our fast architecture is interesting for character level encoders where the input is significantly longer than for words. Also, we plan to investigate the effectiveness of our architecture on other sequence-to-sequence tasks, e.g. summarization, constituency parsing, dialog modeling.
ACKNOWLEDGMENTS
We would like to thank Sumit Chopra and Marc’Aurelio Ranzato for helpful discussions related to this work.
A ALIGNMENT VISUALIZATION
In Figure 4 and Figure 5, we plot attention scores for a sample WMT’15 English-German and WMT’14 English-French translation with BiLSTM and deep convolutional encoders. The translation is on the x-axis and the source sentence on the y-axis.
The attention scores of the BiLSTM output are sharp but do not necessarily represent a correct alignment. For CNN encoders the scores are less focused but still indicate an approximate source location, e.g., in Figure 4b, when moving the clause ”over 1,000 people were taken hostage” to the back of the translation. For some models, attention maxima are consistently shifted by one token as both in Figure 4b and Figure 5b.
Interestingly, convolutional encoders tend to focus on the last token (Figure 4b) or both the first and last tokens (Figure 5b). Motivated by the hypothesis that the this may be due to the decoder depending on the length of the source sentence (which it cannot determine without position embeddings), we explicitly provided a distributed representation of the input length to the decoder and attention module. However, this did not cause a change in attention patterns nor did it improve translation accuracy.
B PERFORMANCE BY SENTENCE LENGTH
One characteristic of our convolutional encoder architecture is that the context over which outputs are computed depends on the number of layers. With bi-directional RNNs, every encoder output depends on the entire source sentence. In Figure 3, we evaluate whether limited context affects the translation quality on longer sentences of WMT’15 English-German which often requires moving verbs over long distances. We sort the newstest2015 test set by source length, partition it into 15 equally-sized buckets, and compare the BLEU scores of models listed in Table 2 on a per-bucket basis.
There is no clear evidence for sub-par translations on sentences that are longer than the observable context per encoder output. We include a small encoder with a 6-layer CNN-c and a 3-layer CNN-a in the comparison which performs worse than a 2-layer BiLSTM (23.3 BLEU vs2̇3.6). With 6 convolutional layers at kernel width 3, each encoder output contains information of 13 adjacent source words . Looking at the accuracy for sentences with 15 words or more, this relatively shallow CNN is either on par or better than the BiLSTM for 5 out of 10 buckets; the BiLSTM has access to the entire source context. Similar observations can be made for the deeper convolutional encoders.

The system described works comparably to bi-directional LSTM baseline for NMT, and CNN's are naturally parallelizable.
Key ideas include the use of two stacked CNN's (one for each of encoding and decoding) for translation, with res connections and position embeddings. The use of CNN's for translation has been attempted previously (as described by the authors), but presumably it is the authors' combination of various architectural choices (attention, position embeddings, etc) that make the present system competitive with RNN's, whereas earlier attempts were not. They describe system's sensitivity to some of these choices (e.g. experiments to choose appropriate number of layers in each of the CNN's).
The experimental results are well reported in detail.
One or two figures would definitely be required to help clarify the architecture.
This paper is less about new ways of learning representations than about the combination of choices made (over the set of existing techniques) in order to get the good results that they do on the reported NMT tasks. In this respect, while I am fairly confident that the paper represents good work in machine learning, I am not quite as confident about its fit for this particular conference.