Recent research on deep convolutional neural networks (CNNs) has focused primarily on improving accuracy. For a given accuracy level, it is typically possible to identify multiple CNN architectures that achieve that accuracy level. With equivalent accuracy, smaller CNN architectures offer at least three advantages: (1) Smaller CNNs require less communication across servers during distributed training. (2) Smaller CNNs require less bandwidth to export a new model from the cloud to an autonomous car. (3) Smaller CNNs are more feasible to deploy on FPGAs and other hardware with limited memory. To provide all of these advantages, we propose a small CNN architecture called SqueezeNet. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. Additionally, with model compression techniques, we are able to compress SqueezeNet to less than 0.5MB (510× smaller than AlexNet). The SqueezeNet architecture is available for download here: https://github.com/DeepScale/SqueezeNet
1 INTRODUCTION AND MOTIVATION
Much of the recent research on deep convolutional neural networks (CNNs) has focused on increasing accuracy on computer vision datasets. For a given accuracy level, there typically exist multiple CNN architectures that achieve that accuracy level. Given equivalent accuracy, a CNN architecture with fewer parameters has several advantages:
• More efficient distributed training. Communication among servers is the limiting factor to the scalability of distributed CNN training. For distributed data-parallel training, communication overhead is directly proportional to the number of parameters in the model (Iandola et al., 2016). In short, small models train faster due to requiring less communication.
• Less overhead when exporting new models to clients. For autonomous driving, companies such as Tesla periodically copy new models from their servers to customers’ cars. This practice is often referred to as an over-the-air update. Consumer Reports has found that the safety of Tesla’s Autopilot semi-autonomous driving functionality has incrementally improved with recent over-the-air updates (Consumer Reports, 2016). However, over-theair updates of today’s typical CNN/DNN models can require large data transfers. With AlexNet, this would require 240MB of communication from the server to the car. Smaller models require less communication, making frequent updates more feasible. • Feasible FPGA and embedded deployment. FPGAs often have less than 10MB1 of onchip memory and no off-chip memory or storage. For inference, a sufficiently small model could be stored directly on the FPGA instead of being bottlenecked by memory bandwidth (Qiu et al., 2016), while video frames stream through the FPGA in real time. Further, when deploying CNNs on Application-Specific Integrated Circuits (ASICs), a sufficiently small model could be stored directly on-chip, and smaller models may enable the ASIC to fit on a smaller die.
∗http://deepscale.ai 1For example, the Xilinx Vertex-7 FPGA has a maximum of 8.5 MBytes (i.e. 68 Mbits) of on-chip memory
and does not provide off-chip memory.
As you can see, there are several advantages of smaller CNN architectures. With this in mind, we focus directly on the problem of identifying a CNN architecture with fewer parameters but equivalent accuracy compared to a well-known model. We have discovered such an architecture, which we call SqueezeNet. In addition, we present our attempt at a more disciplined approach to searching the design space for novel CNN architectures.
The rest of the paper is organized as follows. In Section 2 we review the related work. Then, in Sections 3 and 4 we describe and evaluate the SqueezeNet architecture. After that, we turn our attention to understanding how CNN architectural design choices impact model size and accuracy. We gain this understanding by exploring the design space of SqueezeNet-like architectures. In Section 5, we do design space exploration on the CNN microarchitecture, which we define as the organization and dimensionality of individual layers and modules. In Section 6, we do design space exploration on the CNN macroarchitecture, which we define as high-level organization of layers in a CNN. Finally, we conclude in Section 7. In short, Sections 3 and 4 are useful for CNN researchers as well as practitioners who simply want to apply SqueezeNet to a new application. The remaining sections are aimed at advanced researchers who intend to design their own CNN architectures.
2 RELATED WORK
2.1 MODEL COMPRESSION
The overarching goal of our work is to identify a model that has very few parameters while preserving accuracy. To address this problem, a sensible approach is to take an existing CNN model and compress it in a lossy fashion. In fact, a research community has emerged around the topic of model compression, and several approaches have been reported. A fairly straightforward approach by Denton et al. is to apply singular value decomposition (SVD) to a pretrained CNN model (Denton et al., 2014). Han et al. developed Network Pruning, which begins with a pretrained model, then replaces parameters that are below a certain threshold with zeros to form a sparse matrix, and finally performs a few iterations of training on the sparse CNN (Han et al., 2015b). Recently, Han et al. extended their work by combining Network Pruning with quantization (to 8 bits or less) and huffman encoding to create an approach called Deep Compression (Han et al., 2015a), and further designed a hardware accelerator called EIE (Han et al., 2016a) that operates directly on the compressed model, achieving substantial speedups and energy savings.
2.2 CNN MICROARCHITECTURE
Convolutions have been used in artificial neural networks for at least 25 years; LeCun et al. helped to popularize CNNs for digit recognition applications in the late 1980s (LeCun et al., 1989). In neural networks, convolution filters are typically 3D, with height, width, and channels as the key dimensions. When applied to images, CNN filters typically have 3 channels in their first layer (i.e. RGB), and in each subsequent layer Li the filters have the same number of channels as Li−1 has filters. The early work by LeCun et al. (LeCun et al., 1989) uses 5x5xChannels2 filters, and the recent VGG (Simonyan & Zisserman, 2014) architectures extensively use 3x3 filters. Models such as Network-in-Network (Lin et al., 2013) and the GoogLeNet family of architectures (Szegedy et al., 2014; Ioffe & Szegedy, 2015; Szegedy et al., 2015; 2016) use 1x1 filters in some layers.
With the trend of designing very deep CNNs, it becomes cumbersome to manually select filter dimensions for each layer. To address this, various higher level building blocks, or modules, comprised of multiple convolution layers with a specific fixed organization have been proposed. For example, the GoogLeNet papers propose Inception modules, which are comprised of a number of different dimensionalities of filters, usually including 1x1 and 3x3, plus sometimes 5x5 (Szegedy et al., 2014) and sometimes 1x3 and 3x1 (Szegedy et al., 2015). Many such modules are then combined, perhaps with additional ad-hoc layers, to form a complete network. We use the term CNN microarchitecture to refer to the particular organization and dimensions of the individual modules.
2.3 CNN MACROARCHITECTURE
While the CNN microarchitecture refers to individual layers and modules, we define the CNN macroarchitecture as the system-level organization of multiple modules into an end-to-end CNN architecture.
2From now on, we will simply abbreviate HxWxChannels to HxW.
Perhaps the mostly widely studied CNN macroarchitecture topic in the recent literature is the impact of depth (i.e. number of layers) in networks. Simoyan and Zisserman proposed the VGG (Simonyan & Zisserman, 2014) family of CNNs with 12 to 19 layers and reported that deeper networks produce higher accuracy on the ImageNet-1k dataset (Deng et al., 2009). K. He et al. proposed deeper CNNs with up to 30 layers that deliver even higher ImageNet accuracy (He et al., 2015a).
The choice of connections across multiple layers or modules is an emerging area of CNN macroarchitectural research. Residual Networks (ResNet) (He et al., 2015b) and Highway Networks (Srivastava et al., 2015) each propose the use of connections that skip over multiple layers, for example additively connecting the activations from layer 3 to the activations from layer 6. We refer to these connections as bypass connections. The authors of ResNet provide an A/B comparison of a 34-layer CNN with and without bypass connections; adding bypass connections delivers a 2 percentage-point improvement on Top-5 ImageNet accuracy.
2.4 NEURAL NETWORK DESIGN SPACE EXPLORATION
Neural networks (including deep and convolutional NNs) have a large design space, with numerous options for microarchitectures, macroarchitectures, solvers, and other hyperparameters. It seems natural that the community would want to gain intuition about how these factors impact a NN’s accuracy (i.e. the shape of the design space). Much of the work on design space exploration (DSE) of NNs has focused on developing automated approaches for finding NN architectures that deliver higher accuracy. These automated DSE approaches include bayesian optimization (Snoek et al., 2012), simulated annealing (Ludermir et al., 2006), randomized search (Bergstra & Bengio, 2012), and genetic algorithms (Stanley & Miikkulainen, 2002). To their credit, each of these papers provides a case in which the proposed DSE approach produces a NN architecture that achieves higher accuracy compared to a representative baseline. However, these papers make no attempt to provide intuition about the shape of the NN design space. Later in this paper, we eschew automated approaches – instead, we refactor CNNs in such a way that we can do principled A/B comparisons to investigate how CNN architectural decisions influence model size and accuracy.
In the following sections, we first propose and evaluate the SqueezeNet architecture with and without model compression. Then, we explore the impact of design choices in microarchitecture and macroarchitecture for SqueezeNet-like CNN architectures.
3 SQUEEZENET: PRESERVING ACCURACY WITH FEW PARAMETERS
In this section, we begin by outlining our design strategies for CNN architectures with few parameters. Then, we introduce the Fire module, our new building block out of which to build CNN architectures. Finally, we use our design strategies to construct SqueezeNet, which is comprised mainly of Fire modules.
3.1 ARCHITECTURAL DESIGN STRATEGIES
Our overarching objective in this paper is to identify CNN architectures that have few parameters while maintaining competitive accuracy. To achieve this, we employ three main strategies when designing CNN architectures:
Strategy 1. Replace 3x3 filters with 1x1 filters. Given a budget of a certain number of convolution filters, we will choose to make the majority of these filters 1x1, since a 1x1 filter has 9X fewer parameters than a 3x3 filter.
Strategy 2. Decrease the number of input channels to 3x3 filters. Consider a convolution layer that is comprised entirely of 3x3 filters. The total quantity of parameters in this layer is (number of input channels) * (number of filters) * (3*3). So, to maintain a small total number of parameters in a CNN, it is important not only to decrease the number of 3x3 filters (see Strategy 1 above), but also to decrease the number of input channels to the 3x3 filters. We decrease the number of input channels to 3x3 filters using squeeze layers, which we describe in the next section.
Strategy 3. Downsample late in the network so that convolution layers have large activation maps. In a convolutional network, each convolution layer produces an output activation map with a spatial resolution that is at least 1x1 and often much larger than 1x1. The height and width of these activation maps are controlled by: (1) the size of the input data (e.g. 256x256 images) and (2)
Under review as a conference paper at ICLR 2017
h"p://www.presenta.on0process.com/lego0blocks0in0powerpoint.html88
the choice of layers in which to downsample in the CNN architecture. Most commonly, downsampling is engineered into CNN architectures by setting the (stride > 1) in some of the convolution or pooling layers (e.g. (Szegedy et al., 2014; Simonyan & Zisserman, 2014; Krizhevsky et al., 2012)). If early3 layers in the network have large strides, then most layers will have small activation maps. Conversely, if most layers in the network have a stride of 1, and the strides greater than 1 are concentrated toward the end4 of the network, then many layers in the network will have large activation maps. Our intuition is that large activation maps (due to delayed downsampling) can lead to higher classification accuracy, with all else held equal. Indeed, K. He and H. Sun applied delayed downsampling to four different CNN architectures, and in each case delayed downsampling led to higher classification accuracy (He & Sun, 2015).
Strategies 1 and 2 are about judiciously decreasing the quantity of parameters in a CNN while attempting to preserve accuracy. Strategy 3 is about maximizing accuracy on a limited budget of parameters. Next, we describe the Fire module, which is our building block for CNN architectures that enables us to successfully employ Strategies 1, 2, and 3.
3.2 THE FIRE MODULE
We define the Fire module as follows. A Fire module is comprised of: a squeeze convolution layer (which has only 1x1 filters), feeding into an expand layer that has a mix of 1x1 and 3x3 convolution filters; we illustrate this in Figure 1. The liberal use of 1x1 filters in Fire modules is an application of Strategy 1 from Section 3.1. We expose three tunable dimensions (hyperparameters) in a Fire module: s1x1, e1x1, and e3x3. In a Fire module, s1x1 is the number of filters in the squeeze layer (all 1x1), e1x1 is the number of 1x1 filters in the expand layer, and e3x3 is the number of 3x3 filters in the expand layer. When we use Fire modules we set s1x1 to be less than (e1x1 + e3x3), so the squeeze layer helps to limit the number of input channels to the 3x3 filters, as per Strategy 2 from Section 3.1.
3.3 THE SQUEEZENET ARCHITECTURE
We now describe the SqueezeNet CNN architecture. We illustrate in Figure 2 that SqueezeNet begins with a standalone convolution layer (conv1), followed by 8 Fire modules (fire2-9), ending with a final conv layer (conv10). We gradually increase the number of filters per fire module from the beginning to the end of the network. SqueezeNet performs max-pooling with a stride of 2 after layers conv1, fire4, fire8, and conv10; these relatively late placements of pooling are per Strategy 3 from Section 3.1. We present the full SqueezeNet architecture in Table 1.
3In our terminology, an “early” layer is close to the input data. 4In our terminology, the “end” of the network is the classifier.
3.3.1 OTHER SQUEEZENET DETAILS
For brevity, we have omitted number of details and design choices about SqueezeNet from Table 1 and Figure 2. We provide these design choices in the following. The intuition behind these choices may be found in the papers cited below.
• So that the output activations from 1x1 and 3x3 filters have the same height and width, we add a 1-pixel border of zero-padding in the input data to 3x3 filters of expand modules.
• ReLU (Nair & Hinton, 2010) is applied to activations from squeeze and expand layers. • Dropout (Srivastava et al., 2014) with a ratio of 50% is applied after the fire9 module. • Note the lack of fully-connected layers in SqueezeNet; this design choice was inspired by
the NiN (Lin et al., 2013) architecture. • When training SqueezeNet, we begin with a learning rate of 0.04, and we lin-
early decrease the learning rate throughout training, as described in (Mishkin et al., 2016). For details on the training protocol (e.g. batch size, learning rate, parameter initialization), please refer to our Caffe-compatible configuration files located here: https://github.com/DeepScale/SqueezeNet.
• The Caffe framework does not natively support a convolution layer that contains multiple filter resolutions (e.g. 1x1 and 3x3) (Jia et al., 2014). To get around this, we implement our expand layer with two separate convolution layers: a layer with 1x1 filters, and a layer with 3x3 filters. Then, we concatenate the outputs of these layers together in the channel dimension. This is numerically equivalent to implementing one layer that contains both 1x1 and 3x3 filters.
We released the SqueezeNet configuration files in the format defined by the Caffe CNN framework. However, in addition to Caffe, several other CNN frameworks have emerged, including MXNet (Chen et al., 2015a), Chainer (Tokui et al., 2015), Keras (Chollet, 2016), and Torch (Collobert et al., 2011). Each of these has its own native format for representing a CNN architecture. That said, most of these libraries use the same underlying computational back-ends such as cuDNN (Chetlur et al., 2014) and MKL-DNN (Das et al., 2016). The research community has
ported the SqueezeNet CNN architecture for compatibility with a number of other CNN software frameworks:
• MXNet (Chen et al., 2015a) port of SqueezeNet: (Haria, 2016) • Chainer (Tokui et al., 2015) port of SqueezeNet: (Bell, 2016) • Keras (Chollet, 2016) port of SqueezeNet: (DT42, 2016) • Torch (Collobert et al., 2011) port of SqueezeNet’s Fire Modules: (Waghmare, 2016)
4 EVALUATION OF SQUEEZENET
We now turn our attention to evaluating SqueezeNet. In each of the CNN model compression papers reviewed in Section 2.1, the goal was to compress an AlexNet (Krizhevsky et al., 2012) model that was trained to classify images using the ImageNet (Deng et al., 2009) (ILSVRC 2012) dataset. Therefore, we use AlexNet5 and the associated model compression results as a basis for comparison when evaluating SqueezeNet.
In Table 2, we review SqueezeNet in the context of recent model compression results. The SVDbased approach is able to compress a pretrained AlexNet model by a factor of 5x, while diminishing top-1 accuracy to 56.0% (Denton et al., 2014). Network Pruning achieves a 9x reduction in model size while maintaining the baseline of 57.2% top-1 and 80.3% top-5 accuracy on ImageNet (Han et al., 2015b). Deep Compression achieves a 35x reduction in model size while still maintaining the baseline accuracy level (Han et al., 2015a). Now, with SqueezeNet, we achieve a 50X reduction in model size compared to AlexNet, while meeting or exceeding the top-1 and top-5 accuracy of AlexNet. We summarize all of the aforementioned results in Table 2.
It appears that we have surpassed the state-of-the-art results from the model compression community: even when using uncompressed 32-bit values to represent the model, SqueezeNet has a 1.4× smaller model size than the best efforts from the model compression community while maintaining or exceeding the baseline accuracy. Until now, an open question has been: are small models amenable to compression, or do small models “need” all of the representational power afforded by dense floating-point values? To find out, we applied Deep Compression (Han et al., 2015a)
5Our baseline is bvlc alexnet from the Caffe codebase (Jia et al., 2014).
6
to SqueezeNet, using 33% sparsity6 and 8-bit quantization. This yields a 0.66 MB model (363× smaller than 32-bit AlexNet) with equivalent accuracy to AlexNet. Further, applying Deep Compression with 6-bit quantization and 33% sparsity on SqueezeNet, we produce a 0.47MB model (510× smaller than 32-bit AlexNet) with equivalent accuracy. Our small model is indeed amenable to compression.
In addition, these results demonstrate that Deep Compression (Han et al., 2015a) not only works well on CNN architectures with many parameters (e.g. AlexNet and VGG), but it is also able to compress the already compact, fully convolutional SqueezeNet architecture. Deep Compression compressed SqueezeNet by 10× while preserving the baseline accuracy. In summary: by combining CNN architectural innovation (SqueezeNet) with state-of-the-art compression techniques (Deep Compression), we achieved a 510× reduction in model size with no decrease in accuracy compared to the baseline.
Finally, note that Deep Compression (Han et al., 2015b) uses a codebook as part of its scheme for quantizing CNN parameters to 6- or 8-bits of precision. Therefore, on most commodity processors, it is not trivial to achieve a speedup of 328 = 4x with 8-bit quantization or 32 6 = 5.3x with 6-bit quantization using the scheme developed in Deep Compression. However, Han et al. developed custom hardware – Efficient Inference Engine (EIE) – that can compute codebook-quantized CNNs more efficiently (Han et al., 2016a). In addition, in the months since we released SqueezeNet, P. Gysel developed a strategy called Ristretto for linearly quantizing SqueezeNet to 8 bits (Gysel, 2016). Specifically, Ristretto does computation in 8 bits, and it stores parameters and activations in 8-bit data types. Using the Ristretto strategy for 8-bit computation in SqueezeNet inference, Gysel observed less than 1 percentage-point of drop in accuracy when using 8-bit instead of 32-bit data types.
5 CNN MICROARCHITECTURE DESIGN SPACE EXPLORATION
So far, we have proposed architectural design strategies for small models, followed these principles to create SqueezeNet, and discovered that SqueezeNet is 50x smaller than AlexNet with equivalent accuracy. However, SqueezeNet and other models reside in a broad and largely unexplored design space of CNN architectures. Now, in Sections 5 and 6, we explore several aspects of the design space. We divide this architectural exploration into two main topics: microarchitectural exploration (per-module layer dimensions and configurations) and macroarchitectural exploration (high-level end-to-end organization of modules and other layers).
In this section, we design and execute experiments with the goal of providing intuition about the shape of the microarchitectural design space with respect to the design strategies that we proposed in Section 3.1. Note that our goal here is not to maximize accuracy in every experiment, but rather to understand the impact of CNN architectural choices on model size and accuracy.
6Note that, due to the storage overhead of storing sparse matrix indices, 33% sparsity leads to somewhat less than a 3× decrease in model size.
5.1 CNN MICROARCHITECTURE METAPARAMETERS
In SqueezeNet, each Fire module has three dimensional hyperparameters that we defined in Section 3.2: s1x1, e1x1, and e3x3. SqueezeNet has 8 Fire modules with a total of 24 dimensional hyperparameters. To do broad sweeps of the design space of SqueezeNet-like architectures, we define the following set of higher level metaparameters which control the dimensions of all Fire modules in a CNN. We define basee as the number of expand filters in the first Fire module in a CNN. After every freq Fire modules, we increase the number of expand filters by incre. In other words, for Fire module i, the number of expand filters is ei = basee + (incre ∗ ⌊ i
freq
⌋ ). In the
expand layer of a Fire module, some filters are 1x1 and some are 3x3; we define ei = ei,1x1+ ei,3x3 with pct3x3 (in the range [0, 1], shared over all Fire modules) as the percentage of expand filters that are 3x3. In other words, ei,3x3 = ei ∗ pct3x3, and ei,1x1 = ei ∗ (1 − pct3x3). Finally, we define the number of filters in the squeeze layer of a Fire module using a metaparameter called the squeeze ratio (SR) (again, in the range [0, 1], shared by all Fire modules): si,1x1 = SR ∗ ei (or equivalently si,1x1 = SR ∗ (ei,1x1 + ei,3x3)). SqueezeNet (Table 1) is an example architecture that we generated with the aforementioned set of metaparameters. Specifically, SqueezeNet has the following metaparameters: basee = 128, incre = 128, pct3x3 = 0.5, freq = 2, and SR = 0.125.
5.2 SQUEEZE RATIO
In Section 3.1, we proposed decreasing the number of parameters by using squeeze layers to decrease the number of input channels seen by 3x3 filters. We defined the squeeze ratio (SR) as the ratio between the number of filters in squeeze layers and the number of filters in expand layers. We now design an experiment to investigate the effect of the squeeze ratio on model size and accuracy.
In these experiments, we use SqueezeNet (Figure 2) as a starting point. As in SqueezeNet, these experiments use the following metaparameters: basee = 128, incre = 128, pct3x3 = 0.5, and freq = 2. We train multiple models, where each model has a different squeeze ratio (SR)7 in the range [0.125, 1.0]. In Figure 3(a), we show the results of this experiment, where each point on the graph is an independent model that was trained from scratch. SqueezeNet is the SR=0.125 point in this figure.8 From this figure, we learn that increasing SR beyond 0.125 can further increase ImageNet top-5 accuracy from 80.3% (i.e. AlexNet-level) with a 4.8MB model to 86.0% with a 19MB model. Accuracy plateaus at 86.0% with SR=0.75 (a 19MB model), and setting SR=1.0 further increases model size without improving accuracy.
5.3 TRADING OFF 1X1 AND 3X3 FILTERS
In Section 3.1, we proposed decreasing the number of parameters in a CNN by replacing some 3x3 filters with 1x1 filters. An open question is, how important is spatial resolution in CNNs? The
7Note that, for a given model, all Fire layers share the same squeeze ratio. 8Note that we named it SqueezeNet because it has a low squeeze ratio (SR). That is, the squeeze layers in
SqueezeNet have 0.125x the number of filters as the expand layers.
VGG (Simonyan & Zisserman, 2014) architectures have 3x3 spatial resolution in most layers’ filters; GoogLeNet (Szegedy et al., 2014) and Network-in-Network (NiN) (Lin et al., 2013) have 1x1 filters in some layers. In GoogLeNet and NiN, the authors simply propose a specific quantity of 1x1 and 3x3 filters without further analysis.9 Here, we attempt to shed light on how the proportion of 1x1 and 3x3 filters affects model size and accuracy.
We use the following metaparameters in this experiment: basee = incre = 128, freq = 2, SR = 0.500, and we vary pct3x3 from 1% to 99%. In other words, each Fire module’s expand layer has a predefined number of filters partitioned between 1x1 and 3x3, and here we turn the knob on these filters from “mostly 1x1” to “mostly 3x3”. As in the previous experiment, these models have 8 Fire modules, following the same organization of layers as in Figure 2. We show the results of this experiment in Figure 3(b). Note that the 13MB models in Figure 3(a) and Figure 3(b) are the same architecture: SR = 0.500 and pct3x3 = 50%. We see in Figure 3(b) that the top-5 accuracy plateaus at 85.6% using 50% 3x3 filters, and further increasing the percentage of 3x3 filters leads to a larger model size but provides no improvement in accuracy on ImageNet.
6 CNN MACROARCHITECTURE DESIGN SPACE EXPLORATION
So far we have explored the design space at the microarchitecture level, i.e. the contents of individual modules of the CNN. Now, we explore design decisions at the macroarchitecture level concerning the high-level connections among Fire modules. Inspired by ResNet (He et al., 2015b), we explored three different architectures:
• Vanilla SqueezeNet (as per the prior sections). • SqueezeNet with simple bypass connections between some Fire modules. (Inspired by (Sri-
vastava et al., 2015; He et al., 2015b).) • SqueezeNet with complex bypass connections between the remaining Fire modules.
We illustrate these three variants of SqueezeNet in Figure 2.
Our simple bypass architecture adds bypass connections around Fire modules 3, 5, 7, and 9, requiring these modules to learn a residual function between input and output. As in ResNet, to implement a bypass connection around Fire3, we set the input to Fire4 equal to (output of Fire2 + output of Fire3), where the + operator is elementwise addition. This changes the regularization applied to the parameters of these Fire modules, and, as per ResNet, can improve the final accuracy and/or ability to train the full model.
One limitation is that, in the straightforward case, the number of input channels and number of output channels has to be the same; as a result, only half of the Fire modules can have simple bypass connections, as shown in the middle diagram of Fig 2. When the “same number of channels” requirement can’t be met, we use a complex bypass connection, as illustrated on the right of Figure 2. While a simple bypass is “just a wire,” we define a complex bypass as a bypass that includes a 1x1 convolution layer with the number of filters set equal to the number of output channels that are needed. Note that complex bypass connections add extra parameters to the model, while simple bypass connections do not.
In addition to changing the regularization, it is intuitive to us that adding bypass connections would help to alleviate the representational bottleneck introduced by squeeze layers. In SqueezeNet, the squeeze ratio (SR) is 0.125, meaning that every squeeze layer has 8x fewer output channels than the accompanying expand layer. Due to this severe dimensionality reduction, a limited amount of information can pass through squeeze layers. However, by adding bypass connections to SqueezeNet, we open up avenues for information to flow around the squeeze layers.
We trained SqueezeNet with the three macroarchitectures in Figure 2 and compared the accuracy and model size in Table 3. We fixed the microarchitecture to match SqueezeNet as described in Table 1 throughout the macroarchitecture exploration. Complex and simple bypass connections both yielded an accuracy improvement over the vanilla SqueezeNet architecture. Interestingly, the simple bypass enabled a higher accuracy accuracy improvement than complex bypass. Adding the
9To be clear, each filter is 1x1xChannels or 3x3xChannels, which we abbreviate to 1x1 and 3x3.
simple bypass connections yielded an increase of 2.9 percentage-points in top-1 accuracy and 2.2 percentage-points in top-5 accuracy without increasing model size.
7 CONCLUSIONS
In this paper, we have proposed steps toward a more disciplined approach to the design-space exploration of convolutional neural networks. Toward this goal we have presented SqueezeNet, a CNN architecture that has 50× fewer parameters than AlexNet and maintains AlexNet-level accuracy on ImageNet. We also compressed SqueezeNet to less than 0.5MB, or 510× smaller than AlexNet without compression. Since we released this paper as a technical report in 2016, Song Han and his collaborators have experimented further with SqueezeNet and model compression. Using a new approach called Dense-Sparse-Dense (DSD) (Han et al., 2016b), Han et al. use model compression during training as a regularizer to further improve accuracy, producing a compressed set of SqueezeNet parameters that is 1.2 percentage-points more accurate on ImageNet-1k, and also producing an uncompressed set of SqueezeNet parameters that is 4.3 percentage-points more accurate, compared to our results in Table 2.
We mentioned near the beginning of this paper that small models are more amenable to on-chip implementations on FPGAs. Since we released the SqueezeNet model, Gschwend has developed a variant of SqueezeNet and implemented it on an FPGA (Gschwend, 2016). As we anticipated, Gschwend was able to able to store the parameters of a SqueezeNet-like model entirely within the FPGA and eliminate the need for off-chip memory accesses to load model parameters.
In the context of this paper, we focused on ImageNet as a target dataset. However, it has become common practice to apply ImageNet-trained CNN representations to a variety of applications such as fine-grained object recognition (Zhang et al., 2013; Donahue et al., 2013), logo identification in images (Iandola et al., 2015), and generating sentences about images (Fang et al., 2015). ImageNettrained CNNs have also been applied to a number of applications pertaining to autonomous driving, including pedestrian and vehicle detection in images (Iandola et al., 2014; Girshick et al., 2015; Ashraf et al., 2016) and videos (Chen et al., 2015b), as well as segmenting the shape of the road (Badrinarayanan et al., 2015). We think SqueezeNet will be a good candidate CNN architecture for a variety of applications, especially those in which small model size is of importance.
SqueezeNet is one of several new CNNs that we have discovered while broadly exploring the design space of CNN architectures. We hope that SqueezeNet will inspire the reader to consider and explore the broad range of possibilities in the design space of CNN architectures and to perform that exploration in a more systematic manner.

The Squeezenet paper came out in Feb 2016, and I read it with interest. It has a series of completely reasonable engineering suggestions for how to save parameter memory for CNNs for object recognition (imagenet). The suggestions make a lot of sense, and provide an excellent compression of about 50x versus AlexNet. (Looks like ~500x if combined with Han, 2015). So, very nice results, definitely worth publishing.
Since the arxiv paper came out, people have noticed and worked to extend the paper. This is already evidence that this paper will have impact --- and deserves to have a permanent published home.
On the negative side, the architecture was only tested on ImageNet -- unclear whether the ideas transfer to other tasks (e.g., audio or text recognition). And, as with many other architecture-tweaking papers, there is no real mathematical or theoretical support for the ideas: they are just sensible and empirically work.
Oh the whole, I think the paper deserves to appear at ICLR, being in the mainline of work on deep learning architectures.


Problem-specific algorithms and generic machine learning approaches have complementary strengths and weaknesses, trading-off data efficiency and generality. To find the right balance between these, we propose to use problem-specific information encoded in algorithms together with the ability to learn details about the problem-instance from data. We demonstrate this approach in the context of state estimation in robotics, where we propose end-to-end learnable histogram filters— a differentiable implementation of histogram filters that encodes the structure of recursive state estimation using prediction and measurement update but allows the specific models to be learned end-to-end, i.e. in such a way that they optimize the performance of the filter, using either supervised or unsupervised learning.
1 INTRODUCTION
Traditionally, computer scientists solve problems by designing algorithms. Recently, this practice has received competition from machine learning methods that automatically extract solutions from data. One example of this development is the field of computer vision, where the state of the art is based on deep neural networks rather than on human-designed algorithms (He et al., 2015). But these two approaches to problem solving—algorithms and learning—are not mutually exclusive; in fact, they can complement each other. Effective problem solving exploits all available information, whether it be encoded in algorithms or captured by data. This paper presents a step towards tightly combining these sources of information.
We demonstrate the combination of problem-specific algorithms with generic machine learning in the context of state estimation in robotics. The state estimation problem exhibits a clear algorithmic structure, captured in a provably optimal way by Bayes filters (Thrun et al., 2005). But the use of such a filter requires the specification of a motion model and a measurement model that is specific to a particular problem instance. We want to leverage the general knowledge captured in the Bayes filter, while extracting the instance-specific models from data using deep learning (Goodfellow et al., 2016). We achieve this by implementing a differentiable version of the histogram filter—a specific type of Bayes filter that represents probability distributions with histograms—including learnable motion and measurement models (see Fig. 1). With this implementation, we can learn these models end-to-end using backpropagation, while still taking advantage of the structure encoded in Bayes filters. Interestingly, this combination also enables unsupervised learning.
Our contributions are both conceptual and technical. Our conceptual contribution is the principle of tightly combining algorithms and machine learning to balance data-efficiency and generality. Our technical contribution is the end-to-end learnable histogram filter, which enables the use of this Bayes filter variant in a more generic way. Our experiments show that our method is more data-efficient than generic neural networks, improves performance compared to standard histogram filters, and—most importantly—enables unsupervised learning of recursive state estimation loops.
2 COMBINING ALGORITHMS AND MACHINE LEARNING
Every information that is contained in the solution to a problem must either be provided as prior knowledge (prior for short) or learned from data. Different approaches balance these sources of information differently. In the classic approach to computer science, all required information is provided by a human (e.g. in the form of algorithms and models). In the machine learning approach, only a minimal amount of prior knowledge is provided (in form of a learning algorithm) while most information is extracted from data (see Fig. 2). When trading-off how
much and which information should be provided as a prior or emerge from data, we should consider the entire spectrum rather than limit ourselves to these two end points.
In the context of robotics, for example, it is clear that the left end of this spectrum will not enable intelligent robots, because we cannot foresee and specify every detail for solving a wide range of tasks in initially unknown environments. Robots need to collect data and learn from them. But if we go all the way to the right end of the spectrum, we need large amounts of data, which is very difficult to obtain in robotics where data collection is slow and costly. Luckily, robotic tasks include rich structure that can be used as prior. Physics, for example, governs the interaction of any robot and its environment and physics-based priors can substantially improve learning (Scholz et al., 2014; Jonschkowski & Brock, 2015). But robotic tasks include additional structure that can be exploited.
Every algorithm that has proven successful in robotics implicitly encodes information about the structure of robotic tasks. We propose to use this robotics-specific information captured by robotic algorithms and combine it with machine learning to fill in the task-specific details based on data. By tightly combining algorithms and machine learning, we can strike the right balance between generality and data-efficiency.
3 RELATED WORK
Algorithms and machine learning can be combined in different ways, using algorithms either 1) as fixed parts of solutions, 2) as parts of the learning process, or 3) as both. The first approach learns task-specific models in isolation and then combines them with algorithms in the solution. Examples for this approach are numerous, e.g. a Go player that applies a planning algorithm on learned models (Silver et al., 2016), a perception pipeline that combines the iterative closest point algorithm with learned object segmentation (Zeng et al., 2016), or robot control based on learned motion models (Nguyen-Tuong & Peters, 2011).
The second approach uses algorithms as teachers to generate training data. With this data, we can learn a function that generalizes beyond the capabilities of the original algorithm or that can be finetuned to a specific problem instance. For example, self-play in Go (using the algorithm as part of the solution) can be used to create new samples to learn from (Silver et al., 2016), training data for learning segmentation can be generated by simple algorithms such as background subtraction (Zeng et al., 2016), and reinforcement learning problems can be solved using training samples generated via trajectory optimization (Levine & Koltun, 2013).
The third approach—the one that we are focusing on in this paper—uses the same algorithms in the learning process and in the solution. The main idea is to optimize the models for the algorithms that use them rather than learning them in isolation. To achieve this, the algorithms need to be differentiable, such that we can compute how changes in the model affect the output of the algorithm, which allows to train the models end-to-end. This idea has been applied to different algorithms, e.g. in the form of neural Turing machines (Graves et al., 2014) and neural programmer-interpreters (Reed & de Freitas, 2015). In the context of robotics, Tamar et al. (2016) have presented a differentiable planning algorithm based on value iteration. And, most directly related to our work, Haarnoja et al. (2016) have applied this idea to Kalman filters, showing that measurement models based on visual input can be learned end-to-end as part of the filter. Our work differs from this by representing the belief with a histogram rather than a Gaussian, which allows to track multiple hypotheses—a neces-
sity for many robotic tasks. Furthermore, we focus on tasks where the robot has information about its actions and learn both the measurement model and the motion model jointly. Our paper extends an earlier workshop submission (Jonschkowski & Brock, 2016).
4 PRELIMINARIES: HISTOGRAM FILTERS AND OTHER BAYES FILTERS
A Bayes filter (Thrun et al., 2005) is an algorithm to recursively estimate a probability distribution over a latent state s (e.g. robot pose) conditioned on the history of observations o (e.g. camera images) and actions a (e.g. velocity commands). This posterior over states is also called belief, Bel(st) = p(st|a1:t−1, o1:t). A histogram filter is a type of Bayes filter that represents the belief as a histogram; a discretization of the state space with one probability value per discrete state s. One of the key assumptions in Bayes filters is the Markov property of states, from which follows that the current belief Bel(st) summarizes all information of the entire history of observations and actions that is relevant for predicting the future.
Other key assumptions determine how the belief is recursively updated using two alternating steps: the prediction step based on the last action at−1 and the measurement update step based on the current measurement ot. Note that these two sources of information are separated, which results from the assumption of conditional independence of observation and action given the state.
The prediction step assumes actions to change the state according to the known motion model p(st | st−1, at−1). After performing an action at−1, the new belief for a given state st is computed by summing over all possible ways through which state st could have come about,
Bel(st) = ∑ st−1 p(st | st−1, at−1)Bel(st−1). (1)
The measurement update step assumes observations to only depend on the current state as defined by a known measurement model p(ot | st). After receiving an observation ot, the belief for every state st is updated using Bayes’ rule,
Bel(st) ∝ p(ot | st)Bel(st). (2)
If motion model and measurement model are unknown, we want the robot to learn these models from data. Apart from the assumptions already mentioned, learning explicit models allow us to restrict their hypothesis space according to assumptions (e.g. linear motion). Our goal is to train these models end-to-end such that we find the models that optimize state estimation performance, while preserving the useful assumptions of Bayes filters. Towards this end, we formulate the belief, the prediction, the measurement update, and the corresponding models in the deep learning framework.
5 END-TO-END LEARNABLE HISTOGRAM FILTERS
An end-to-end learnable histogram filter (E2E-HF) is a differentiable implementation of a histogram filter that allows both motion model and measurement model to be learned end-to-end by backpropagation through time (Werbos, 1990). Alternatively, we can view the E2E-HF as a new recurrent neural network architecture that implements the structure of a histogram filter (see Fig. 3).
5.1 END-TO-END LEARNING AND DIFFERENTIABILITY
If we want to use the structure of a histogram filter as a prior and fit the measurement model and the motion model to data, we can essentially do one of two things: a) learn the models in isolation to optimize a quality measure of the model or b) learn the models end-to-end, i.e. train the models as part of the entire system and optimize the end-to-end performance.
In either way, we might want to optimize the models using gradient descent, for example by computing the gradient of the learning objective with respect to the model parameters using backpropagation (repeated application of the chain rule). Therefore, the motion model and the measurement model need to be differentiable regardless of whether we choose option a) or option b). For b) endto-end learning, we need to backpropagate the gradient through the histogram filter algorithm (not to
change the algorithm but to compute how to change the models to improve the algorithm’s output). Therefore, in addition to the models, the algorithm itself needs to be differentiable.
The remainder of this section describes how histogram filters can be implemented in a differentiable way and how they can be learned in isolation or end-to-end. To comply with the deep learning framework, we will define the E2E-HF using vector and matrix operations. We will also introduce additional priors for computational or data efficiency. For the sake of readability, we assume a one-dimensional state space here. All formulas can easily be adapted to higher dimensions.
5.2 BELIEF
The histogram over states is implemented as a vector b of probabilities with one entry per bin,
bt = [Bel(St = 1), Bel(St = 2), . . . , Bel(St = |S|)].
We can also think of the belief as a neural network layer where the activation of each unit represents the value of a histogram bin. The belief bt constitutes the output of the histogram filter at the current step t and an input at the next step t+ 1—together with an action at and an observation ot+1 (see Fig. 3).
5.3 PREDICTION (MOTION UPDATE)
The most direct implementation of the prediction step (which we replace shortly) defines a learnable function f for the motion model, f : st, st−1, at−1 7→ p(st | st−1, at−1), and employs f in the prediction step (Eq. 1). The equation can be vectorized for computational efficiency by defining a |S| × |S| matrix F with Fi,j(a) = f(i, j, a), such that bt = F (at−1)bt−1.
However, this approach is computationally expensive because it requires |S|2 evaluations of f for a single prediction step. We can make this computation more efficient, if we assume robot motion to be local and consistent across the state space, i.e.
p(st | st−1, at−1) = p(∆st | at−1), ∀t|∆st| ≤ k,
where ∆st = st−st−1 and k is the maximum state change. Accordingly, we define a new learnable function for the motion model, g : ∆st, at−1 7→ p(∆st | at−1) and use g instead of f . For vectorization, we define a (2k+1)-dimensional vector g(a), whose elements gi(a) = g(i−k−1, a) represent the probabilities of all positive and negative state changes up to k. We can now reformulate the prediction step (Eq. 1) as a convolution (∗),
bt = bt−1 ∗ g(at−1),
where the belief bt−1 is convolved with the motion kernel g(at−1) for action at−1 (see Fig. 3).
5.3.1 MOTION MODEL
The learnable motion model g can be implemented as any feedforward network that maps ∆s and a to a probability. The prior that g(a) represents a probability mass function, i.e. that the elements of g(a) should be positive and sum to one, can be enforced using the softmax nonlinearity on the vector of unnormalized network outputs g̃(a), such that gi(a) = e
g̃i(a)∑ j e g̃j(a) .
Another useful prior for g is smoothness with respect to ∆s and a, i.e. that similar combinations of ∆s and a lead to similar probabilities. This smoothness is the reason why (for standard feedforward networks), we should use ∆s as an input rather than as index for different output dimensions. With additional knowledge about robot motion, we can replace smoothnes by a stronger prior. For the
experiments in this paper, we assumed linear motion with zero mean Gaussian noise, and therefore defined the motion model with only two learnable parameters α and σ,
g̃(∆s, a) = e− (∆s−αa)2 σ2 ,
and the obligatory normalization, g(∆s, a) = g̃(∆s,a)∑k j=−k g̃(j,a) .
5.4 MEASUREMENT UPDATE
Analogously to the motion model in the prediction step, we define a learnable function h that represents the measurement model for the measurement update, h : st, ot 7→ p(ot | st). To vectorize the update equation (Eq. 2), we define a vector h(o) with elements hi(o) = h(i, o), such that the measurement update corresponds to element-wise multiplication ( ) with this vector,
b̃t = h(o) bt,
followed by a normalization, bt = b̃t∑ j b̃t,j (see Fig. 3).
5.4.1 MEASUREMENT MODEL
The learnable function h that represents the measurement model can again be implemented by any feedforward network. Since h corresponds to p(ot | st)—a probability distribution over observations—it needs to be normalized across observations, not across states. To realize the correct normalization, we need to compute the unnormalized likelihood vector h̃(o) for every observation o and compute the softmax over the corresponding scalars in different vectors rather than over the scalars of the same vector: h(o) = e
h̃(o)∑ o′ e
h̃(o′) . If the observations are continuous instead of discrete, this summation must be approximated using sampled observations.
For the experiments in this paper, we represented h by a network with three hidden layers of 32 rectified linear units (Nair & Hinton, 2010), followed by a linear function and a normalization as described above. Using the observation and state as input rather than output dimensions again incorporates the smoothness prior on these quantities.
5.5 LEARNING
We can learn the motion model g and the measurement model h using different learning objectives based on different sequences of data. We will first look at a number of supervised learning objectives that require o1:T , a1:T , s1:T , and sometimes x1:T —the underlying continuous state. Then, we will describe unsupervised learning that only needs o1:T and a1:T .
5.5.1 SUPERVISED LEARNING IN ISOLATION
Both models can be learned in isolation by optimizing an objective function, e.g. the cross-entropy between experienced state change / observation and the corresponding outputs of g and h,
Lg = − 1
T − 1 T∑ t=2 e(∆st−k−1) log(g(at−1)),
Lh = − 1
T T∑ t=1 e(ot) log(h(ot)),
where e(i) denotes a standard basis vector with all zeros except for a one at position i, that is the position that represents the experienced state change or observation.
5.5.2 SUPERVISED END-TO-END LEARNING
Due to our differentiable implementation, the models can also be learned end-to-end using backpropagation through time (Werbos, 1990), which we apply on several overlapping subsequences
of length C (in our experiments, C = 32). In the corresponding learning objectives, we compare the belief at the final time step of this subsequence with the true state. If we want to optimize the accuracy of the filter with respect to its discrete states, we can again use a cross-entropy loss,
Lacc. = − 1
T − C T∑ t=C+1 e(st) log(b (t−C:t) t ),
where b(t−C:t)t denotes the final belief at time step t when the histogram filter is applied on the subsequence that spans steps t− C to t. Alternatively, we might want to optimize other objectives, e.g. the mean square error with respect to the underlying continuous state,
Lmse = − 1
T − C T∑ t=C+1 (xt − xb(t−C:t)t )2,
where x denotes a vector of the continuous values to which the discrete states correspond, such that xb (t−C:t) t is the weighted average of these values according to the final belief in this subsequence.
5.5.3 UNSUPERVISED END-TO-END LEARNING
By exploiting the structure of the histogram filter algorithm and the differentiability, we can even train the models without any state labels by predicting future observations, but later use the models for state estimation. Similarly to supervised end-to-end learning, we apply the filter on different subsequences of length C, but then we follow this with D steps without performing the measurement update (in our experiments, D = 32). Instead, we use the measurement model to predict the observations. Pred(ot) = ∑ st p(ot | st)Bel(st) = h(ot)bt. To predict the probabilities for all observations, we define a matrix H with elements Hi,j = h(i, j) as defined in Section 5.4. Putting everything together, we get the following loss for unsupervised end-to-end learning:
Lunsup. = − 1
(T − C)D T∑ t=C+1 D∑ d=1 e(ot+d) log(H>b (t−C:t+d) t+d ).
6 EXPERIMENTS
We consider the problem of learning to estimate the robot’s state in unknown environments with partial observations. In this problem, we compare histogram filters for which the models are learned in isolation (HF), end-to-end learnable histogram filters (E2E-HFs), and two-layer long-short-term memory networks (LSTMs, Hochreiter & Schmidhuber, 1997). The models of the HFs are learned by optimizing the loss functions Lg and Lh presented in the previous section. For the E2E-HFs and LSTMs, we compare end-to-end learning using Lacc., Lmse, and Lunsup..
Our results show that 1) the algorithmic prior in HFs and E2E-HFs increases data efficiency for learning localization compared to generic LSTMs, 2) end-to-end learning improves the performance of E2E-HFs compared to HFs, and 3) E2E-HFs are able to learn state estimation without state labels.
6.1 PROBLEM: LEARNING RECURSIVE STATE ESTIMATION IN UNKNOWN ENVIRONMENTS
An important state estimation problem in partially observable environments is localization: a robot moves through an environment by performing actions and receives partial observations, such that it needs to filter this information over time to estimate its state, i.e. its position. In our experiments, the robot does not know the environment beforehand and thus has to learn state estimation from data.
We performed experiments in two localization tasks: a) a hallway localization task (Thrun et al., 2005) and b) a drone localization task (see Fig. 4). The tasks are similar in that they have continuous actions and binary observations (door/wall and purple/white tile), both of which are subject to 10% random error. The tasks differ in their dimensionality. In the hallway task, the robot only needs to estimate a one-dimensional state (its position along the hallway), which for all methods is discretized into 100 states. The drone localization task has a two-dimensional state, which is discretized into 50 bins per dimension resulting in 2500 bins in total. The challenge in both tasks is that the door/tile
locations, the scale of the actions, and the amount of random noise are unknown and need to be learned from data, i.e. a sequence of observations, actions, and—in the supervised setting—states produced by the robot moving randomly through the environment. More details about the tasks, the experimental setting, learning parameters, etc. can be found in Appendix A.
6.2 RESULTS: IMPROVED DATA-EFFICIENCY
Hallway task: We performed multiple experiments in the hallway localization task with different amounts of training data. The learning curves with respect to mean squared error for supervised learning show large differences in data efficiency (see solid lines in Fig. 5a): E2E-HFs require substantially less training samples than LSTMs to achieve good performance (2000 rather than > 8000). HFs are even more data-efficient but quickly stop improving with additional data.
Drone task: For the drone localization task, we performed an experiment using 4000 training steps (see Table 1). Our results show that this data is sufficient for the E2E-HF (but not for the LSTM) to achieve good performance. Our method only required a similar amount of data as for the 1D hallway task, even though the histogram size had increased from 100 to 2500 bins.
Discussion: The priors encoded in the E2E-HF improve data efficiency because any information contained in these priors does not need to be extracted from data. This leads to better generalization, e.g. the ability to robustly and accurately track multiple hypotheses (see Fig.6).
Note on computational limits: The size of the histogram is exponential in the number of state dimensions. A comparison between the 1D and the 2D task suggests that data might not be the bottleneck for applying the method to higher dimensional problems, since the data requirements were similar. However, the increased histogram size directly translates into longer training times, such that computation quickly becomes the bottleneck for scaling this method to higher-dimensional problems. Addressing this problem will require to change the belief representation, e.g. to particles or a mixture of Gaussians, which is an important direction for future work.
6.3 RESULTS: OPTIMIZATION OF END-TO-END PERFORMANCE
Hallway task: While HFs excel with very few data, E2E-HFs surpass them if more than 2000 training samples are available (see gray and yellow lines in Fig. 5a). For the mean squared error metric, the best method is the E2E-HF with a mean squared error objective (yellow line). However, if we care about a different metric, e.g. accuracy of estimating the discrete state, the methods rank differently (see Fig. 5b). The best method for the previous metric (yellow line) is outperformed by HFs (gray line) and even more so by E2E-HFs that are optimized for accuracy (teal line). For yet another metric, i.e. accuracy of predicting future observations, HFs outperform both other approaches but are equal to E2E-HFs optimized for predicting future observations (see Fig. 5c).
Drone task: The results of the drone localization task show the same pattern (see Table 1). The best method for every metric is the E2E-HF that optimizes this metric.
Discussion: E2E-HFs perform better than HFs because they optimize the models for the filtering process (with respect to the metric they were trained for) rather than optimizing model accuracy. This can be advantageous because “inaccurate” models can improve end-to-end performance (compare the HF model learned in isolation to the models learned end-to-end in Fig. 6a).
6.4 RESULTS: ENABLING UNSUPERVISED LEARNING
Hallway and drone tasks: In both tasks, unsupervised E2E-HFs were similar to HFs and better than all other methods for predicting future observations. Interestingly, they also had comparatively low mean squared error for state estimation even though they had never seen any state labels (see dashed green line in Fig. 5 and second line in Table 1). In fact, the qualitative results for both tasks show a remarkable similarity between the learned models and the estimated belief between HFs and unsupervised E2E-HFs (compare HF and E2E-HF (unsup.) in Fig. 6) and Fig. 7.
Discussion: E2E-HFs can learn state estimation purely based on observations and actions. By predicting future observations using the structure of the histogram filter algorithm, the method discovers a state representation that works well with this algorithm, which is surprisingly close to the “correct” models learned by HFs, although no state labels are used.
7 CONCLUSION
We proposed to tightly combine prior knowledge captured in algorithms with the ability to learn from data. We demonstrated the feasibility and the advantages of this idea in the context of state estimation in robotics. Algorithmic priors lead to data-efficient learning, as knowledge about the problem structure encoded in the algorithm is provided explicitly and does not have to be extracted from data. The ability to learn from data enables the use of algorithms when task-specifics are unknown. The tight combination of both improves performance as the models are optimized for use in the algorithm. Furthermore, the explicit algorithmic structure enables unsupervised learning. We view our results as a proof of concept and are convinced that the combination of algorithms and machine learning will help solve novel problems, while balancing data efficiency and generality.
ACKNOWLEDGMENTS
We gratefully acknowledge the funding provided by the Alexander von Humboldt foundation and the Federal Ministry of Education and Research (BMBF).
A ADDITIONAL EXPERIMENT DETAILS
A.1 HALLWAY LOCALIZATION TASK
The hallway has a length of 10 meters, where every full meter is either occupied by a door or by a wall. At the beginning of every experiment trial, 5 doors are randomly arranged in the 10 spots in the hallway. The binary observation of the robot senses whether the center of the robot is next to a door or next to a wall. With probability 0.1, the observation returns the wrong information, e.g. “wall” instead of “door” if the robot is next to a door.
The robot is represented as a single point. It moves with a velocity between -1 and 1 meter per time step and stops when it reaches either end of the hallway. The action information that the robot receives is the step that it performed as measured by odometry. This odometry measurement is corrupted with zero mean Gaussian noise with standard deviation of 10% of it’s actual movement. Additionally, the odometry is scaled by a number between 0.5 and 5.0, which is randomly sampled at the beginning of every trial, i.e. the robot does not know its exact embodiment. This makes the exact motion model unknown, such that the robot needs to learn it from data.
Both during training and during testing the robot moves randomly, i.e. it randomly accelerates by a value between -0.5 and 0.5 at each time step. Apart from this acceleration, its velocity is affected by 10% friction at each time step and is set to zero when the robot reaches either end of the hallway. For each trial, the training data consists of a single random walk of the robot of length between 500 steps and 8000 steps. The data for the unsupervised learning, includes only the sequence of noisy observations and actions. For supervised learning, it additionally includes the groundtruth continuous and discrete state, i.e. the position of the robot.
The test data consisted of 1000 short time sequences of the robot moving in the same fashion starting from a random position. For all performance metrics, the belief was tracked for 32 steps. For the metric that measured observation prediction accuracy, the task was to predict 32 future observations given a sequence of 32 actions based on the current belief.
A.2 DRONE LOCALIZATION TASK
The area for the drone localization task has a size of 5 times 5 meters, where every one meter tile is either purple or white. At the beginning of every experiment, the color of each tile is decided by a fair coin flip. Analogously to the hallway task, the binary observations inform the robot about the color of the tile which is directly underneath it. With probability of 0.1, this observation returns the wrong color.
The drone is represented as a single point in 2D space. It moves with velocities between -0.5 and 0.5 meter per time step and stops when it reaches the boundary of the area. The other aspects of its movement, the noisy odometry, and the movement generation for training and test data are analogous to the hallway localization task.
A.3 EXPERIMENTAL DETAILS
LSTM baseline: The LSTM baseline consists of two LSTM layers with 32 units per layer, followed by a softmax layer.
Training procedure: All methods where trained via minibatch stochastic gradient descent with batch size 32 using Adam (Kingma & Ba, 2014) with learning rate 0.001. The training length was determined using early stopping with patience, where 20% of the training data was used for validation. After 100 epochs without an improvement on the validation data, the parameters that achieved highest validation performance were returned.
A.4 SOFTWARE
We used v-rep for simulation (E. Rohmer, 2013) and theano (Theano Development Team, 2016) with Lasagne (Dieleman et al., 2015) as deep learning framework for our implementation.

Summary: This paper presents a differentiable histogram filter for state estimation/tracking. The proposed histogram filter is a particular Bayesian filter that represents the discretized states using beliefs. The prediction step is parameterized by a locally linear and translation-invariant motion model while the measurement model is represented by a multi-layered neural network. The whole system is learned with both supervised and unsupervised objectives and experiments are carried out on two synthetic robot localization tasks (1D and 2D). The major claim of this paper is that the problem-specific model structure (Bayesian filter for state estimation) should improve pure deep learning approach in data-efficiency and generalization ability.
+This paper has nice arguments about the importance of prior knowledge to deep learning approach for specific tasks.
+An end-to-end histogram filter is derived for state estimation and unsupervised learning is possible in this model.
-This paper seems to have a hidden assumption that deep learning (RNN) is a natural choice for recursive state estimation and the rest of paper is built upon this assumption including LSTM baselines. However, this assumption itself may not be true, because Bayesian filter is a first-established approach for this classic problem, so it it more important to justify if deep learning is even necessary for solving the tasks presented. This requests pure Bayesian filter baselines in the experiments.
-The derived histogram filter seems to be particularly designed for discretized state space. It is not clear how well it can be generalized to continuous state space using the notation "x". More interestingly, the observation is discrete (binary) as well, which eventually makes it possible to derive a closed-form measurement update model. This setup might be too constrained. Generalizing to continuous observations is not a trivial task, not even to mention using images as observations like Haarnoja et al 2016. These design choices overall narrow down the scope of applicability.


Denoising auto-encoders (DAE)s were proposed as a simple yet powerful way to obtain representations in an unsupervised manner by learning a map that approximates the clean inputs from their corrupted versions. However, the original objective function proposed for DAEs does not guarantee that denoising happens only at the encoding stages. We argue that a better representation can be obtained if the encoder is forced to carry out most of the denoising effort. Here, we propose a simple modification to the DAE’s objective function that accomplishes the above goal.
1 INTRODUCTION
Auto-encoders (AE)s are unsupervised learning algorithms that capture structure in data by finding an representation of the inputs (encoding) from which they can be recovered at least approximately. By learning a transformation G (encoding) from the input x ∈ X to z = G(x) ∈ Z , the autoencoder tries to capture the structure of the input. To guarantee the transformation G preserves the information about x, a decoder G̃−1 (an approximate inverse map) is also learned such that a measure of fidelity E[D(X, X̃)] between the input and its reconstruction is optimized.
Learning non-linear encoding and decoding mappings has proved to be a non-trivial problem. For example, there is no consensus on what the dimensionality of the encoding space should be. On one side of the spectrum, AE architectures such as bottle-neck networks, map input data x ∈ X ⊆ Rd to a lower dimensional space Z ⊆ Rp, and then map it back to input space X through dimensionality expansion. The intuition behind bottleneck networks is that the lower dimensional space forces the encoder to capture meaningful relations between the variables in the input space. While this is clear when one is restricted to linear mappings, the problem becomes less well-understood if nonlinear mappings are allowed. The idea of using a low dimensional latent space has been recently employed in the context of modern generative models such as variational auto-encoders (VAE)s Kingma & Welling (2013) and generative adversarial networks (GAN)s Goodfellow et al. (2014).
On the other hand, AE architectures can contain over-complete representations where inputs are mapped to a high dimensional encoding spaces; much higher than the dimensionality of the input space (dim(X ) < dim(Z)). Over-complete representations have proved very useful in supervised learning. However, effective learning of this high dimensional mappings relies on specialized mechanisms that attempt to avoid trivial solutions or on the large amount of constraint imposed by the supervised task. Approaches such as sparse encoding, which can be applied to over-complete scenarios, use the concept of effective dimensionality. A sparsity constraint induces an active set of variables with an expected L0 norm smaller than the input space dimensionality. Many sparse coding procedures require solving an optimization problem at inference time since the encoding mapping is not explicit. However, it has been shown that efficient inference can be made possible by training a nonlinear feed-forward network to mimic the output of a sparse encoding algorithm Ranzato et al. (2006) and more recently with the proposal of techniques that impose very strong sparsity constraints such as winner take all auto-encoders Makhzani & Frey (2015).
An important set of techniques that do not necessarily fall into either extreme in terms of dimensionality are based on the concept of robustness of the representation. A representation is said to be
∗
robust if is able to retain information about the input even if the input or the representation undergo perturbations. For example, contractive auto-encoders (CAE)s Rifai et al. (2011a;b) penalize the sensitivity of the encoding map G(x) to perturbations of the input x by minimizing the expected Frobenius norm of the Jacobian E[JG(X)] of G while maximizing fidelity of the reconstruction. Within this category, learning representations by local denoising, or better known as de-noising auto-encoders, provide a very general way to carry out unsupervised learning. De-noising autoencoders Vincent et al. (2008) were conceived as a very intuitive way to capture robustness in the encoding-decoding mapping by simply minimizing the error between the output reconstruction of the auto-encoder when a corrupted version of the input is feed to the map and the uncorrupted input. However, as already pointed out in Rifai et al. (2011a), the DAE objective minimizes the reconstruction error, therefore DAEs lack of explicit robustness in the encoding phase. A workaround to the above problem can make DAEs an excellent choice for unsupervised learning. DAEs can be a very appealing alternative due to their simplicity for training and elegant interpretation. Due to their simplicity in the training criterion, DAEs have the potential to adapt to different kinds of architectures and to scale with the application. We argue that the current limitation of current DAEs can be overcome by making a very simple modification to the training objective leading to learning better representations in terms of robustness. The modified objective for DAE and is justification are the main contributions of our work.
2 DENOISING AUTO-ENCODERS
Let X be the random variable representing the input, and X̂ be a corrupted version obtained by an
an stochastic operator qC(X̂ |X). The goal of the DAE is to learn a deterministic encoder-decoder composition f(·) = G̃−1(G(·)) such that, for Y = f(X̂) the quantity E[D(Y,X)] is minimized. The corruption process leads to learning non-trivial transformations of the input in particular for the over-complete cases where identity mapping can be learned if there is no appropriate capacity control mechanism in place. Let G and G̃−1 be parametrized by θ = {θenc, θdec}. Parameter learning of the DAE can be formulated as the following optimization problem:
minimize θ∈Θ
1
N
N ∑
i=1
E X̂i [D(fθ(X̂i),xi)], (1)
where X̂i ∼ qC(X̂|X = xi). As we already mentioned, a solution to (1) does not impose explicit restrictions to the encoder. In this approach, only the dimensionality and the range of the encoder can be controlled by modifying its architecture. Therefore, there exist the possibility that the encoder could be learning a feature map, for which some dimensions still carry the effects of perturbing the original input. In many cases, the actual denoising could be taking place in the decoding phase. Below, we present a simple way to overcome the above limitation based on a modification to the DAE objective (1).
2.1 MODIFIED DENOISING OBJECTIVE
In order to explicitly enforce robustness in the encoding phase, we can measure the amount of distortion in the encoder by comparing the resulting feature values for the original uncorrupted input against the feature values obtained from encoding its corrupted counterpart. Likewise D in (1), the difference between can be measured by a functionDenc(·, ·), for instance cross-entropy can be used for encoder that map input values to [0, 1]. This leads to the followingmodified optimization problem:
minimize θ∈Θ
1
N
N ∑
i=1
E X̂i
[D(fθ(X̂i),xi)] + 1
λ E X̂i [Denc(G(X̂i), G(xi))], (2)
where λ is a tradeoff parameter. The original DAE objective can be simple obtained by setting λ to infinity.
2.2 ANALYSIS OF THE MODIFIED DAE
The modified objective (2) appears as an intuitive way to explicitly enforce robustness in the encoding phase. Unlike the plain objective where Y = f(X), the stochastic operator qC in conjunction
with the encoder function G(.) lead to a random map from an instance x of X to the random variable Z|X = x. In this case, the minimization of the reconstruction error leads to maximization of a lower bound on the mutual information between Z and X1. However, the information maximization perspective does not provide insights about how the noise acts as capacity control on the encoder. Since the only constraint on the entropyH(Z) is the encoder architecture, mutual information I(X ;Z) = H(X)−H(X |Z) = H(Z)−H(Z|X) does not tell us much about the conditional entropyH(Z|X), which is directly related to the encoder. Bear in mind that the corruption process is sill under consideration, thus the map fromX toZ is stochastic. The conditional entropyH(Z|X) can be upper bounded byH(Z|G(X))2:
H(Z|X) ≤ H(Z|G(X)) = E X̂,G(X)[− log(p(G(X̂)|G(X)))] (3)
∝ E X̂,X [Denc(G(X̂)|G(X))], (4)
which corresponds to the population version of the second term of the objective (2).
2.3 ARCHITECTURAL CONSIDERATIONS AND THE ENCODING LOSS
The above formulation assumes that the encoding loss term matches the conditional log likelihood fun-ction. Thus, the encoding loss should also match the encoding architecture. For instance, if we use sigmoidal units, we can assume a multivariate Bernoulli distribution of the code and use cross-entropy as the encoding loss. Architectures inducing a continuous code space may require further considerations for the choice of the loss or architectural constraints. For instance, a linear encoder with the euclidean distance as the encoding loss may lead to a trivial minimization of the modified objective. By reducing the norm of the encoder weights, the distance between the codes of the clean and corrupted samples can be made arbitrarily small, only being compensated by scaling of the decoder weights. In this case, a simple way to circumvent this issue is by using tied weights in the decode. In linear case, the role of tied weights is easy to understand. However, a similar interpretation may not extend to non-linear cases. To avoid trivial shrinkage of the encoding space, and without resorting to tied weights, we can employ a normalized distance function such as squared euclidean distance divided by the total variance of the encoded data or by adding a batch normalization layer to the top layer of encoder.
2.4 RELATION TO OTHER APPROACHES
In a similar spirit to the proposed modified denoising auto-encoder objective, contractive autoencoders achieve robustness in the encoder by explicitly computing a regularization term based on the L2 of the Jacobian of the encoder. In the regime of small Gaussian perturbations the modified objective can be approximated by the Jacobian of the encoding transformation. Nevertheless, the setting of the modified denoising auto-encoder is more general in terms of distances and forms of input corruption, which can lead to different properties to the ones obtained by manipulating the Jacobian of the transformation. Another interpretation of the learning algorithm is as a Siamese network of the encoders, where the goal is to map the clean and corrupted input to the “same” code.
3 EXPERIMENTS
This section describe some of the experiments we have carried out with the modified DAE. We qualitatively illustrate how in over-complete representations, the DAE benefits from adding the penalty to the encoding space.
3.1 SYNTHETIC DATA
Gaussian distributed data: The first example corresponds to a set of data point drawn from a bivariate Gaussian distribution with zero mean and covariance matrix
ΣX =
(
1 0.95 0.95 1
)
. (5)
1Note that the information maximization view applies to conventional auto-encoder, as well.The main difference being the stochastic map
2This follows from the data processing inequality.
We use a linear encoding, that is g(x) = x, that is also over-complete since the encoder project the 2-dimensional data points to 10 different directions. The conventional auto encoder would over fit being able to achieve zero reconstruction error, but it won’t be able to implicitly retain what is thought to be the structure in the data. We also compare this output to the outputs of two nonlinear auto-encoders, one uses the logsig units g(x) = 1/(1 + exp(−x)), and the other a rectified linear units (ReLU) g(x) = max{0, x}. Figure 1 shows the outputs of the three auto-encoders on the
Gaussian distributed data. It can be seen that the outputs approximately align with what corresponds roughly to the first principal component of the data. Notice, that no bottleneck neither shrinkage of parameters was explicitly defined. The parameters of our cost function are λ = 1 for the distortion trade-off, and σ = 0.5 for the noise level. Mixture of Gaussians: The second example, employs a mixture of three Gaussian distributions to show the output of modified DAE in a nonlinear scenario, where an over-complete representation
followed by a nonlinearity can be advantageous. The means of and covariances of the mixture components are
µ1 =
(
2 −2
)
, µ2 =
(
−2 −2
)
, µ1 =
(
6 −2
)
; and
Σ1 =
(
1 −0.95 −0.95 1
)
, Σ2 = Σ3 =
(
1 0.95 0.95 1
)
,
(6)
respectively, and the mixing weights are p1 = 0.5, and p2 = p3 = 0.25. Figure 2 shows the outputs of the four auto-encoders on the mixture of Gaussian distributions. The auto-encoders employ: linear, rectified linear, sigmoidal, and saturated linear units. It can be seen that the outputs approximately align with what can be though as the principal curves of the data. Again, we want to stress that no bottleneck neither shrinkage was explicitly defined. In this case each of the auto-encoder has 20 units for encoding, which would easily over fit the data in the absence of any regularization or add-hoc constraints such as tied weights. The parameters of our cost function are λ = 1 for the distortion level σ = 0.5 . The linear units seem to fit the data, but as we previously mentioned they favor the principal components. Increasing the noise would collapse the reconstructed points into a line. This is not necessarily the case when nonlinear units are considered. Finally, in Figure 3 we show the resulting energy ‖x − x̂‖2 landscapes for the over-complete autoencoder with rectified linear units after being trained with: original DAE objective and the proposed modified DAE objective. The modified objective makes the AE carve well-defined ravines in the energy landscape at the points were majority of the data lies.
3.2 MNIST
Here, we observe the influence of the extra term in the objective function of the DAE. We train a single hidden layer DAE with logsig activation and cross entropy loss on both encoding and decoding layers. The number of fully connected hidden units is 2048. Input images are corrupted with zero-mask noise with 30% corruption level. Once the DAE is trained for 100 epochs, we use the encoding layer as a feature extractor for a multi class logistic regression classifier. Figure 4(a), displays the average test error over 30 runs of the DAE training using random initial conditions for different values of λ. We remind the reader that there is no fine tuning of the weights of the encoder layer, after unsupervised pre-training only the weights of the logistic regression layer are trained using label information. Moreover, since the main goal of here is to observe the influence of the encoding cost term, we focus on comparisons at the single hidden layer level rather than trying to increase performance by stacking multiple layers.
4 CONCLUSIONS
We presented an algorithm for learning auto-encoders based on an modified objective for Denoising that explicitly enforces the denoising to be carried out during the encoding phase. Moreover,
2 3 4 5 1.7
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
log10 (λ)
T es t er ro r %
Original DA, λ = ∞ Modified DA
(a) Test errors for different values of λ
we described how the modified objective can be understood as minimizing an upper bound on the conditional entropy of the code given the inputs assuming the corruption process to be part of the encoding. By simply minimizing the distance between the encoding of the uncorrupted inputs and the encoding of their corrupted counterparts, we show that robustness of the auto encoder can be guaranteed at the encoding phase. Experiments using over-complete bases showed that the modified DAE was able to learn useful encoding mapping (representation) and can also learn a regularized input-output in an implicit manner.

The paper proposes a modified DAE objective where it is the mapped representation of the corrupted input that is pushed closer to the representation of the uncorrupted input. This thus borrows from both denoising (DAE) for the stochasticity and from the contractive (CAE) auto-encoders objectives (which the paper doesn’t compare to) for the representational closeness, and as such appears rather incremental. In common with the CAE, a collapse of the representation can only be avoided by additional external constraints, such as tied weights, batch normalization or other normalization heuristics. While I appreciates that the authors added a paragraph discussing this point and the usual remediations after I had raised it in an earlier question, I think it would deserve a proper formal treatment. Note that such external constraints do not seem to arise from the information-theoretic formalism as articulated by the authors. This casts doubt regarding the validity or completeness of the proposed formal motivation as currently exposed.  What the extra regularization does from an information-theoretic perspective remains unclearly articulated (e.g. interpretation of lambda strength?).
On the experimental front, empirical support for the approach is very weak: few experiments on synthetic and small scale data. The modified DAE's test errors on MNIST are larger than those of Original DAE all the time expect for one precise setting of lambda, and then the original DAE performance is still within the displayed error-bar of the modified DAE. So, it is unclear whether the improvement is actually statistically significant.