We propose a new approach to the problem of optimizing autoencoders for lossy image compression. New media formats, changing hardware technology, as well as diverse requirements and content types create a need for compression algorithms which are more flexible than existing codecs. Autoencoders have the potential to address this need, but are difficult to optimize directly due to the inherent non-differentiabilty of the compression loss. We here show that minimal changes to the loss are sufficient to train deep autoencoders competitive with JPEG 2000 and outperforming recently proposed approaches based on RNNs. Our network is furthermore computationally efficient thanks to a sub-pixel architecture, which makes it suitable for high-resolution images. This is in contrast to previous work on autoencoders for compression using coarser approximations, shallower architectures, computationally expensive methods, or focusing on small images.
1 INTRODUCTION
Advances in training of neural networks have helped to improve performance in a number of domains, but neural networks have yet to surpass existing codecs in lossy image compression. Promising first results have recently been achieved using autoencoders (Ballé et al., 2016; Toderici et al., 2016b) – in particular on small images (Toderici et al., 2016a; Gregor et al., 2016; van den Oord et al., 2016b) – and neural networks are already achieving state-of-the-art results in lossless image compression (Theis & Bethge, 2015; van den Oord et al., 2016a).
Autoencoders have the potential to address an increasing need for flexible lossy compression algorithms. Depending on the situation, encoders and decoders of different computational complexity are required. When sending data from a server to a mobile device, it may be desirable to pair a powerful encoder with a less complex decoder, but the requirements are reversed when sending data in the other direction. The amount of computational power and bandwidth available also changes over time as new technologies become available. For the purpose of archiving, encoding and decoding times matter less than for streaming applications. Finally, existing compression algorithms may be far from optimal for new media formats such as lightfield images, 360 video or VR content. While the development of a new codec can take years, a more general compression framework based on neural networks may be able to adapt much quicker to these changing tasks and environments.
Unfortunately, lossy compression is an inherently non-differentiable problem. In particular, quantization is an integral part of the compression pipeline but is not differentiable. This makes it difficult to train neural networks for this task. Existing transformations have typically been manually chosen (e.g., the DCT transformation used in JPEG) or have been optimized for a task different from lossy compression (e.g. Testa & Rossi, 2016, used denoising autoencoders for compression). In contrast to most previous work, but in line with Ballé et al. (2016), we here aim at directly optimizing the rate-distortion tradeoff produced by an autoencoder. We propose a simple but effective approach for dealing with the non-differentiability of rounding-based quantization, and for approximating the non-differentiable cost of coding the generated coefficients.
Using this approach, we achieve performance similar to or better than JPEG 2000 when evaluated for perceptual quality. Unlike JPEG 2000, however, our framework can be optimized for specific content (e.g., thumbnails or non-natural images), arbitrary metrics, and is readily generalizable to other
forms of media. Notably, we achieve this performance using efficient neural network architectures which would allow near real-time decoding of large images even on low-powered consumer devices.
2 COMPRESSIVE AUTOENCODERS
We define a compressive autoencoder (CAE) to have three components: an encoder f , a decoder g, and a probabilistic model Q,
f : RN → RM , g : RM → RN , Q : ZM → [0, 1]. (1)
The discrete probability distribution defined by Q is used to assign a number of bits to representations based on their frequencies, that is, for entropy coding. All three components may have parameters and our goal is to optimize the tradeoff between using a small number of bits and having small distortion,
− log2Q ([f(x)])︸ ︷︷ ︸ Number of bits +β · d (x, g([f(x)]))︸ ︷︷ ︸ Distortion . (2)
Here, β controls the tradeoff, square brackets indicate quantization through rounding to the nearest integer, and d measures the distortion introduced by coding and decoding. The quantized output of the encoder is the code used to represent an image and is stored losslessly. The main source of information loss is the quantization (Appendix A.3). Additional information may be discarded by the encoder, and the decoder may not perfectly decode the available information, increasing distortion.
Unfortunately we cannot optimize Equation 2 directly using gradient-based techniques, as Q and [·] are non-differentiable. The following two sections propose a solution to deal with this problem.
2.1 QUANTIZATION AND DIFFERENTIABLE ALTERNATIVES
The derivative of the rounding function is zero everywhere except at integers, where it is undefined. We propose to replace its derivative in the backward pass of backpropagation (Rumelhart et al., 1986) with the derivative of a smooth approximation, r, that is, effectively defining the derivative to be
d
dy [y] :=
d
dy r(y). (3)
Importantly, we do not fully replace the rounding function with a smooth approximation but only its derivative, which means that quantization is still performed as usual in the forward pass. If we replaced rounding with a smooth approximation completely, the decoder might learn to invert the
smooth approximation, thereby removing the information bottle neck that forces the network to compress information.
Empirically, we found the identity, r(y) = y, to work as well as more sophisticated choices. This makes this operation easy to implement, as we simply have to pass gradients without modification from the decoder to the encoder.
Note that the gradient with respect to the decoder’s parameters can be computed without resorting to approximations, assuming d is differentiable. In contrast to related approaches, our approach has the advantage that it does not change the gradients of the decoder, since the forward pass is kept the same.
In the following, we discuss alternative approaches proposed by other authors. Motivated by theoretical links to dithering, Ballé et al. (2016) proposed to replace quantization by additive uniform noise,
[f(x)] ≈ f(x) + u. (4)
Toderici et al. (2016a), on the other hand, used a stochastic form of binarization (Williams, 1992). Generalizing this idea to integers, we define the following stochastic rounding operation:
{y} ≈ byc+ ε, ε ∈ {0, 1}, P (ε = 1) = y − byc, (5)
where b·c is the floor operator. In the backward pass, the derivative is replaced with the derivative of the expectation,
d dy {y} := d dy E [{y}] = d dy y = 1. (6)
Figure 1 shows the effect of using these two alternatives as part of JPEG, whose encoder and decoder are based on a block-wise DCT transformation (Pennebaker & Mitchell, 1993). Note that the output is visibly different from the output produced with regular quantization by rounding and that the error signal sent to the autoencoder depends on these images. Whereas in Fig. 1B the error signal received by the decoder would be to remove blocking artefacts, the signal in Fig. 1D will be to remove high-frequency noise. We expect this difference to be less of a problem with simple metrics such as mean-squared error and to have a bigger impact when using more perceptually meaningful measures of distortion.
An alternative would be to use the latter approximations only for the gradient of the encoder but not for the gradients of the decoder. While this is possible, it comes at the cost of increased computational and implementational complexity, since we would have to perform the forward and backward pass through the decoder twice: once using rounding, once using the approximation. With our approach the gradient of the decoder is correct even for a single forward and backward pass.
2.2 ENTROPY RATE ESTIMATION
SinceQ is a discrete function, we cannot differentiate it with respect to its argument, which prevents us from computing a gradient for the encoder. To solve this problem, we use a continuous, differentiable approximation. We upper-bound the non-differentiable number of bits by first expressing the model’s distribution Q in terms of a probability density q,
Q(z) = ∫ [−.5,.5[M q(z+ u) du. (7)
An upper bound is given by: − log2Q (z) = − log2 ∫ [−.5,.5[M q(z+ u) du ≤ ∫ [−.5,.5[M − log2 q(z+ u) du, (8)
where the second step follows from Jensen’s inequality (see also Theis et al., 2016). An unbiased estimate of the upper bound is obtained by sampling u from the unit cube [−.5, .5[M . If we use a differentiable density, this estimate will be differentiable in z and therefore can be used to train the encoder.
2.3 VARIABLE BIT RATES
In practice we often want fine-gained control over the number of bits used. One way to achieve this is to train an autoencoder for different rate-distortion tradeoffs. But this would require us to train and store a potentially large number of models. To reduce these costs, we finetune a pre-trained autoencoder for different rates by introducing scale parameters1 λ ∈ RM ,
− log2 q ([f(x) ◦ λ] + u) + β · d (x, g([f(x) ◦ λ] /λ)) . (9)
Here, ◦ indicates point-wise multiplication and division is also performed point-wise. To reduce the number of trainable scales, they may furthermore be shared across dimensions. Where f and g are convolutional, for example, we share scale parameters across spatial dimensions but not across channels.
An example of learned scale parameters is shown in Figure 3A. For more fine-grained control over bit rates, the optimized scales can be interpolated.
2.4 RELATED WORK
Perhaps most closely related to our work is the work of Ballé et al. (2016). The main differences lie in the way we deal with quantization (see Section 2.1) and entropy rate estimation. The transformations used by Ballé et al. (2016) consist of a single linear layer combined with a form of contrast gain control, while our framework relies on more standard deep convolutional neural networks.
Toderici et al. (2016a) proposed to use recurrent neural networks (RNNs) for compression. Instead of entropy coding as in our work, the network tries to minimize the distortion for a given number of bits. The image is encoded in an iterative manner, and decoding is performed in each step to be able to take into account residuals at the next iteration. An advantage of this design is that it allows for progressive coding of images. A disadvantage is that compression is much more time consuming than in our approach, as we use efficient convolutional neural networks and do not necessarily require decoding at the encoding stage.
Gregor et al. (2016) explored using variational autoencoders with recurrent encoders and decoders for compression of small images. This type of autoencoder is trained to maximize the lower bound of a log-likelihood, or equivalently to minimize
−Ep(y|x) [ log
q(y)q(x | y) p(y | x)
] , (10)
where p(y | x) plays the role of the encoder, and q(x | y) plays the role of the decoder. While Gregor et al. (2016) used a Gaussian distribution for the encoder, we can link their approach to the work of Ballé et al. (2016) by assuming it to be uniform, p(y | x) = f(x) + u. If we also assume a Gaussian likelihood with fixed variance, q(x | y) = N (x | g(y), σ2I), the objective function can be written
Eu [ − log q(f(x) + u) + 1
2σ2 ||x− g(f(x) + u)||2
] + C. (11)
Here, C is a constant which encompasses the negative entropy of the encoder and the normalization constant of the Gaussian likelihood. Note that this equation is identical to a rate-distortion trade-off with β = σ−2/2 and quantization replaced by additive uniform noise. However, not all distortions have an equivalent formulation as a variational autoencoder (Kingma & Welling, 2014). This only works if e−d(x,y) is normalizable in x and the normalization constant does not depend on y, or otherwise C will not be constant. An direct empirical comparison of our approach with variational autoencoders is provided in Appendix A.5.
Ollivier (2015) discusses variational autoencoders for lossless compression as well as connections to denoising autoencoders.
3 EXPERIMENTS
3.1 ENCODER, DECODER, AND ENTROPY MODEL
We use common convolutional neural networks (LeCun et al., 1998) for the encoder and the decoder of the compressive autoencoder. Our architecture was inspired by the work of Shi et al. (2016), who demonstrated that super-resolution can be achieved much more efficiently by operating in the lowresolution space, that is, by convolving images and then upsampling instead of upsampling first and then convolving an image.
The first two layers of the encoder perform preprocessing, namely mirror padding and a fixed pixelwise normalization. The mirror-padding was chosen such that the output of the encoder has the same spatial extent as an 8 times downsampled image. The normalization centers the distribution of each channel’s values and ensures it has approximately unit variance. Afterwards, the image is convolved and spatially downsampled while at the same time increasing the number of channels to 128. This is followed by three residual blocks (He et al., 2015), where each block consists of an additional two convolutional layers with 128 filters each. A final convolutional layer is applied and the coefficients downsampled again before quantization through rounding to the nearest integer.
The decoder mirrors the architecture of the encoder (Figure 9). Instead of mirror-padding and valid convolutions, we use zero-padded convolutions. Upsampling is achieved through convolution followed by a reorganization of the coefficients. This reorganization turns a tensor with many channels into a tensor of the same dimensionality but with fewer channels and larger spatial extent (for details, see Shi et al., 2016). A convolution and reorganization of coefficients together form a sub-pixel convolution layer. Following three residual blocks, two sub-pixel convolution layers upsample the image to the resolution of the input. Finally, after denormalization, the pixel values are clipped to
1To ensure positivity, we use a different parametrization and optimize log-scales rather than scales.
A B
the range of 0 to 255. Similar to how we deal with gradients of the rounding function, we redefine the gradient of the clipping function to be 1 outside the clipped range. This ensures that the training signal is non-zero even when the decoded pixels are outside this range (Appendix A.1).
To model the distribution of coefficients and estimate the bit rate, we use independent Gaussian scale mixtures (GSMs),
log2 q(z+ u) = ∑ i,j,k log2 ∑ s πksN (zkij + ukij ; 0, σ2ks), (12)
where i and j iterate over spatial positions, and k iterates over channels of the coefficients for a single image z. GSMs are well established as useful building blocks for modelling filter responses of natural images (e.g., Portilla et al., 2003). We used 6 scales in each GSM. Rather than using the more common parametrization above, we parametrized the GSM so that it can be easily used with gradient based methods, optimizing log-weights and log-precisions rather than weights and variances. We note that the leptokurtic nature of GSMs (Andrews & Mallows, 1974) means that the rate term encourages sparsity of coefficients.
All networks were implemented in Python using Theano (2016) and Lasagne (Dieleman et al., 2015). For entropy encoding of the quantized coefficients, we first created Laplace-smoothed histogram estimates of the coefficient distributions across a training set. The estimated probabilities were then used with a publicly available BSD licensed implementation of a range coder2.
3.2 INCREMENTAL TRAINING
All models were trained using Adam (Kingma & Ba, 2015) applied to batches of 32 images 128×128 pixels in size. We found it beneficial to optimize coefficients in an incremental manner (Figure 3B). This is done by introducing an additional binary mask m,
− log2 q ([f(x)] ◦m+ u) + β · d (x, g([f(x)] ◦m)) . (13) Initially, all but 2 entries of the mask are set to zero. Networks are trained until performance improvements reach below a threshold, and then another coefficient is enabled by setting an entry of the binary mask to 1. After all coefficients have been enabled, the learning rate is reduced from an initial value of 10−4 to 10−5. Training was performed for up to 106 updates but usually reached good performance much earlier.
After a model has been trained for a fixed rate-distortion trade-off (β), we introduce and fine-tune scale parameters (Equation 9) for other values of β while keeping all other parameters fixed. Here
2https://github.com/kazuho/rangecoder/
we used an initial learning rate of 10−3 and continuously decreased it by a factor of τκ/(τ + t)κ, where t is the current number of updates performed, κ = .8, and τ = 1000. Scales were optimized for 10,000 iterations. For even more fine-grained control over the bit rates, we interpolated between scales optimized for nearby rate-distortion tradeoffs.
3.3 NATURAL IMAGES
We trained compressive autoencoders on 434 high quality images licensed under creative commons and obtained from flickr.com. The images were downsampled to below 1536 × 1536 pixels and stored as lossless PNGs to avoid compression artefacts. From these images, we extracted 128 × 128 crops to train the network. Mean squared error was used as a measure of distortion during training. Hyperparameters affecting network architecture and training were evaluated on a small set of held-out Flickr images. For testing, we use the commonly used Kodak PhotoCD dataset of 24 uncompressed 768× 512 pixel images3. We compared our method to JPEG (Wallace, 1991), JPEG 2000 (Skodras et al., 2001), and the RNNbased method of (Toderici et al., 2016b)4. Bits for header information were not counted towards the bit rate of JPEG and JPEG 2000. Among the different variants of JPEG, we found that optimized JPEG with 4:2:0 chroma sub-sampling generally worked best (Appendix A.2).
While fine-tuning a single compressive autoencoder for a wide range of bit rates worked well, optimizing all parameters of a network for a particular rate distortion trade-off still worked better. We here chose the compromise of combining autoencoders trained for low, medium or high bit rates (see Appendix A.4 for details).
For each image and bit rate, we choose the autoencoder producing the smallest distortion. This increases the time needed to compress an image, since an image has to be encoded and decoded multiple times. However, decoding an image is still as fast, since it only requires choosing and running one decoder network. A more efficient but potentially less performant solution would be to always choose the same autoencoder for a given rate-distortion tradeoff. We added 1 byte to the coding cost to encode which autoencoder of an ensemble is used.
Rate-distortion curves averaged over all test images are shown in Figure 4. We evaluated the different methods in terms of PSNR, SSIM (Wang et al., 2004a), and multiscale SSIM (MS-SSIM; Wang et al., 2004b). We used the implementation of van der Walt et al. (2014) for SSIM and the implementation of Toderici et al. (2016b) for MS-SSIM. We find that in terms of PSNR, our method performs
3http://r0k.us/graphics/kodak/ 4We used the code which was made available on https://github.com/tensorflow/models/ tree/2390974a/compression. We note that at the time of this writing, this implementation does not include entropy coding as in the paper of Toderici et al. (2016b).
similar to JPEG 2000 although slightly worse at low and medium bit rates and slightly better at high bit rates. In terms of SSIM, our method outperforms all other tested methods. MS-SSIM produces very similar scores for all methods, except at very low bit rates. However, we also find these results to be highly image dependent. Results for individual images are provided as supplementary material5.
In Figure 5 we show crops of images compressed to low bit rates. In line with quantitative results, we find that JPEG 2000 reconstructions appear visually more similar to CAE reconstructions than those of other methods. However, artefacts produced by JPEG 2000 seem more noisy than CAE’s, which are smoother and sometimes appear Gábor-filter-like.
To quantify the subjective quality of compressed images, we ran a mean opinion score (MOS) test. While MOS tests have their limitations, they are a widely used standard for evaluating perceptual quality (Streijl et al., 2014). Our MOS test set included the 24 full-resolution uncompressed originals from the Kodak dataset, as well as the same images compressed using each of four algorithms at or near three different bit rates: 0.25, 0.372 and 0.5 bits per pixel. Only the low-bit-rate CAE was included in this test.
For each image, we chose the CAE setting which produced the highest bit rate but did not exceed the target bit rate. The average bit rates of CAE compressed images were 0.24479, 0.36446, and 0.48596, respectively. We then chose the smallest quality factor for JPEG and JPEG 2000 for which the bit rate exceeded that of the CAE. The average bit rates for JPEG were 0.25221, 0.37339 and 0.49534, for JPEG 2000 0.24631, 0.36748 and 0.49373. For some images the bit rate of the CAE at the lowest setting was still higher than the target bit rate. These images were excluded from the final results, leaving 15, 21, and 23 images, respectively.
The perceptual quality of the resulting 273 images was rated by n = 24 non-expert evaluators. One evaluator did not finish the experiment, so her data was discarded. The images were presented to each individual in a random order. The evaluators gave a discrete opinion score for each image from a scale between 1 (bad) to 5 (excellent). Before the rating began, subjects were presented an uncompressed calibration image of the same dimensions as the test images (but not from the Kodak dataset). They were then shown four versions of the calibration image using the worst quality setting of all four compression methods, and given the instruction “These are examples of compressed images. These are some of the worst quality examples.”
Figure 6 shows average MOS results for each algorithm at each bit rate. 95% confidence intervals were computed via bootstrapping. We found that CAE and JPEG 2000 achieved higher MOS than JPEG or the method of Toderici et al. (2016b) at all bit rates we tested. We also found that CAE significantly outperformed JPEG 2000 at 0.375 bpp (p < 0.05) and 0.5 bpp (p < 0.001).
5https://figshare.com/articles/supplementary_zip/4210152
4 DISCUSSION
We have introduced a simple but effective way of dealing with non-differentiability in training autoencoders for lossy compression. Together with an incremental training strategy, this enabled us to achieve better performance than JPEG 2000 in terms of SSIM and MOS scores. Notably, this performance was achieved using an efficient convolutional architecture, combined with simple roundingbased quantization and a simple entropy coding scheme. Existing codecs often benefit from hardware support, allowing them to run at low energy costs. However, hardware chips optimized for convolutional neural networks are likely to be widely available soon, given that these networks are now key to good performance in so many applications.
While other trained algorithms have been shown to provide similar results as JPEG 2000 (e.g. van den Oord & Schrauwen, 2014), to our knowledge this is the first time that an end-to-end trained architecture has been demonstrated to achieve this level of performance on high-resolution images. An end-to-end trained autoencoder has the advantage that it can be optimized for arbitrary metrics. Unfortunately, research on perceptually relevant metrics suitable for optimization is still in its infancy (e.g., Dosovitskiy & Brox, 2016; Ballé et al., 2016). While perceptual metrics exist which correlate well with human perception for certain types of distortions (e.g., Wang et al., 2004a; Laparra et al., 2016), developing a perceptual metric which can be optimized is a more challenging task, since this requires the metric to behave well for a much larger variety of distortions and image pairs.
In future work, we would like to explore the optimization of compressive autoencoders for different metrics. A promising direction was presented by Bruna et al. (2016), who achieved interesting superresolution results using metrics based on neural networks trained for image classification. Gatys et al. (2016) used similar representations to achieve a breakthrough in perceptually meaningful style transfer. An alternative to perceptual metrics may be to use generative adversarial networks (GANs; Goodfellow et al., 2014). Building on the work of Bruna et al. (2016) and Dosovitskiy & Brox (2016), Ledig et al. (2016) recently demonstrated impressive super-resolution results by combining GANs with feature-based metrics.
ACKNOWLEDGMENTS
We would like to thank Zehan Wang, Aly Tejani, Clément Farabet, and Luke Alonso for helpful feedback on the manuscript.
A APPENDIX
A.1 GRADIENT OF CLIPPING
We redefine the gradient of the clipping operation to be constant, d
dx̂ clip0,255(x̂) := 1. (14)
Consider how this affects the gradients of a squared loss, d
dx̂ (clip0,255(x̂)− x)2 = 2(clip0,255(x̂)− x)
d
dx̂ clip0,255(x̂). (15)
Assume that x̂ is larger than 255. Without redefinition of the derivative, the error signal will be 0 and not helpful. Without any clipping, the error signal will depend on the value of x̂, even though any value above 255 will have the same effect on the loss at test time. On the other hand, using clipping but a different signal in the backward pass is intuitive, as it yields an error signal which is proportional to the error that would also be incurred at test time.
A.2 DIFFERENT MODES OF JPEG
We compared optimized and non-optimized JPEG with (4:2:0) and without (4:4:4) chroma subsampling. Optimized JPEG computes a Huffman table specific to a given image, while unoptimized JPEG uses a predefined Huffman table. We did not count bits allocated to the header of the file format, but for optimized JPEG we counted the bits required to store the Huffman table. We found that on average, chroma-subsampled and optimized JPEG performed better on the Kodak dataset (Figure 7).
A.3 COMPRESSION VS DIMENSIONALITY REDUCTION
Since a single real number can carry as much information as a high-dimensional entity, dimensionality reduction alone does not amount to compression. However, if we constrain the architecture of the encoder, it may be forced to discard certain information. To better understand how much information is lost due to dimensionality reduction and how much information is lost due to quantization, Figure 8 shows reconstructions produced by a compressive autoencoder with and without quantization. The effect of dimensionality reduction is minimal compared to the effect of quantization.
A.4 ENSEMBLE
To bring the parameter controlling the rate-distortion trade-off into a more intuitive range, we rescaled the distortion term and expressed the objective as follows:
− α N ln q ([f(x) ◦ λ] + u) + 1− α 1000 ·M · ||x− g([f(x) ◦ λ] /λ)||2. (16)
Here, N is the number of coefficients produced by the encoder and M is the dimensionality of x (i.e., 3 times the number of pixels).
The high-bit-rate CAE was trained with α = 0.01 and 96 output channels, the medium-bit-rate CAE was trained with α = 0.05 and 96 output channels, and the low-bit-rate CAE was trained with α = 0.2 and 64 output channels.
A.5 COMPARISON WITH VAE
A.6 COMPLETE IMAGES
Below we show complete images corresponding to the crops in Figure 5. For each image, we show the original image (top left), reconstructions using CAE (top right), reconstructions using JPEG 2000 (bottom left) and reconstructions using the method of (Toderici et al., 2016b) (bottom right). The images are best viewed on a monitor screen.

The paper proposes a neural approach to learning an image compression-decompression scheme as an auto-encoder. While the idea is certainly interesting and well-motivated, in practice, it turns out to achieve effectively identical rates to JPEG-2000.
Now, as the authors argue, there is some value to the fact that this scheme was learned automatically rather than by expert design---which means it has benefits beyond the compression of natural images (e.g., it could be used to automatically learning a compression scheme for signals for which we don't have as much domain knowledge). However, I still believe that this makes the paper unsuitable for publication in its current form because of the following reasons---
1. Firstly, the fact that the learned encoder is competitive---and not clearly better---than JPEG 2000 means that the focus of the paper should more be about the aspects in which the encoder is similar to, and the aspects in which it differs, from JPEG 2000. Is it learning similar filters or completely different ones ? For what kinds of textures does it do better and for what kinds does it do worse (the paper could show the best and worst 10 patches at different bit-rates) ?
2. Secondly, I think it's crucial that the paper demonstrate that the benefits come from a better coding scheme (as opposed to just a better decoder), as suggested in my initial pre-review question. How would a decoder trained on JPEG-2000 codes (and perhaps also on encoded random projections) do worse or better ?
3. Finally, I think the fact that it does as well/worse than JPEG-2000 significantly diminishes the case for using a 'deep' auto-encoder. JPEG-2000 essentially uses a wavelet transform, which is a basis that past studies have shown could be recovered using a simple sparse dictionary algorithm like K-SVD. This is why I feel that the method needs to clearly outperform JPEG-2000, or show comparisons to (or atleast discuss) a well-crafted traditional/generative model-based baseline.


It has long been recognized that the invariance and equivariance properties of a representation are critically important for success in many vision tasks. In this paper we present Steerable Convolutional Neural Networks, an efficient and flexible class of equivariant convolutional networks. We show that steerable CNNs achieve state of the art results on the CIFAR image classification benchmark. The mathematical theory of steerable representations reveals a type system in which any steerable representation is a composition of elementary feature types, each one associated with a particular kind of symmetry. We show how the parameter cost of a steerable filter bank depends on the types of the input and output features, and show how to use this knowledge to construct CNNs that utilize parameters effectively.
It has long been recognized that the invariance and equivariance properties of a representation are critically important for success in many vision tasks. In this paper we present Steerable Convolutional Neural Networks, an efficient and flexible class of equivariant convolutional networks. We show that steerable CNNs achieve state of the art results on the CIFAR image classification benchmark. The mathematical theory of steerable representations reveals a type system in which any steerable representation is a composition of elementary feature types, each one associated with a particular kind of symmetry. We show how the parameter cost of a steerable filter bank depends on the types of the input and output features, and show how to use this knowledge to construct CNNs that utilize parameters effectively.
1 INTRODUCTION
Much of the recent progress in computer vision can be attributed to the availability of large labelled datasets and deep neural networks capable of absorbing large amounts of information. While many practical problems can now be solved, the requirement for big (labelled) data is a fundamentally unsatisfactory state of affairs. Human beings are able to learn new concepts with very few labels, and reproducing this ability is an important challenge for artificial intelligence research. From an applied perspective, improving the statistical efficiency of deep learning is vital because in many domains (e.g. medical image analysis), acquiring large amounts of labelled data is costly.
To improve the statistical efficiency of machine learning methods, many have sought to learn invariant representations. In deep learning, however, intermediate layers should not be invariant, because the relative pose of local features must be preserved for further layers (Cohen & Welling, 2016; Hinton et al., 2011). Thus, one is led to the idea of equivariance: a network is equivariant if the representations it produces transform in a predictable way under transformations of the input. In other words, equivariant networks produce representations that are steerable. Steerability makes it possible to apply filters not just in every position (as in a standard convolution layer), but in every pose, thus allowing for increased parameter sharing.
Previous work has shown that equivariant CNNs yield state of the art results on classification tasks (Cohen & Welling, 2016; Dieleman et al., 2016), even though they only enforce equivariance to small groups of transformations like rotations by multiples of 90 degrees. Learning representations that are equivariant to larger groups is likely to result in further gains, but the computational cost of current methods scales linearly with the size of the group, making this impractical. In this paper we present the general theory of steerable CNNs, which covers previous approaches but also shows how the computational cost can be decoupled from the size of the symmetry group, thus paving the way for future scaling.
To better understand the structure of steerable representations, we analyze them mathematically. We show that any steerable representation is a composition of low-dimensional elementary feature types. Each elementary feature can be steered independently of the others, and captures a distinct characteristic of the input that has an invariant or “objective” meaning. This doctrine of “observerindependent quantities” was put forward by (Weyl, 1939, ch. 1.4) and is used throughout physics. It has been applied to vision and representation learning by Kanatani (1990); Cohen (2013).
The mentioned type system puts constraints on the network weights and architecture. Specifically, since an equivariant filter bank is required to map given input feature types to given output feature types, the number of parameters required by such a filter bank is reduced. Furthermore, by the same logic that tells us not to add meters to seconds, steerability considerations prevent us from adding features of different types (e.g. for residual learning (He et al., 2016a)).
The rest of this paper is organized as follows. The theory of steerable CNNs is introduced in Section 2. Related work is discussed in Section 3, which is followed by classification experiments (Section 4) and a discussion and conclusion in Section 5.
2 STEERABLE CNNS
2.1 FEATURE MAPS AND FIBERS
Consider a 2D signal f : Z2 → RK with K channels. The signal may be an input to the network or a feature representation computed by a CNN. Since signals can be added and multiplied by scalars, the set of signals of this signature forms a linear space F . Each layer of the network has its own feature space Fl, but we will often suppress the layer index to reduce clutter. It is customary in deep learning to describe f ∈ F as a stack of feature maps fk (for k = 1, . . . ,K). In this paper we also consider another decomposition of F into fibers. The fiber Fx at position x in the “base space” Z2 is the K-dimensional vector space spanned by all channels at position x. Thus, f ∈ F is comprised of feature vectors f(x) that live in the fibers Fx (see Figure 1(a)).
Given some group of transformations G that acts on points in Z2, we can transform signals f ∈ F0:
[π0(g)f ] (x) = f(g −1x) (1)
This says that the pixel at g−1x gets moved to x by the transformation g ∈ G. We note that π0(g) is a linear operator.
An important property of π0 is that π0(gh) = π0(g)π0(h). Here, gh means composition of transformations in G, while π0(g)π0(h) denotes matrix multiplication. A vector space such as F0 equipped with a set of linear operators π0 satisfying this condition is known as a group representation (or just representation, for short). A lot is known about group representations (Serre, 1977), and we will make extensive use of the theory, explaining the relevant concepts as needed.
2.2 STEERABLE REPRESENTATIONS
Let (F , π) be a feature space with a group representation and Φ : F → F ′ a convolutional network. The feature space F ′ is said to be (linearly) steerable with respect to G, if for all transformations g ∈ G, the features Φf and Φπ(g)f are related by a linear transformation π′(g) that does not depend on f . So π′(g) allows us to “steer” the features in F ′ without referring to the input in F from which they were computed.
Combining the definition of steerability (i.e. Φπ(g) = π′(g)Φ) with the fact that π is a group representation, we find that π′ must also be a group representation:
π′(gh)Φf = Φπ(gh)f = Φπ(g)π(h)f = π′(g)Φπ(h)f = π′(g)π′(h)Φf (2)
That is, π′(gh) = π′(g)π′(h) (at least in the span of the image of Φ). Figure 2 gives an illustration.
For simplicity, we will restrict our attention to discrete groups of transformations. The theory for continuous groups is almost completely analogous. Our running example will be the group p4m which consists of translations, rotations by 90 degrees around any point, and reflections. We further restrict our attention to groups that are constructed1 from the group of translations Z2 and a group H of transformations that fixes the origin 0 ∈ Z2. For p4m, we have H = D4, the 8-element group of reflections and rotations about the origin.
Using this division, we can first construct a filter bank that generates H-steerable fibers, and then show that convolution with such a filter bank produces a feature space that is steerable with respect to the whole group G.
2.3 EQUIVARIANT FILTER BANKS
A filter bank can be described as an array of dimension (K ′,K, s, s), where K,K ′ denote the number of input / output channels and s is the kernel size. For our purposes it is useful to think of a filter bank as a linear map Ψ : F → RK′ that takes as input a signal f ∈ F and produces a K ′-dimensional feature vector. The filter bank only looks at an s × s patch in F , so the matrix representing Ψ has shape K ′ ×K · s2. To correlate a signal f using Ψ, one would simply apply Ψ to translated copies of f , producing the output signal one fiber at a time.
We assume (by induction) that we have a representation π that allows us to steer F . In order to make the output of the convolution steerable, we need the filter bank Ψ : F → RK′ to be H-equivariant:
ρ(h) Ψ = Ψπ(h), ∀h ∈ H (3)
for some representation ρ of H that acts on the output fibers (see Figure 3). Note that we only require equivariance with respect to H (which excludes translations) and not G, because translations can move patterns into and out of the receptive field of a fiber, making full translation equivariance impossible.
The space of maps satisfying the equivariance constraint is denoted HomH(π, ρ), because an equivariant map Ψ is a “homomorphism of group representations”, meaning it respects the structure of the representations. Equivariant maps are also sometimes called intertwiners (Serre, 1977).
Since the equivariance constraint (eq. 3) is linear in Ψ, the space HomH(π, ρ) of admissible filter banks is a vector space: any linear combination of maps Ψ,Ψ′ ∈ HomH(π, ρ) is again
an intertwiner. Hence, given π and ρ, we can compute a basis for HomH(π, ρ) by solving a linear system.
1as a semi-direct product
Computation of the intertwiner basis is done offline, before training. Once we have such a basis ψ1, . . . , ψn for HomH(π, ρ), we can express any equivariant filter bank Ψ as a linear combination Ψ = ∑ i αiψi using parameters αi. As shown in Section 2.8, this can be done efficiently even in high dimensions.
2.4 INDUCTION
We have shown how to parameterize filter banks that intertwine π and ρ, making the output fibersHsteerable by ρ if the input space F is H-steerable by π. In this section we show how H-steerability of fibers F ′x leads to G-steerability of the whole feature space F ′. This happens through a natural and important construction known as the induced representation (Mackey, 1952; 1953; 1968; Serre, 1977; Taylor, 1986; Folland, 1995; Kaniuth & Taylor, 2013).
As stated before, the correlation Ψ ? f could be computed by translating f before applying Ψ: [Ψ ? f ] (x) = Ψ [ π(x)−1f ] . (4)
Where x ∈ Z2 is interpreted as a translation when given as input to π. We can now calculate the transformation law of the output space. To do so, we apply a translation t and transformation r ∈ H to f ∈ F , yielding π(tr)f , and then perform the correlation with Ψ. With a some algebra (Appendix A), we find:
[Ψ ? [π(tr)f ]] (x) = ρ(r) [ [Ψ ? f ] ((tr)−1x) ] (5)
Now if we define π′ as [π′(tr)f ] (x) = ρ(r) [ f((tr)−1x) ] (6)
then Ψ ? π(g)f = π′(g)Ψ ? f (see Fig. 4). This representation π′ is known as the representation of G induced by the representation ρ of H , and is denoted π′ = IndGH ρ.
When parsing eq. 6, it is important to keep in mind that (as indicated by the square brackets) π′ acts on the whole feature space F ′ while ρ acts on individual fibers. If we compare the induced representation (eq. 6) to the representation π0 defined in eq. 1, we see that the difference lies only in the presence of a factor ρ(r) applied to the fibers. This factor describes how the feature channels are mixed by the transformation. The color channels in the input space do not get mixed by geometrical transformations, so we say that π0 is induced from the trivial representation ρ0(h) = I .
Now that we have a G-steerable feature space F ′, we can iterate the procedure by computing a basis for the space of intertwiners between π′ (restricted to H) and some ρ′ of our choosing.
2.5 FEATURE TYPES AND CHARACTER THEORY
By now, the reader may be wondering how to choose ρ, or indeed what the space of representations that we can choose from looks like in the first place. We will answer these questions in this section by showing that each representation has a type (encoded as a short list of integers) that corresponds to a certain symmetry or invariance of the feature. We further show how the number of parameters of an equivariant filter bank depends on the types of the representations π and ρ that it intertwines. Our discussion will make use of a number of important elementary results from group representation theory which are stated but not proved. The reader wishing to go deeper may consult chapters 1 and 2 of the excellent book by Serre (1977).
Recall that a group representation is a set of invertible linear maps ρ(g) : RK → RK satisfying ρ(gh) = ρ(g)ρ(h) for all elements g, h ∈ H . It can be shown that any representation is a direct sum (i.e. block_diag plus change of basis) of a number of “elementary” representations associated with G. These building blocks are called irreducible representations (or irreps), because they can
themselves not be block-diagonalized. In other words, if ϕi are the irreducible representations of H , any representation ρ of H can be written in block-diagonal form:
ρ(g) = A ϕi1(g) . . . ϕin A−1 (7) for some basis matrix A, and some ik that index the irreps (each irrep may occur 0 or more times).
Each irreducible representation corresponds to a type of symmetry, as shown in table 1. For example, as can be seen in this table, the representations B1 and B2 represent the 90-degree rotation r as the matrix [−1], so the basis filters for these representations change sign when rotated by r. It should be noted that in the higher layers l > 0, elementary basis filters can look different because they depend on the representation πl that is being decomposed.
The fact that all representations can be decomposed into a direct sum of irreducibles implies that each representation has a basis-independent type: which irreducible representations appear in it, and with what multiplicity. For example, the input representation π0 (table 1) has type (3, 0, 1, 1, 2). This means that, for instance, π0(r) is block-diagonalized as:
A−1π0(r)A = block_diag([1] , [1] , [1] , [−1] , [−1] , [0 −1; 1 0] , [0 1;−1 0]). (8)
Where the block matrix contains (3, 0, 1, 1, 2) copies of the irreps (A1, A2, B1, B2, E), evaluated at r (see column r in table 1). The change of basis matrix A is constructed from the basis filters shown in table 1 (and the same A block-diagonalizes π0(g) for all g).
So the most general way in which we can choose a representation ρ is to choose multiplicities mi ≥ 0 and a basis matrixA. In Section 2.7 we will find that there is an important restriction on this freedom, which alleviates the need to choose a basis. The choice of multiplicities is then the only hyperparameter, analogous to the choice of the number of channels in an ordinary CNN. Indeed, the multiplicities determine the number of channels: K = ∑ imi dimϕi.
2.6 DETERMINING THE TYPE OF THE INDUCED REPRESENTATION
By choosing the type of ρ, we also determine the type of π = IndGH ρ (restricted toH), but what is it? Explicit formulas exist (Reeder (2014); Serre (1977)) but are rather complicated, so we will present a simple computational procedure that can be used to determine the type of any representation. This procedure relies on the character χρ(g) = Tr(ρ(g)) of the representation to be decomposed. The most important fact about characters is that the characters of irreps ϕi, ϕj are orthogonal:
〈χϕi , χϕj 〉 ≡ 1 |H| ∑ h∈H χϕi(h)χϕj (h) = δij . (9)
Furthermore, since the trace of a direct sum equals the sum of the traces (i.e. χρ⊕ρ′ = χρ + χρ′ ), and every representation ρ is a direct sum of irreps, it follows that we can obtain the multiplicity of irrep ϕi in ρ by computing the inner product with the i-th character:
〈χρ, χϕi〉 = 〈χ⊕jmjϕj , χϕi〉 = 〈∑ j mjχϕj , χϕi 〉 = ∑ j mj〈χϕj , χϕi〉 = mi (10)
So a simple dot product of characters is all we need to determine the type of a representation. As we will see next, the type of the input and output representation of a layer determines the parameter cost of that layer.
2.6.1 THE PARAMETER COST OF EQUIVARIANT CONVOLUTION LAYERS
Steerable CNNs use parameters much more efficiently than ordinary CNNs. In this section we show how the number of parameters required by an equivariant layer is determined by the feature types of the input and output space, and how the efficiency of a choice of feature types may be evaluated.
In section 2.3, we found that a filter bank Ψ is equivariant if and only if it lies in the vector space called HomH(π, ρ). It follows that the number of parameters for such a filter bank is equal to the dimensionality of this space, n = dim HomH(π, ρ). This number is known as the intertwining number of π and ρ and plays an important role in the theory of group representations.
As with multiplicities, the intertwining number is easily computed using characters. It can be shown (Reeder, 2014) that the intertwining number equals:
dim HomH(π, ρ) = 〈χπ, χρ〉. (11) By linearity and the orthogonality of characters, we find that dim HomH(π, ρ) = ∑ imim ′ i, for representations π, ρ of type (m1, . . . ,mJ) and (m′1, . . . ,m ′ J), respectively. Thus, as far as the number of parameters of a steerable convolution layer is concerned, the only choice we have to make for ρ is its type – a short list of integers mi.
The efficiency of a choice of type can be assessed using a quantity we call the parameter utilization:
µ = dimπ · dim ρ
dim HomH(π, ρ) . (12)
The numerator equals s2K · K ′: the number of parameters for a non-equivariant filter bank. The denominator equals the parameter cost of an equivariant filter bank with the same filter size and number of input/output channels. Typical values of µ in effective architectures are around |H|, e.g. µ = 8 for H = D4. Such a layer utilizes its parameters 8 times more intensively than an ordinary convolution layer.
2.7 EQUIVARIANT NONLINEARITIES & CAPSULES
In the previous section we showed that only the basis-independent types of π and ρ play a role in determining the parameter cost of an equivariant filter bank. An equivalent representation ρ′(g) = Aρ(g)A−1 will have the same type, and hence the same parameter cost as ρ. However, when it comes to nonlinearities, different bases behave differently.
Just like a convolution layer (eq. 3), a layer of nonlinearities must commute with the group action. An elementwise nonlinearity ν : R → R (or more generally, a fiber-wise nonlinearity ν : RK → RK′ ) is admissible for an input representation ρ if there exists an output representation ρ′ such that ν applied after ρ equals ρ′ applied after ν.
Since commutation with nonlinearities depends on the basis, we need a more granular notion than the feature type. We define a ρ-capsule as a (typically low-dimensional) feature vector that transforms according to a representation ρ (we may also refer to ρ as the capsule). Thus, while a capsule has a type, not all representations of that type are equivalent as capsules. Given a catalogue of capsules ρi (for i = 1, . . . , C) with multiplicities mi, we can construct a fiber as a stack of capsules that is steerable by a block-diagonal representation ρ with mi copies of ρi on the diagonal.
Like the capsules of Hinton et al. (2011), our capsules encode the pose of a pattern in the input, and consist of a number of units (dimensions) that do not get mixed with the units of other capsules by symmetries. In this sense, a stack of capsules is disentangled (Cohen & Welling, 2014).
We have found a few simple types of capsules and corresponding admissible nonlinearities. It is easy to see that any nonlinearity is admissible for ρ when the latter is realized by permutation matrices: permuting a list of coordinates and then applying a nonlinearity is the same as applying the nonlinearity and then permuting. If ρ is realized by a signed permutation matrix, then CReLU(α) = (ReLU(α),ReLU(−α)) introduced by Shang et al. (2016), or any concatenated nonlinearity ν′(α) = (ν(α), ν(−α)), will be admissible. Any scale-free concatenated nonlinearity such as CReLU is admissible for a representation realized by monomial matrices (having the same nonzero pattern as a permutation matrix). Finally, we can always make a representation of a finite group orthogonal by a suitable choice of basis, which means that we can use any nonlinearity that acts only on the length of the vector.
For many groups, the irreps can be realized using signed permutation matrices, so we can use irreducible ϕi-capsules with concatenated nonlinearities such as CReLU. Another class of capsules, which we call quotient capsules, are naturally realized by permutation matrices, and are thus compatible with any nonlinearity. These are described in Appendix C.
2.8 COMPUTATIONAL EFFICIENCY
Modern convolutional networks often use on the order of hundreds of channels K per layer Zagoruyko & Komodakis (2016). When using 3 × 3 filters, a filter bank can have on the order of 9K2 ≈ 106 dimensions. The number of parameters for an equivariant filter bank is about µ ≈ 10 times smaller, but a basis for the space of equivariant filter banks would still be about 106 × 105, which is too large to be practical.
Fortunately, the block-diagonal structure of π and ρ induces a block structure in Ψ. Suppose π = block_diag(π1, . . . , πP ) and ρ = block_diag(ρ1, . . . , ρQ). Then an intertwiner is a matrix of shape K ′ ×Ks2, where K ′ = ∑ i dim ρ i and Ks2 = ∑ i dimπ
i. This matrix has the following block structure:
Ψ =  h11 ∈ HomH(ρ 1, π1) · · · h1P ∈ HomH(ρ1, πP ) ... . . .
... hR1 ∈ HomH(ρR, π1) · · · hRP ∈ HomH(ρR, πP )  (13) Each block hij corresponds to an input-output pair of capsules, and can be parameterized by a linear combination of basis matrices ψijk ∈ HomH(ρi, πj).
In practice, we typically use many copies of the same capsule (say ni copies of ρi and mj copies of πj). Therefore, many of the blocks hij can be constructed using the same intertwiner basis. If we order equivalent capsules to be adjacent, the intertwiner consists of “blocks of blocks”. Each superblock Hij has shape ni dim ρi × mj dimπj , and consists of subblocks of shape dim ρi × dimπj .
The computation graph for an equivariant convolution layer is constructed as follows. Given a catalogue of capsules ρi and corresponding post-activation capsules Actν ρi, we compute the induced representations πi = IndGH Actν ρ
i and the bases for HomH(ρi, πj) in an offline step. The bases are stored as matrices ψij of shape dim ρi · dimπj × dim HomH(ρi, πj). Then, given a list of input / output multiplicities ni,mj for the capsules, a parameter matrix Θij of shape dim HomH(ρ
i, πj) × nimj is instantiated. The superblocks Hij are obtained by a matrix multiplication ψijΘij plus reshaping to shape dim ρi ·dimπj ×nimj . Once all superblocks are filled in, the matrix Ψ is reshaped from K ′ ×Ks2 to K ′ ×K × s× s and convolved with the input.
2.9 USING STEERABLE CNNS IN PRACTICE
A full understanding of the theory of steerable CNNs requires some knowledge of group representation theory, but using steerable CNN technology is not much harder than using ordinary CNNs. Instead of choosing a number of channels for a given layer, one chooses a list of multiplicities mi for each capsule in a library of capsules provided by the developer. To preserve equivariance, the activation function applied to a capsule must be chosen from a list of admissible nonlinearities for that capsule (which sometimes includes all nonlinearities). Finally, one must respect the type system and only add identical capsules (e.g. in ResNets). These constraints can all be checked automatically.
3 RELATED WORK
Steerable filters were first studied for applications in signal processing and low-level vision (Freeman & Adelson, 1991; Greenspan et al., 1994; Simoncelli & Freeman, 1995). More or less explicit connections between steerability and group representation theory have been observed by Lenz (1989); Koenderink & Van Doorn (1990); Teo (1998); Krajsek & Mester (2007). As we have tried to demonstrate in this paper, representation theory is indeed the natural mathematical framework in which to study steerability.
In machine learning, equivariant kernels were studied by Reisert (2008); Skibbe (2013). In the context of neural networks, various authors have studied equivariant representations. Capsules were introduced in Hinton et al. (2011), and significantly improved by Tieleman (2014). A theoretical account of equivariant representation learning in the brain is given by Anselmi et al. (2014). Group equivariant scattering networks were defined and studied by Mallat (2012) for compact groups, and by Sifre & Mallat (2013); Oyallon & Mallat (2015) for the roto-translation group. Jacobsen et al. (2016) describe a network that uses a fixed set of (possibly steerable) basis filters with learned weights. Lenc & Vedaldi (2015) showed empirically that convolutional networks tend to learn equivariant representations, which suggests that equivariance could be a good inductive bias.
Invariant and equivariant CNNs have been studied by Gens & Domingos (2014); Kanazawa et al. (2014); Dieleman et al. (2015; 2016); Cohen & Welling (2016); Marcos et al. (2016). All of these models, as well as scattering networks, implicitly use the regular representation: feature maps are (often implicitly) conceived of as functions onG, and the action ofG on the space of functions onG is known as the regular representation (Serre (1977), Appendix B). Our work is the first to consider other kinds of equivariance in the context of CNNs.
The idea of adding a type system to neural networks has been explored by Olah (2015); Balduzzi & Ghifary (2016). We have shown that a type system emerges naturally from the decomposition of a linear representation of a mathematical structure (a group, in our case) associated with the representation learned by a neural network.
4 EXPERIMENTS
We implemented steerable CNNs in Chainer (Tokui et al., 2015) and performed experiments on the CIFAR10 dataset (Krizhevsky, 2009) to determine if steerability is a useful inductive bias, and to determine the relative merits of the various types of capsules. In order to run experiments faster, and to see how steerable CNNs perform in the small-data regime, we used only 2000 training samples for our initial experiments.
As a baseline, we used the competitive wide residual networks (ResNets) architecture (He et al., 2016a;b; Zagoruyko & Komodakis, 2016). We tuned the capacity of this network for the reduced dataset size and settled on a 20 layer architecture (three residual blocks per stage, with two layers each, for three stages with feature maps of size 32 × 32, 16 × 16 and 8 × 8, various widths). We compared the baseline architecture to various kinds of steerable CNN, obtained by replacing the convolution layers by steerable convolution layers. To make sure that differences in performance were not simply due to underfitting or overfitting, we tuned the width (number of channels, K) using a validation set. The rest of the training procedure is identical to Cohen & Welling (2016), and is fixed for all of our experiments.
We first tested steerable CNNs that consist entirely of a single kind of capsule. We found that architectures with only one type do not perform very well (roughly 30-40% error, vs. 30% for plain ResNets trained on 2k samples from CIFAR10), except for those that use the regular representation capsule (Appendix C), which outperforms standard CNNs (26.75% error). This is not too surprising, because many capsules are quite restrictive in the spatial patterns they can express. The strong performance of regular capsules is consistent with the results of Cohen & Welling (2016), and can be explained by the fact that the regular representation contains all other (irreducible and quotient) representations as subrepresentations, and can therefore learn arbitrary spatial patterns.
We then created networks that use a mix of the more successful kinds of capsules. After a few preliminary experiments, we settled on a residual network that uses one mix of capsules for the input and output layer of a residual block, and another for the intermediate layer. The first representation
consists of quotient capsules: regular, qm, qmr2, qmr3 (see Appendix C) followed by ReLUs. The second consists of irreducible capsules: A1, A2, B1, B2, E(2x) followed by CReLUs. On CIFAR10 with 2k labels, this architecture works better than standard ResNets and regular capsules at 24.48% error.
When tested on CIFAR10 with 4k labels (table 2), the method comes close to the state of the art in semi-supervised methods, that use additional unlabelled data (Rasmus et al., 2015), and better than transfer learning approaches such as DCGAN which achieves 26.2% error (Radford et al., 2015). When tested on the full CIFAR10 and CIFAR100 dataset, the steerable CNN substantially outperforms the ResNet (He et al., 2016b) baseline and achieves state of the art results (improving over wide and dense nets (Zagoruyko & Komodakis, 2016; Huang et al., 2016)).
5 CONCLUSION & FUTURE WORK
We have presented a theoretical framework for understanding steerable representations in convolutional networks, and have shown that steerability is a useful inductive bias that can improve model accuracy, particularly when little data is available. Our experiments show that a simple steerable architecture achieves state of the art results on CIFAR10 and CIFAR100, outperforming recent architectures such as wide and dense residual networks.
The mathematical connection between representation learning and representation theory that we have established improves our understanding of the inner workings of (equivariant) convolutional networks, revealing the humble CNN as an elegant geometrical computation engine. We expect that this new tool (representation theory), developed over more than a century by mathematicians and physicists, will greatly benefit future investigations in this area.
For concreteness, we have used the group of flips and rotations by multiples of 90 degrees as a running example throughout this paper. This group already has some nontrivial characteristics (such as non-commutativity), but it is still small and discrete. The theory of steerable CNNs, however, readily extends to the continuous setting. Evaluating steerable CNNs for large, continuous and high-dimensional groups is an important piece of future work.
Another direction for future work is learning the feature types, which may be easier in the continuous setting because (for non-compact groups) the irreps live in a continuous space where optimization may be possible. Beyond classification, steerable CNNs are likely to be useful in geometrical tasks such as action recognition, pose and motion estimation, and continuous control tasks.
ACKNOWLEDGMENTS
We kindly thank Kenta Oono, Shuang Wu, Thomas Kipf and the anonymous reviewers for their feedback and suggestions. This research was supported by Facebook, Google and NWO (grant number NAI.14.108).
APPENDIX A: INDUCTION
In this section we will show that a stack of feature maps produced by convolution with an Hequivariant filter bank transforms according to the induced representation. That is, we will derive eq. 5, repeated here for convenience:
[Ψ ? [πl(tr)f ]] (x) = ρl+1(r) [ [Ψ ? f ] ((tr)−1x) ] (14)
In the main text, we mentioned that x ∈ Z2 can be interpreted as a point or as a translation. Here we make this difference explicit, by writing x ∈ Z2 for a point and x̄ ∈ G for a translation. (The operation ·̄ defines a section of the projection map G→ Z2 that forgets the non-translational part of the transformation (Kaniuth & Taylor, 2013)).
With this notation, the convolution is defined as:
[Ψ ? f ] (x) = Ψπ(x̄−1)f (15)
Although the induced representation can be described in a more general setting, we will use an explicit matrix representation of G to make it easier to check our computations. A general element of G is written as:
g = tr = [ I T 0 1 ] [ R 0 0 1 ] = [ R T 0 1 ] (16)
Where R is the matrix representation of r (e.g. a 2 × 2 rotation / reflection matrix), and T is a translation vector. The section we use is:
x = [ I x 0 1 ] (17)
Finally, we will distinguish the action of G on itself, written gh for g, h ∈ G (implemented as matrix-matrix multiplication) and its action on Z2, written g ·x for g ∈ G and x ∈ Z2 (implemented as matrix-vector multiplication by adding a homogeneous coordinate to x).
To keep notation uncluttered, we will write π = πl and ρ = ρl+1. In full detail, the derivation of the transformation law for the feature space induced by ρ proceeds as follows:
[Ψ ? [π(tr)f ]] (x) = Ψπ(x̄−1)π(tr)f
= Ψπ(x̄−1tr)f
= Ψπ(rr−1x̄−1tr)f
= Ψπ(r)π(r−1x̄−1tr)f
= ρ(r)Ψπ(r−1x̄−1tr)f
= ρ(r)Ψπ((r−1t−1x̄r)−1)f = ρ(r)Ψπ ( (tr)−1 · x) −1) f
= ρ(r)[Ψ ? f ]((tr)−1 · x)
(18)
The last line is the result shown in the paper. The justification of each step is:
1. Definition of ?
2. π is a homomorphism / group representation
3. rr−1 is the identity, so can always multiply by it 4. π is a homomorphism / group representation 5. Ψ ∈ HomH(π, ρ) is equivariant to r ∈ H . 6. Invert twice. 7. (tr)−1 · x = r−1t−1xr can be checked by multiplying the matrices / vectors. 8. Definition of ?
The derivation above is somewhat involved and messy, so the reader may prefer to think geometrically (using the figures in the paper) instead of algebraically. This complexity is an artifact of the lack of abstraction in our presentation. The induced representation is really a very natural object to consider (abstractly, it is the “adjoint functor” to the restriction functor. A more abstract treatment of the induced representation can be found in Serre (1977); Mackey (1952); Reeder (2014). A treatment that is close to our own, but more general is the “alternate description” found on page 49 of Kaniuth & Taylor (2013).
APPENDIX B: RELATION TO GROUP EQUIVARIANT CNNS
In this section we show that the recently introduced Group Equivariant Convolutional Networks (GCNNs, Cohen & Welling (2016)) are a special kind of steerable CNN. Specifically, a G-CNN is a steerable CNN with regular capsules.
In a G-CNN, the feature maps (except those of the input) are thought of as functions f : G → RK instead of functions on the plane f : Z2 → RK , as we do here. It is shown that the feature maps transform according to π(g)f(h) = f(g−1h). (19) This defines a linear representation of G known as the regular representation. It is easy to see that the regular representation is naturally realized by permutation matrices. Furthermore, it is known that the regular representation of G is induced by the regular representation of H . The latter is defined in Appendix C, and is what we refer to as “regular capsules” in the paper.
APPENDIX C: REGULAR AND QUOTIENT FEATURES
Let H be a finite group. A subgroup of H is a subset that is also itself a group (i.e. closed under composition and inverses). The (left) coset of a subgroup K in H are the sets hK = {hk|k ∈ K}. The cosets are disjoint and jointly cover the whole group H (i.e. they partition H). The set of all cosets of K in H is denoted H/K, and is also called the quotient of H by K.
The coset space caries a natural left action by H . Let a, b ∈ H , then a · bK = (ab)K. This action translates into an action on the space of functions on H/K. Let Q denote the space of functions f : H/K → R. Then we have the following representation of H:
ρ(a)f(bK) = f(a−1 · bK). (20)
The function f attaches a value to every coset. The H-action permutes these values, because it permutes the cosets. Hence, ρ can be realized by permutation matrices. For small groups the explicit computations can easily be done by hand, while for large groups this task can be automated.
In this way, we get one permutation representation for each subgroup K of H . In particular, for the subgroup K = {e} (the trivial subgroup containing only the identity e), we have H/K ∼= H . The representation in the space of functions on H is known as the “regular representation”. Using such regular representations in a steerable CNN is equivalent to using the group convolutions introduced in Cohen & Welling (2016), so steerable CNNs are a strict generalization of G-CNNs. At the other extreme, we take K = H , which gives the quotient H/K ∼= {e}, the trivial group, which gives the trivial representation A1.
For the roto-reflection group H = D4, we have the following subgroups and associated quotient features
Subgroup K quotient feature name dimensionality {e} regular 8 {e,m} qm 4 {e,mr} qmr 4 {e,mr2} qmr2 4 {e,mr3} qmr3 4 {e, r2} r2 4
{e, r, r2, r3} r 2 e, r2,m,mr2 r2m 2 e, r2,mr,mr3 r2mr 2
H A1 1

The authors propose a parameterization of CNNs that guarantees equivariance wrt a large family of geometric transformations.
The mathematical analysis is rigorous and the material is very interesting and novel. The paper overall reads well; there is a real effort to explain the math accessibly, though some small improvements could be made.
The theory is general enough to include continuous transformations, although the experiments are restricted to discrete ones. While this could be seen as a negative point, it is justified by the experiments, which show that this set of transformations is powerful enough to yield very good results on CIFAR.
Another form of intertwiner has been studied recently by Lenc & Vedaldi [1]; they have studied equivariance empirically in CNNs, which offers an orthogonal view.
In addition to the recent references on scale/rotation deep networks suggested below, geometric equivariance has been studied extensively in the 2000's; mentioning at least one work would be appropriate. The one that probably comes closest to the proposed method is the work by Reisert [2], who studied steerable filters for invariance and equivariance, using Lie group theory. The difference, of course, is that the focus at the time was on kernel machines rather than CNNs, but many of the tools and theorems are relatable.
Some of the notation could be simplified, to make the formulas easier to grasp on a first read:
Working over a lattice Z^d is unnecessarily abstract -- since the inputs are always images, Z^2 would make much of the later math easier to parse. Generalization is straightforward, so I don't think the results lose anything by it; and the authors go back to 2D latices later anyway.
It could be more natural to do away with the layer index l which appears throughout the paper, and have notation for current/next layer instead (e.g. pi and pi'; K and D instead of K_{l+1} and K_l).
In any case I leave it up to the authors to decide whether to include these suggestions on notation, but I urge them to consider them (or other ways to unburden notation).
A few minor issues: Some statements would be better supported with an accompanying reference (e.g. "Explicit formulas exist" on page 5, the introduction of intertwiners on page 3). Finally, there is a tiny mistake in the Balduzzi & Ghifary reference (some extra information was included as an author name).
[1] Lenc & Vedaldi, "Understanding image representations by measuring their equivariance and equivalence", 2015
[2] Reisert, "Group integration techniques in pattern analysis: a kernel view", 2008