summaries: This paper proposes DiffStride as a drop-in replacement to standard downsampling layers. It extends previous work Spectral Poolnig and learns the size of the cropping box in the frequency domain by backpropagation. Experiments are conducted on audio and image classification, and the results show that the model can learn non-integer stride and adapt different initial stride well. In this work the authors introduce a differentiable stride formulation, which allows for learning the stride value. To this end, they propose learning the size of a cropping mask in the Fourier domain, which allows for learning how to perform resize in a differentiable way. Authors present experiments on several datasets on different domains (image, audio), including large scale datasets. In this paper, the authors propose DiffStride, a technique for learning the stride of downsampling operations in neural networks by gradient descent. Specifically, similarily to SpectralPool, the feature map is transformed to the frequency domain by a Discrete Fourier Transform, where it is then cropped according to learnable parameters. The authors envision a simple soft-relaxation of the cropping in order to allow the gradient to flow towards cropping parameters. Performances against both standard and random stride policies are reported on a number of datasets, both for audio and image recognition. Moreover, the authors introduce a regularization objective that penalizes the use of small strides, in the interest of encouraging downsampling for improving computational and memory cost. This paper proposes an approach to learning optimal striding parameters in convolutional networks. The proposed approach, DiffStride, is a downsampling layer that builds on spectral pooling to allow for integer output dimensions, but arbitrary strides by cropping in the Fourier domain. Unlike spectral pooling, DiffStride relaxes the parameters of the cropping mask to be differentiable, and using the stop-gradient operator in the cropping. Results show that the proposed approach can work well as a drop-in replacement for normally fixed pooling layers. It is also demonstrated that from different random initialisations a variety of different pooling approaches can be learned that acheive similar performance. Use of a regularisation term the attempts to encourage time and space efficiency reduces this variability and allows accuracy to be traded off.
abstractText: Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hyperparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires crossvalidation or discrete optimization (e.g. architecture search), which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at a lower computational cost. This work introduces DiffStride, the first downsampling layer with learnable strides. Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image classification show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations. Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.
references.authors: Dario Amodei Sundaram Ananthanarayanan Rishita Anubhai Jingliang Bai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Qiang Cheng Guoliang Chen
references.title: Deep speech 2: End-toend speech recognition in english and mandarin
references.venue: In International conference on machine learning,
references.authors: Bowen Baker Otkrist Gupta Nikhil Naik Ramesh Raskar
references.title: Designing neural network architectures using reinforcement learning
references.venue: In 5th International Conference on Learning Representations,
references.authors: Irwan Bello William Fedus Xianzhi Du Ekin Dogus Cubuk A. Srinivas Tsung-Yi Lin Jonathon Shlens Barret Zoph
references.title: Revisiting resnets: Improved training and scaling strategies
references.venue: ArXiv, abs/2103.07579,
references.authors: Yoshua Bengio Nicholas Léonard Aaron Courville
references.title: Estimating or propagating gradients through stochastic neurons for conditional computation
references.venue: arXiv preprint arXiv:1308.3432,
references.authors: Simone Bianco Rémi Cadène Luigi Celona Paolo Napoletano
references.title: Benchmark analysis of representative deep neural network architectures
references.venue: IEEE Access,
references.authors: Léon Bottou
references.title: Online learning and stochastic approximations
references.venue: On-line learning in neural networks,
references.authors: Y-Lan Boureau Jean Ponce Yann LeCun
references.title: A theoretical analysis of feature pooling in visual recognition
references.venue: In Proceedings of the 27th international conference on machine learning
references.authors: S. Chopra R. Hadsell Y. LeCun
references.title: Learning a similarity metric discriminatively, with application to face verification
references.venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition
references.authors: Alexis Conneau Holger Schwenk Loı̈c Barrault Yann
references.title: Lecun. Very deep convolutional networks for text classification
references.venue: In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers,
references.authors: Zihang Dai Hanxiao Liu Quoc V Le Mingxing Tan
references.title: Coatnet: Marrying convolution and attention for all data sizes
references.venue: arXiv preprint arXiv:2106.04803,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: In 2009 IEEE conference on computer vision and pattern recognition,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: In International Conference on Learning Representations,
references.authors: Taffeta M Elliott Frédéric E Theunissen
references.title: The modulation transfer function for speech intelligibility
references.venue: PLoS computational biology,
references.authors: Jesse Engel Cinjon Resnick Adam Roberts Sander Dieleman Douglas Eck Karen Simonyan Mohammad Norouzi
references.title: Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
references.venue: Technical report,
references.authors: Adeen Flinker Werner K Doyle Ashesh D Mehta Orrin Devinsky David Poeppel
references.title: Spectrotemporal modulation provides a unifying framework for auditory cortical asymmetries
references.venue: Nature human behaviour,
references.authors: Eduardo Fonseca Andrés Ferraro Xavier Serra
references.title: Improving sound event classification by increasing shift invariance in convolutional neural networks
references.venue: ArXiv, abs/2107.00623,
references.authors: Pierre Foret Ariel Kleiner Hossein Mobahi Behnam Neyshabur
references.title: Sharpness-aware minimization for efficiently improving generalization
references.venue: In International Conference on Learning Representations,
references.authors: K. Fukushima
references.title: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
references.venue: Biological Cybernetics,
references.authors: Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N Dauphin
references.title: Convolutional sequence to sequence learning
references.venue: In International Conference on Machine Learning,
references.authors: Xavier Glorot Antoine Bordes Yoshua Bengio
references.title: Deep sparse rectifier neural networks
references.venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,
references.authors: Benjamin Graham
references.title: Fractional max-pooling
references.venue: arXiv preprint arXiv:1412.6071,
references.authors: Caglar Gulcehre Kyunghyun Cho Razvan Pascanu Yoshua Bengio
references.title: Learned-norm pooling for deep feedforward and recurrent neural networks
references.venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Identity mappings in deep residual networks
references.venue: In European conference on computer vision,
references.authors: Toni Heittola Annamaria Mesaros Tuomas Virtanen
references.title: TUT Urban Acoustic Scenes 2018
references.venue: Development dataset,
references.authors: Geoffrey Hinton
references.title: What’s wrong with convolutional nets. MIT Brain and Cognitive Sciences-Fall Colloquium Series, Dec 2014a
references.authors: Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision
references.venue: applications. ArXiv,
references.authors: Andrew G. Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen Mingxing Tan Weijun Wang Yukun Zhu Ruoming Pang Vijay Vasudevan Quoc V. Le Hartwig Adam
references.title: Searching for mobilenetv3
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV),
references.authors: Gao Huang Zhuang Liu Laurens Van Der Maaten Kilian Q Weinberger
references.title: Densely connected convolutional networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Patrick W Hullett Liberty S Hamilton Nima Mesgarani Christoph E Schreiner Edward F Chang
references.title: Human superior temporal gyrus organization of spectrotemporal modulation tuning derived from speech stimuli
references.venue: Journal of Neuroscience,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: In International conference on machine learning,
references.authors: Chen Jin Ryutaro Tanno Thomy Mertzanidou Eleftheria Panagiotaki Daniel C. Alexander
references.title: Learning to downsample for segmentation of ultra-high resolution images, 2021
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In ICLR (Poster),
references.authors: Qiuqiang Kong Yin Cao Turab Iqbal Yuxuan Wang Wenwu Wang Mark D Plumbley
references.title: Panns: Large-scale pretrained audio neural networks for audio pattern recognition
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: A. Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.venue: Advances in neural information processing systems,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.authors: Scinob Kuroki Masataka Sawayama Shin’ya Nishida
references.title: Haptic texture perception on 3d-printed surfaces transcribed from visual natural textures
references.venue: In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications,
references.authors: Yann LeCun Bernhard Boser John Denker Donnie Henderson Richard Howard Wayne Hubbard Lawrence Jackel
references.title: Handwritten digit recognition with a back-propagation network
references.venue: Advances in neural information processing systems,
references.authors: Chen-Yu Lee Patrick W Gallagher Zhuowen Tu
references.title: Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree
references.venue: In Artificial intelligence and statistics,
references.authors: Richard G. Lyons
references.title: Understanding Digital Signal Processing (2nd Edition)
references.venue: Prentice Hall PTR, USA,
references.authors: Hannah Muckenhirn M. Magimai.-Doss S. Marcel
references.title: Towards directly modeling raw speech signal for speaker verification using cnns
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Biswajit Paria Chih-Kuan Yeh N. Xu B. Póczos Pradeep Ravikumar I.E. Yen
references.title: Minimizing flops to learn efficient sparse representations
references.venue: ArXiv, abs/2004.05665,
references.authors: Ning Qian
references.title: On the momentum term in gradient descent learning algorithms
references.venue: Neural networks : the official journal of the International Neural Network Society,
references.authors: Antônio H. Ribeiro Thomas B. Schön
references.title: How convolutional neural networks deal with aliasing
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing. ICASSP,
references.authors: Oren Rippel Jasper Snoek Ryan P Adams
references.title: Spectral representations for convolutional neural networks
references.venue: In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume
references.authors: Daniel L Ruderman
references.title: The statistics of natural images. Network: computation in neural systems
references.authors: Faraz Saeedan Nicolas Weber Michael Goesele Stefan Roth
references.title: Detail-preserving pooling in deep networks
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Tom Sercu Christian Puhrsch Brian Kingsbury Yann LeCun
references.title: Very deep multilingual convolutional neural networks for lvcsr
references.venue: In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP),
references.authors: Nandini C Singh Frédéric E Theunissen
references.title: Modulation spectra of natural sounds and ethological theories of auditory processing
references.venue: The Journal of the Acoustical Society of America,
references.authors: Dan Stowell — Mike Wood — Hanna Pamuła Yannis Stylianou Hervé Glotin
references.title: Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
references.venue: Technical report,
references.authors: Sainbayar Sukhbaatar Édouard Grave Piotr Bojanowski Armand Joulin
references.title: Adaptive attention span in transformers
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Christian Szegedy Wei Liu Yangqing Jia Pierre Sermanet Scott Reed Dragomir Anguelov Dumitru Erhan Vincent Vanhoucke Andrew Rabinovich
references.title: Going deeper with convolutions
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Marco Tagliasacchi Beat Gfeller Félix de Chaumont Quitry Dominik Roblek
references.title: Self-supervised audio representation learning for mobile devices
references.venue: arXiv preprint arXiv:1905.11796,
references.authors: Hossein Talebi Peyman Milanfar
references.title: Learning to resize images for computer vision
references.venue: tasks. ICCV,
references.authors: Mingxing Tan Quoc Le
references.title: Efficientnet: Rethinking model scaling for convolutional neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Mingxing Tan Bo Chen Ruoming Pang Vijay Vasudevan Mark Sandler Andrew Howard Quoc V Le
references.title: Mnasnet: Platform-aware neural architecture search for mobile
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Aäron van den Oord Nal Kalchbrenner Lasse Espeholt K. Kavukcuoglu Oriol Vinyals A. Graves
references.title: Conditional image generation with pixelcnn decoders
references.venue: In NIPS,
references.authors: Pete Warden
references.title: Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
references.venue: Technical report,
references.authors: P Yin J Lyu S Zhang S Osher YY Qi J Xin
references.title: Understanding straight-through estimator in training activation quantized neural nets
references.venue: In International Conference on Learning Representations,
references.authors: Neil Zeghidour Qiantong Xu Vitaliy Liptchinsky Nicolas Usunier Gabriel Synnaeve Ronan Collobert
references.title: Fully convolutional speech recognition
references.venue: arXiv preprint arXiv:1812.06864,
references.authors: Hao Zhang Jianwei Ma
references.title: Hartley spectral pooling for deep learning
references.venue: arXiv preprint arXiv:1810.04028,
references.authors: Richard Zhang
references.title: Making convolutional networks shift-invariant again
references.venue: In ICML,
references.authors: Barret Zoph Quoc V. Le
references.title: Neural architecture search with reinforcement learning. 2017
references.venue: URL https://arxiv.org/abs/1611.01578
references.authors: Tan Le
references.title: 2019), a lightweight model discovered by architecture
sections.heading: 1 INTRODUCTION
sections.text: Convolutional neural networks (CNNs) (Fukushima, 1980; LeCun et al., 1989) have been the most widely used neural architecture across a wide range of tasks, including image classification (Krizhevsky et al., 2012; He et al., 2016a; Huang et al., 2017; Bello et al., 2021), audio pattern recognition (Kong et al., 2020), text classification (Conneau et al., 2017), machine translation (Gehring et al., 2017) and speech recognition (Amodei et al., 2016; Sercu et al., 2016; Zeghidour et al., 2018). Convolution layers, which are the building block of CNNs, project input features to a higher-level representation while preserving their resolution. When composed with non-linearities and normalization layers, this allows for learning rich mappings at a constant resolution, e.g. autogressive image synthesis (van den Oord et al., 2016). However, many tasks infer high-level low-resolution information (identity of a speaker (Muckenhirn et al., 2018), presence of a face (Chopra et al., 2005)) by integrating over low-level, high-resolution measurements (waveform, pixels). This integration requires extracting the right features, discarding irrelevant information over several downsampling steps. To that end, pooling layers and strided convolutions aggressively reduce the resolution of their inputs, providing several benefits. First, they act as a bottleneck that forces features to focus on information relevant to the task at hand. Second, pooling layers such as low-pass filters (Zhang, 2019) improve shift-invariance. Third, a reduced resolution implies a reduced number of floating-point operations and a higher receptive field in the subsequent layers.
  Pooling layers can usually be decomposed into two basic steps: (1) computing local statistics densely over the whole input (2) sub-sampling these statistics by an integer striding factor. Past work has mostly focused on improving (1), by proposing better alternatives to max and average pooling that avoid aliasing (Zhang, 2019; Fonseca et al., 2021), preserve the important local details (Saeedan et al., 2018), or adapt to the training data distribution (Gulcehre et al., 2014; Lee et al., 2016). Ob-
  serving that integer strides reduce resolution too quickly (e.g. a (2, 2) striding reduces the output size by 75%), Graham (2014) proposed fractional max-pooling, that allows for fractional (i.e. rational) strides, allowing for integration of more downsampling layers into a network. Similarly, Rippel et al. (2015) introduce spectral pooling which, by cropping its inputs in the Fourier domain, performs downsampling with fractional strides while emphasizing lower frequencies.
  While fractional strides give more flexibility in designing downsampling layers, they increase the size of an already gigantic search space. Indeed, as strides are hyperparameters, finding the best combination requires cross-validation or architecture search (Zoph & Le, 2017; Baker et al., 2017; Tan et al., 2019), which rapidly become infeasible as the number of configurations grows exponentially with the number of downsampling layers. This led Zoph & Le (2017) not to search for strides in most of their experiments. Talebi & Milanfar (2021) and Jin et al. (2021) proposed a neural network that learns a resizing function for natural images, but the scaling factor (i.e. the stride) still required cross-validation. Thus, the nature of strides as hyperparameters — rather than trainable parameters — hinders the discovery of convolutional architectures and learning strides by backpropagation would unlock a virtually infinite search space.
  In this work, we introduce DiffStride, the first downsampling layer that learns its strides jointly with the rest of the network. Inspired by Rippel et al. (2015), DiffStride casts downsampling in the spatial domain as cropping in the frequency domain. However, and unlike Rippel et al. (2015), rather than cropping with a fixed bounding box controlled by a striding hyperparameter, DiffStride learns the size of its cropping box by backpropagation. To do so, we propose a 2D version of an attention window with learnable size proposed by Sukhbaatar et al. (2019) for language modeling. On five audio classification tasks, using DiffStride as a drop-in replacement to strided convolutions improves performance overall while providing interpretability on the optimal per-task receptive field. By integrating DiffStride into a ResNet-18 (He et al., 2016a), we show on CIFAR (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) that even when initializing strides randomly, our model converges to the best performance obtained with the properly cross-validated strides of He et al. (2016a). Moreover, casting strides as learnable parameters allows us to propose a regularization that directly minimizes computation and memory usage.
sections.heading: 2 METHODS
sections.text: We first provide background on spatial and spectral pooling, and propose DiffStride for learning strides of downsampling layers. We focus on 2D CNNs since they are generic enough to be used for image (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016a) and audio (Amodei et al., 2016; Kong et al., 2020) processing (taking time-frequency representations as inputs). However, these methods are equally applicable to the 1D (e.g. time-series) and 3D (e.g. video) cases.
sections.heading: 2.1 NOTATIONS
sections.text: Let x ∈ RH×W , its Discrete Fourier Transform (DFT) y = F(x) ∈ CH×W is obtained through the decomposition on a fixed set of basis filters (Lyons, 2004):
  F(x)mn = 1√ HW H−1∑ h=0 W−1∑ w=0 xhwe −2πi(mhH + nw W ),∀m ∈ {0, . . . ,H − 1},∀n ∈ {0, . . . ,W − 1}.
  (1)
  The DFT transformation is linear and its inverse is given by its conjugate F(.)−1 = F(.)∗. The Fourier transform of a real-valued signal x ∈ RH×W being conjugate symmetric (Hermitiansymmetry), we can reconstruct x from the positive half frequencies for the width dimension and omit the negative frequencies (ymn = y∗(H−m)modH,(W−n)modW ). In addition, the DFT and its inverse are differentiable with regard to their inputs and the derivative of the DFT (resp. inverse DFT) is its conjugate linear operator, i.e. the inverse DFT (resp. DFT). More formally, if we consider L : CH×W −→ R as a loss taking as input the Fourier representation y, we can compute the gradient of L with regard to x, by using the inverse DFT:
  x ∈ RH×W , y = F(x), ∂L ∂x = F∗(∂L ∂y ) = F−1(∂L ∂y ). (2)
  We denote by L the total number of convolution layers in a CNN architecture and each layer is indexed by l. The ◦ symbol represents the element-wise product between two tensors, b.c is the floor operation and ⊗ the outer product between two vectors. S represents the stride parameters, and sg is the stop gradient operator (Bengio et al., 2013; Yin et al., 2019), defined has the identity function during forward pass and with zero partial derivatives.
sections.heading: 2.2 DOWNSAMPLING IN CONVOLUTIONAL NEURAL NETWORKS
sections.text: A basic mechanism for downsampling representations in a CNN is strided convolutions which jointly convolve inputs and finite impulse response filters and downsample the output. Alternatively, one can disentangle both operations by first applying a non-strided convolution followed by a pooling operation that computes local statistics (e.g. using an average, max (Boureau et al., 2010)) before downsampling. In both settings, downsampling does not benefit from the global structure of its inputs and can discard important information (Hinton, 2014; Saeedan et al., 2018). Moreover, and as observed by Graham (2014), the integer nature of strides only allows for drastic reductions in resolution: a 2D-convolution with strides S = (2, 2) reduces the dimension of its inputs by 75%. Furthermore, stride configurations are cumbersome to explore as the number of stride combinations grows exponentially with the number of downsampling layers. This means that cross-validation can only explore a limited subset of the stride hyperparameter configurations. This limitation is likely to translate into lower performance, as Section 3.2 shows that an inappropriate choice of strides for a ResNet-18 architecture can account for a drop of > 18% in accuracy on CIFAR-100.
sections.heading: 2.3 SPECTRAL POOLING
sections.text: Energy of natural signals is typically not uniformly distributed in the frequency domain, with signals such as sounds (Singh & Theunissen, 2003), images (Ruderman, 1994) and surfaces (Kuroki et al., 2018) concentrating most of the information in the lower frequencies. Rippel et al. (2015) build on this observation to introduce spectral pooling which alleviates the loss of information of spatial pooling, while enabling fractional downsizing factors. Spectral pooling also preserves low frequencies without aliasing, a known weakness of spatial/temporal convnets (Zhang, 2019; Ribeiro & Schön, 2021).
  We consider an input x ∈ RH×W and strides S = (Sh, Sw) ∈ [1, H) × [1,W ). First, the DFT is computed y = F(x) ∈ CH×W and for simplicity we assume that the center of this matrix is the DC component — the zero frequency. Then, a bounding box of size b HSh c × b W Sw c crops this matrix around its center to produce ỹ ∈ Cb H Sh c×b WSw c. Finally, this output is brought back to the spatial domain with an inverse DFT: x̃ = F−1(ỹ) ∈ Rb H Sh c×b WSw c. In practice, x is typically a multichannel input (i.e. x ∈ RH×W×C) and the same cropping is applied to all channels. Moreover, since x is real-valued and thanks to Hermitian symmetry (see Section 2.1 for more details), only the positive half of the DFT coefficients are computed, which allows saving computation and memory while ensuring that the output x̃ remains real-valued.
  Unlike spatial pooling that requires integer strides, spectral pooling only requires integer output dimensions, which allows for much more fine-grained downsizing. Moreover, spectral pooling acts as a low-pass filter over the entire input, only keeping the lower frequencies i.e. the most relevant information in general and avoiding aliasing (Zhang, 2019). However, and similarly to spatial pooling, spectral pooling is differentiable with respect to its inputs but not with respect to its strides. Thus, one still needs to provide S as hyperparameters for each downsampling layer. In this case, the search space is even bigger than with spatial pooling since strides are not constrained to integer values anymore.
sections.heading: 2.4 DIFFSTRIDE
sections.text: To address the difficulty of searching stride parameters, we propose DiffStride, a novel downsampling layer that allows spectral pooling to learn its strides through backpropagation. To downsample x ∈ RH×W , DiffStride performs cropping in the Fourier domain similarly to spectral pooling. However, instead of using a fixed bounding box, DiffStride learns the box size via backpropagation. The learnable boxW is parametrized by the shape of the input, a smoothness factor R and the strides.
  We design this maskW as the outer product between two differentiable 1D masking functions (depicted in the lower right corner of Figure 1), one along the horizontal axis and one along the vertical axis. These 1D masks are directly derived from the adaptive attention span introduced by Sukhbaatar et al. (2019) to learn the attention span of self-attention models for natural language processing. Exploiting the conjugate symmetry of the coefficients, we only consider positive frequencies along the horizontal axis, while we mirror the vertical mask around frequency zero. Therefore, the two masks are defined as follows:
  maskh(Sh,H,R)(m) = min
  [ max [ 1
  R (R+
  H 2Sh − |H 2 −m|), 0
  ] , 1 ] ,m ∈ [0, H] (3)
  maskw(Sw,W,R)(n) = min
  [ max [ 1
  R (R+
  W
  2Sw + 1− n), 0
  ] , 1 ] , n ∈ [0, W
  2 + 1] (4)
  where S = (Sh, Sw) are the strides and R an hyperparameter that controls the smoothness of the mask. We build the 2D differentiable maskW as the outer product between the two 1D masks:
  W(Sh, Sw, H,W,R) = maskh(Sh,H,R) ⊗mask w (Sw,W,R) (5)
  We useW in two ways: (1) we apply it to the Fourier representation of the inputs via an elementwise product, which performs low-pass filtering (2) we crop the Fourier coefficients where the mask is zero (i.e. the output has dimensions b HSh + 2×Rc × b W Sw + 2×Rc).
  The first step is differentiable with respect to strides S, however the cropping operation is not. Therefore, we apply a stop gradient operator (Bengio et al., 2013) to the mask before cropping. This way, gradients can flow to the strides through the differentiable low-pass filtering operation, but not through the non-differentiable cropping. Finally, the cropped tensor is transformed back into the spatial domain using an inverse DFT. All these steps are summarized by Algorithm 1 and illustrated on a single channel image in the Figure 1.
  During training we constrain strides S = (Sh, Sw) to remain in [1, H) × [1,W ). When x is a multi-channel input (i.e. x ∈ RH×W×C), we learn the same strides S for all channels to ensure uniform spatial dimensions across channels. In spatial and spectral pooling, strides are typically tied along the spatial axes (i.e. Sw = Sh), which we can also do in DiffStride by sharing a single learnable stride on both dimensions. However, our experiments in Section 3 show that learning
  Algorithm 1: DiffStride layer Inputs : Input x ∈ RH×W , strides S = (Sh, Sw) ∈ [1, H)× [1,W ), smoothness factor R. Output: Downsampled output x̃ ∈ Rb H Sh +2×Rc×b WSw +2×Rc
  specific strides for the vertical and horizontal axis is beneficial, not only when processing timefrequency representations of audio, but also — more surprisingly— when classifying natural images. Adding an hyperparameter R to each downsampling layer would conflict with the goal of removing strides as hyperparameters. Thus, not only we use a single R value for all layers, but we found no significant impact of this choice and all our experiments use R = 4. While we focus on the 2D case, using a single 1D mask allows deriving DiffStride in 1D CNNs, while performing the outer product between three 1D masks allows applying DiffStride to 3D inputs.
sections.heading: 2.4.1 RESIDUAL BLOCK WITH DIFFSTRIDE
sections.text: Unlike systems that only feed outputs of the lth layer to the (l + 1)th (Krizhevsky et al., 2012), ResNets (He et al., 2016a;b) introduce skip-connections that operate in parallel to the main branch. ResNets stack two types of blocks: (1) identity blocks that maintain the input channel dimension and spatial resolution and (2) shortcut blocks that increase the output channel dimension while reducing the spatial resolution with a strided convolution (see Figure 2a). We integrate DiffStride into these shortcut blocks by replacing strided convolutions by convolutions without strides followed by DiffStride. Besides, sharing DiffStride strides between the main and residual branches ensures that their respective outputs have identical spatial dimensions and can be summed (See Figure 2b).
sections.heading: 2.4.2 REGULARIZING COMPUTATION AND MEMORY COST WITH DIFFSTRIDE
sections.text: The number of activations in a network depends on the strides and learning these parameters gives control over the space and time complexity of an architecture in a differentiable manner. This contrasts with previous work, as measures of complexity such as the number of floating-point operations (FLOPs) are typically not differentiable with respect to the parameters of a model and searching for efficient architectures is done via high-level exploration (e.g. introducing separable convolutions (Howard et al., 2017)), architecture search (Howard et al., 2019; Tan & Le, 2019) or using continuous relaxations of complexity (Paria et al., 2020).
  A standard 2D convolution with a square kernel of size k2 and C ′ output channels has a computational cost of k2×C×C ′×H×W when operating on x ∈ RH×W×C . Its memory usage— in terms of the number of activations to store— is C ′ × H ×W . Considering a fixed number of channels and kernel size, both the computational complexity and memory usage of a convolution layer are thus linear functions of its input size H ×W . This illustrates our argument made in Section 1 that
  downsampling does not only improve performance by discarding irrelevant information, but also reduces the complexity of the upper layers. More importantly, in the context of DiffStride the input size H l ×W l of layer l is determined as follows: H l ×W l = bH l−1
  Sl−1h +2×Rc × bW
  l−1
  Sl−1w +2×Rc,
  which is differentiable with respect to the strides at the previous layer Sl−1. Furthermore, it also depends on spatial dimensions at the previous layer H l−1×W l−1, which themselves are a function of Sl−2. By induction over layers, the total computational cost and memory usage are proportional to ∑l=L l=1 ∏l i=1
  1 Sih×Siw . Since in the context of DiffStride the kernel size and number of channels remain constant during training, we can directly regularize our model towards time and space efficiency by adding the following regularizer to our training loss:
  λJ((Sl)l=Ll=1 ) = λ l=L∑ l=1 l∏ i=1
  1
  Sih × Siw , (6)
  where λ is the regularization weight. In Section 3.2, we show that training on ImageNet with different values for λ allows us to trade-off accuracy for efficiency in a smooth fashion.
sections.heading: 3 EXPERIMENTS
sections.text: We evaluate DiffStride on eight classification tasks, both on audio and images. For each comparison, we keep the same architecture and replace strided convolutions by convolutions with no stride followed by DiffStride. To avoid the confounding factor of downsampling in the Fourier domain, we also compare our approach to the spectral pooling of Rippel et al. (2015), which only differs from DiffStride by the fact that its strides are not learnable.
sections.heading: 3.1 AUDIO CLASSIFICATION
sections.text: Experimental setup We perform single-task and multi-task audio classification on 5 tasks: acoustic scene classification (Heittola et al., 2018), birdsong detection (Stowell et al., 2018), musical instrumental classification and pitch estimation on the NSynth dataset (Engel et al., 2017) and speech command classification (Warden, 2018). The statistics of the datasets are summarized in Table A.1. The audio sampled at 16 kHz is decomposed into log-compressed mel-spectrograms with 64 channels, computed with a window of 25ms every 10ms.
  A 2D-CNN, based on (Tagliasacchi et al., 2019) takes these spectrograms as inputs and alternates blocks of strided convolutions along time ((3×1) kernel) and frequency ((1×3) kernel). Each strided convolution is followed by a ReLU (Glorot et al., 2011) and batch normalization (Ioffe & Szegedy, 2015). The sequence of channels dimensions is defined as (64, 128, 256, 256, 512, 512) and the strides are initialized as ((2, 2), (2, 2), (1, 1), (2, 2), (1, 1), (2, 2) for all downsampling methods. The output of the CNN passes through a global max-pooling and feeds into a single linear classification layer for single-task, and multiple classification layers for multi-task classification. As examples vary in length, we train models on random 1 s windows with ADAM (Kingma & Ba, 2015) and a learning rate of 10−4 for 1M batches, with batch size 256. Evaluation is run by splitting full sequences into 1 s non-overlapping windows and averaging the logits over windows.
  Results Table 1 summarizes the results for single-task and multi-task audio classification. In both settings, DiffStride improves over strided convolutions and spectral pooling, with strided convolutions only outperforming DiffStride for acoustic scene classification in the single task setting. Table 2 shows the strides learned by the first layer of DiffStride, which downsamples mel-spectrograms along frequency and time axes. Learning allows the strides to deviate from their initialization ((2, 2)) and to adapt to the task at hand. Converting strides to cut-off frequencies shows that the learned strides fall in a range showed by behavioral studies and direct neural recordings (Hullett et al., 2016; Flinker et al., 2019) to be necessary for e.g. speech intelligibility at 25Hz (Elliott & Theunissen, 2009). Moreover, DiffStride learns different strides for the time and frequency axes. Table A.7 shows the benefits of learning a per-dimension value rather than sharing strides. Another notable phenomenon is the per-task discrepancy on NSynth, with the pitch estimation requiring faster spectral modulations (as represented by a higher cutt-off frequency along the frequency axis). Finally, multi-task models do not converge to the mean of strides, but rather to a higher value that passes more frequencies not to negatively impact individual tasks.
sections.heading: 3.2 IMAGE CLASSIFICATION
sections.text: Experimental setup We use the ResNet-18 (He et al., 2016a) architecture, comparing the original strided convolutions (see Figure 2a) to spectral pooling and DiffStride (both as in Figure 2b). We randomly sample 6 striding configurations for the three shortcut blocks of the ResNet-18, each stride being sampled in [1, 3], with (2, 2, 2) being the configuration of the original ResNet of He et al. (2016a). The horizontal and vertical strides are initialized equally at start. These random configurations simulate cross-validation of stride configurations to: (1) showcase the sensitivity of the architecture to these hyperparameters, (2) test our hypothesis that DiffStride can benefit from learning its strides to recover from a poor initialization. On Imagenet, as inputs are bigger than CIFAR we also allow the first ResNet-18 identity block to learn its strides which are 1 by default.
  We first benchmark the three methods on the two CIFAR datasets (Krizhevsky, 2009). CIFAR10 consists of 32 × 32 images labeled in 10 classes with 6000 images per class. We use the official split, with 50,000 images for training and 10,000 images for testing. CIFAR100 uses the same images as CIFAR10, but with a more detailed labelling with 100 classes. We also compare the ResNet-18 architectures on the ImageNet dataset (Deng et al., 2009), which contains 1,000 classes. The models are trained on the official training split of the Imagenet dataset (1.28M images) and we report our results on the validation set (50k images). Here, we evaluate performance in terms of top-1 and top-5 accuracy. We train on all datasets with stochastic gradient descent (SGD) (Bottou et al., 1998) with a learning rate of 0.1, a batch size of 256 and a momentum (Qian, 1999) of 0.9. On CIFAR, we train models for 400 epochs dividing the learning rate by 2 at 200 epochs and again by 2 at 300 epoch, with a weight decay of 5.10−3. For CIFAR, we apply random cropping on the input images and left-right random flipping. On ImageNet, we train with a weight decay of 1.10−3 for 90 epochs, dividing the learning rate by 10 at epochs 30, 60 and 80. We apply random cropping on the input images as in (Szegedy et al., 2015) and left-right random flipping.
  Results We report the results on the CIFAR datasets and Imagenet in Tables 3 and 4 respectively, with the accuracy of our baseline ResNet-18 (first row, Strided Conv.) being consistent with previous work (Bianco et al., 2018). First, we observe that strides are indeed critical hyperparameters for the performance of a standard ResNet-18 on the three datasets, with the accuracy on CIFAR100 dropping from 66.8% average to 48.2% between the best and worst configurations. Remarkably,
  spectral pooling is much more robust to bad initializations than strided convolutions, even though its strides are also fixed. However, DiffStride is overall much more robust to poor choices of strides, converging consistently to a high accuracy on the three datasets, with a variance over initializations that is lower by an order of magnitude. This shows that backpropagation allows DiffStride to find a better configuration during training avoiding a cross-validation which would require 6,561 experiments for testing all combinations of strides in [1, 3] on Imagenet. Tables A.5 and A.6 confirm these observations on the EfficientNet-B0 (Tan & Le, 2019) architecture.
  Learning dynamics and equivalence classes Figure 3 illustrates the learning dynamics of DiffStride on CIFAR10. Figure 3a plots the strides as a function of the epoch for a run with the baseline (2,2,2) configuration as initialization. The strides all deviate from their initialization while converging rapidly, with the lower layers keeping more information while higher layers downsample more drastically. Interestingly, we discover equivalence classes: despite converging to the same accuracy (as reported in Table 3) the various initializations yield very diverse strides configurations at convergence, both in terms of total striding factor (defined as the product of strides, see Figure 3c) and of repartition of downsampling factors along the architecture (see Figure 3b). We obtain similar conclusions on CIFAR100 and Imagenet (see Figures A.1 and A.2). In the non-regularized case, it could seem counter-intuitive that minimizing the training loss yields positive stride updates, i.e. dropping more information through cropping. It highlights that loss optimization is a tradeoff between preserving information (no striding, no cropping) and downscaling such that the next convolution kernel accesses a wider spatial context.
  Regularizing the complexity The existence of equivalence classes suggests that DiffStride can find more computationally efficient configurations for a same accuracy. We thus train DiffStride on ImageNet using the complexity regularizer defined in Equation 6, with λ varying between 0.1 and 10, always initializing strides with the baseline ((1, 1), (2, 2), (2, 2), (2, 2)). Figure 4 plots accuracy versus computational complexity (as measured by the value of the regularization term at convergence) of DiffStride. For comparison, we also plot the models with strided convolutions with the random initializations of Table 4, showing that DiffStride finds configurations with a lower
  computational cost for the same accuracy. Some of these are quite extreme, e.g. with λ = 10 a model converges to strides ((10.51, 32.23), (1.20, 2.68), (1.20, 2.04), (1.96, 4.53)) for a 58.57% top-1 accuracy. When training a ResNet-18 with strided convolutions using the closest integer strides (i.e. ((11, 32), (1, 3), (1, 2), (2, 5))), the model converges to a 24.54% top-1 accuracy. This suggests that performing pooling in the spectral domain is more robust to aggressive downsampling, which corroborates the remarkable advantage of spectral pooling over strided convolutions when using poor strides choices in Tables 3 and 4 despite both models having fixed strides.
  Limitations Pooling in the spectral domain comes at higher computational cost than strided convolutions as it requires (1) computing a non-strided convolution and (2) a DFT and its inverse (see Table A.2). This could be alleviated by computing the convolution in the Fourier domain as an element-wise multiplication and summation over channels. Further improvements could be obtained by replacing the DFT by a real-valued counterpart, such as the Hartley transform (Zhang & Ma, 2018), which would remove the need for complex-valued operations that may be poorly optimized in deep learning frameworks. We also observe no benefits of DiffStride when training DenseNets (Huang et al., 2017), see Tables A.3 and A.4. We hypothesize that this is due to the limited number of downsampling layers, which reduces the space of stride configurations to a few, equivalent ones when sampling strides in [1; 3]. Finally, some hardware (e.g. TPUs) require a static computation graph. As DiffStride changes the spatial dimensions of intermediate representations— and thus the computation graph— between each gradient update, we currently only train on GPUs.
sections.heading: 4 CONCLUSION AND FUTURE WORK
sections.text: We introduce DiffStride the first downsampling layer with learnable strides. We show on audio and image classification that DiffStride can be used as a drop-in replacement to strided convolutions, removing the need for cross-validating strides. As we observe that our method discovers multiple equally-accurate stride configurations, we introduce a regularization term to favor the most computationally advantageous. In future work, we will extend the scope of applications of DiffStride, to e.g. 1D and 3D architectures. Moreover, learning strides by backpropagation opens new avenues in designing adaptive convolutional architectures, such as multi-scale models that would learn to operate at various scales in parallel by using independent branches with separate instances of DiffStride, or by predicting strides parameters of DiffStride on a per-example basis.
sections.heading: 5 REPRODUCIBILITY STATEMENT
sections.text: We describe DiffStride in details in the text as well as with Algorithm 1 and Figure 1. We mention all relevant hyperparameters to reproduce our experiments, as well as describe audio datasets in A.1. Moreover, we will release Tensorflow 2.0 code for training a ResNet-18 with strided convolutions, spectral pooling or DiffStride on CIFAR10 and CIFAR100, with DiffStride being implemented as a stand-alone, reusable Keras layer.
sections.heading: A APPENDIX
sections.text: Analysis of strides learned on CIFAR100 and ImageNet In Figure A.1 (Figure A.2), we show the distributions of learned strides and the global striding factor at convergence on CIFAR100 (Imagenet), starting from random stride initializations. On CIFAR100, we observe equivalence classes, i.e. model that learns various stride configurations for a same accuracy. On Imagenet, even though we also observe a significant variance of the global striding factor, models tend to downsample only in the upper layers. Striding late in the architecture comes at a higher computational cost, which furthermore justifies regularizing DiffStride to reduce complexity as shown in Section 3.2.
  Time and space complexity in practice While Figure 4 reports theoretical estimates of computational complexity based on stride configurations, both spectral pooling and DiffStride require computing a DFT and its inverse. Moreover, DiffStride requires accumulating gradients with respect to the strides during training. Table A.2 reports the duration and peak memory usage of the multitask architecture described in 3.1, for a single batch. Replacing strided convolutions with spectral pooling increases the wall time by 32% due to the DFT and inverse DFT, while the peak memory usage is almost unaffected. DiffStride furthermore increases the wall time (by 43% w.r.t strided convolutions) as the backward pass is more expensive. Similarly, it almost doubles the peak memory usage. However, in inference, DiffStride does not need to compute and store gradients w.r.t. the strides, thus the time and space complexity become identical to that of spectral pooling.
  DenseNet experiments on CIFAR We also evaluate DiffStride in DenseNet (Huang et al., 2017), especially the DenseNet-BC architecture with a depth of 121 and a growth rate of 32. The DenseNet architecture halves spatial dimensions during transition blocks. We replace the 2D average pooling in the transition blocks by spectral pooling or DiffStride. The considered architecture for DenseNet has two downsampling steps. We run a similar experiment as in 3.2 with random strides between the dense blocks on the two CIFAR datasets. We observe that initializing strides randomly does not affect the performance of the standard Densenet-BC architecture with average pooling. Consequently, DiffStride does not improve over alternatives.
  EfficientNet experiments on CIFAR We evaluate DiffStride in an EfficientNet-B0 architecture (Tan & Le, 2019), a lightweight model discovered by architecture search. This architecture has seven strided convolutions. Unlike Tan & Le (2019), we do not pre-train on ImageNet, but rather train from scratch on CIFAR, which explains the lower accuracy of the baseline. As the model has seven downsampling layers, we rescale the images from 32 × 32 to 128 × 128, and only sample strides in [1; 2]. We run a similar experiment as in 3.2 with random strides on the two CIFAR datasets. Consistently with the results obtained with a ResNet-18, spectral pooling is much more robust to poor strides than strided convolutions, with DiffStride outperforming all alternatives.
  Ablation study: learning a per-dimension stride or a shared one We perform multi-task audio classification with either learning a single stride value for each DiffStride layer, or a different one for the time and frequency axes. The overall performance across tasks is improved when learning a different stride value for each dimension (See Table A.7).