summaries: 1. The paper proposes GDWS with an error optimization strategy to replace regular convolution for enhancing both throughput (FPS) and adversarila robustness. 2. Experimental results on CIFAR-10, SVHN, and ImageNet validate that the replacing a CNN's regular conv with GDWS can achieve significantly higher FPS and preserve adversarial robustness. The paper proposes a post-training transformation of a pre-trained 2D convolutional layer into a general depth-wise separable convolution operation. The optimization primarily focuses on minimizing error and limiting computation complexity. The paper claims to achieve a higher robustness (i.e., defend adversarial input attack) and higher frame per second than existing methods. This paper proposed a method named  Generalized Depthwise-Separable (GDWS) to improve the throughput on real-life hardware while simultaneously preserving robustness. The advantage of GDWS is that does not need any additional training.  This paper has demonstrated the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets.   The paper proposes a generalization of depthwise-separable convolutions (called GDWS) where each channel can have multiple depthwise filters (instead of just one) that are subsequently combined using a pointwise convolution. Two algorithms are proposed to minimize the error/complexity of mapping general convolutions to GDWS as a post-processing step after training. Results show small degradation on natural and robust accuracy while increasing the FPS substantially. The authors propose Generalized Depthwise-Separable (GDWS) convolutions for processing images for the purpose of getting high throughput in efficiency and robustness. This is carried out by introducing channel distribution vectors in the standard 2D CNNs. The efficiency is managed by means of the singular value decomposition of the matrix in the convolutional operations after vectorizations. Experiments show that the method provides similar results for some benchmark data. 
abstractText: Despite their tremendous successes, convolutional neural networks (CNNs) incur high computational/storage costs and are vulnerable to adversarial perturbations. Recent works on robust model compression address these challenges by combining model compression techniques with adversarial training. But these methods are unable to improve throughput (frames-per-second) on real-life hardware while simultaneously preserving robustness to adversarial perturbations. To overcome this problem, we propose the method of Generalized Depthwise-Separable (GDWS) convolution – an efficient, universal, post-training approximation of a standard 2D convolution. GDWS dramatically improves the throughput of a standard pre-trained network on real-life hardware while preserving its robustness. Lastly, GDWS is scalable to large problem sizes since it operates on pre-trained models and doesn’t require any additional training. We establish the optimality of GDWS as a 2D convolution approximator and present exact algorithms for constructing optimal GDWS convolutions under complexity and error constraints. We demonstrate the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets. Our code can be found at https://github.com/hsndbk4/ GDWS.
authors.name: Hassan Dbouk
references.authors: Gabriel Bender Pieter-Jan Kindermans Barret Zoph Vijay Vasudevan Quoc Le
references.title: Understanding and simplifying one-shot architecture search
references.venue: In International Conference on Machine Learning,
references.authors: Jungwook Choi Zhuo Wang Swagath Venkataramani Pierce I-Jen Chuang Vijayalakshmi Srinivasan Kailash Gopalakrishnan
references.title: PACT: Parameterized clipping activation for quantized neural networks
references.venue: arXiv preprint arXiv:1805.06085,
references.authors: Hassan Dbouk Hetul Sanghvi Mahesh Mehendale Naresh Shanbhag
references.title: DBQ: A differentiable branch quantizer for lightweight deep neural networks
references.venue: In European Conference on Computer Vision,
references.authors: Ross Girshick
references.title: Fast R-CNN
references.venue: In Proceedings of the IEEE International Conference on Computer Vision (ICCV),
references.authors: Ian J Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: arXiv preprint arXiv:1412.6572,
references.authors: Shupeng Gui Haotao Wang Chen Yu Haichuan Yang Zhangyang Wang Ji Liu
references.title: Model compression with adversarial robustness: A unified optimization framework
references.venue: Advances in Neural Information Processing Systems,
references.authors: Minghao Guo Yuzhe Yang Rui Xu Ziwei Liu Dahua Lin
references.title: When nas meets robustness: In search of robust architectures against adversarial attacks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Yiwen Guo Anbang Yao Yurong Chen
references.title: Dynamic network surgery for efficient DNNs
references.venue: In Advances in Neural Information Processing Systems (NIPS),
references.authors: Song Han Xingyu Liu Huizi Mao Jing Pu Ardavan Pedram Mark A Horowitz William J Dally
references.title: EIE: Efficient inference engine on compressed deep neural network
references.venue: ACM SIGARCH Computer Architecture News,
references.authors: Song Han Huizi Mao William J Dally
references.title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
references.venue: arXiv preprint arXiv:1510.00149,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Andrew G Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision applications
references.venue: arXiv preprint arXiv:1704.04861,
references.authors: Gao Huang Shichen Liu Laurens Van der Maaten Kilian Q Weinberger
references.title: Condensenet: An efficient densenet using learned group convolutions
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio
references.title: Binarized neural networks
references.venue: In Advances in neural information processing systems,
references.authors: Forrest N Iandola Song Han Matthew W Moskewicz Khalid Ashraf William J Dally Kurt Keutzer
references.title: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 MB model size
references.venue: arXiv preprint arXiv:1602.07360,
references.authors: Andrew Ilyas Shibani Santurkar Logan Engstrom Brandon Tran Aleksander Madry
references.title: Adversarial examples are not bugs, they are features
references.venue: Advances in neural information processing systems,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report, Citeseer,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing
references.authors: Ji Lin Chuang Gan Song Han
references.title: Defensive quantization: When efficiency meets robustness
references.venue: In International Conference on Learning Representations,
references.authors: Hanxiao Liu Karen Simonyan Yiming Yang
references.title: DARTS: Differentiable architecture search
references.venue: In International Conference on Learning Representations,
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: In International Conference on Learning Representations,
references.authors: Pratyush Maini Eric Wong Zico Kolter
references.title: Adversarial robustness against the union of multiple perturbation models
references.venue: In International Conference on Machine Learning,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga Adam Lerer
references.title: Automatic differentiation in PyTorch
references.venue: In NIPS Autodiff Workshop,
references.authors: Vivek Ramanujan Mitchell Wortsman Aniruddha Kembhavi Ali Farhadi Mohammad Rastegari
references.title: What’s hidden in a randomly weighted neural network
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Mohammad Rastegari Vicente Ordonez Joseph Redmon Ali Farhadi
references.title: XNOR-net: Imagenet classification using binary convolutional neural networks
references.venue: In European Conference on Computer Vision,
references.authors: Esteban Real Alok Aggarwal Yanping Huang Quoc V Le
references.title: Regularized evolution for image classifier architecture search
references.venue: In Proceedings of the AAAI conference on artificial intelligence,
references.authors: Leslie Rice Eric Wong Zico Kolter
references.title: Overfitting in adversarially robust deep learning
references.venue: In International Conference on Machine Learning,
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International journal of computer vision,
references.authors: Charbel Sakr Yongjune Kim Naresh Shanbhag
references.title: Analytical guarantees on numerical precision of deep neural networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Charbel Sakr Naresh Shanbhag
references.title: An analytical method to determine minimum per-layer precision of deep neural networks
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Mark Sandler Andrew Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen
references.title: Mobilenetv2: Inverted residuals and linear bottlenecks
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Lukas Schott Jonas Rauber Matthias Bethge Wieland Brendel
references.title: Towards the first adversarially robust neural network model on MNIST
references.venue: In International Conference on Learning Representations,
references.authors: Vikash Sehwag Shiqi Wang Prateek Mittal Suman Jana
references.title: HYDRA: Pruning adversarially robust neural networks
references.venue: Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Sanchari Sen Balaraman Ravindran Anand Raghunathan
references.title: EMPIR: Ensembles of mixed precision deep networks for increased robustness against adversarial attacks
references.venue: In International Conference on Learning Representations,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.venue: arXiv preprint arXiv:1312.6199,
references.authors: Mingxing Tan Quoc Le
references.title: Efficientnet: Rethinking model scaling for convolutional neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Florian Tramèr Dan Boneh
references.title: Adversarial training and robustness for multiple perturbations
references.venue: In Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Florian Tramèr Nicholas Carlini Wieland Brendel Aleksander Madry
references.title: On adaptive attacks to adversarial example defenses
references.venue: Advances in Neural Information Processing Systems,
references.authors: Tien-Ju Yang Yu-Hsin Chen Vivienne Sze
references.title: Designing energy-efficient convolutional neural networks using energy-aware pruning
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Shaokai Ye Kaidi Xu Sijia Liu Hao Cheng Jan-Henrik Lambrechts Huan Zhang Aojun Zhou Kaisheng Ma Yanzhi Wang Xue Lin
references.title: Adversarial robustness vs. model compression, or both
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: arXiv preprint arXiv:1605.07146,
references.authors: Dongqing Zhang Jiaolong Yang Dongqiangzi Ye Gang Hua
references.title: LQ-nets: Learned quantization for highly accurate and compact deep neural networks
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Hongyang Zhang Yaodong Yu Jiantao Jiao Eric Xing Laurent El Ghaoui Michael Jordan
references.title: Theoretically principled trade-off between robustness and accuracy
references.venue: In International Conference on Machine Learning,
references.authors: Tianyun Zhang Shaokai Ye Kaiqi Zhang Jian Tang Wujie Wen Makan Fardad Yanzhi Wang
references.title: A systematic DNN weight pruning framework using alternating direction method of multipliers
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun
references.title: Shufflenet: An extremely efficient convolutional neural network for mobile devices
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V Le
references.title: Learning transferable architectures for scalable image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
sections.heading: 1 Introduction
sections.text: Nearly a decade of research after the release of AlexNet [18] in 2012, convolutional neural networks (CNNs) have unequivocally established themselves as the de facto classification algorithm for various machine learning tasks [11, 38, 4]. The tremendous success of CNNs is often attributed to their unrivaled ability to extract correlations from large volumes of data, allowing them to surpass human level accuracy on some tasks such as image classification [11].
  Today, the deployment of CNNs in safety-critical Edge applications is hindered due to their high computational costs [11, 30, 31] and their vulnerability to adversarial samples [37, 5, 16]. Traditionally, those two problems have been addressed in isolation. Recently, very few bodies of works [19, 35, 42, 6, 34, 7] have addressed the daunting task of designing both efficient and robust CNNs. A majority of these methods focus on model compression, i.e. reducing the storage requirements of CNNs. None have demonstrated their real-time benefits in hardware. For instance, Fig. 1a shows recent robust pruning works HYDRA [34] and ADMM [42] achieve high compression ratios (up to 97×) but either fail to achieve high throughput measured in frames-per-second (FPS) or compromise significantly on robustness. Furthermore, the overreliance of current robust complexity reduction techniques on adversarial training (AT) [45, 21] increases their training time significantly (Fig. 1b). This prohibits their application to complex ImageNet scale problems with stronger attack models, such as union of norm-bounded perturbations [22]. Thus, there is critical need for methods to
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  design deep nets that are both adversarially robust and achieve high throughput when mapped to real hardware.
  To address this need, we propose Generalized Depthwise-Separable (GDWS) convolutions, a universal post-training approximation of a standard 2D convolution that dramatically improves the real hardware FPS of pre-trained networks (Fig. 1a) while preserving their robust accuracy. Interestingly, we find GDWS applied to un-pruned robust networks simultaneously achieves higher FPS and higher robustness than robust pruned models obtained from current methods. This in spite of GDWS’s compression ratio being smaller than those obtained from robust pruning methods. Furthermore, GDWS easily scales to large problem sizes since it operates on pre-trained models and doesn’t require any additional training.
  Our contributions:
  1. We propose GDWS, a novel convolutional structure that can be seamlessly mapped onto off-the-shelf hardware and accelerate pre-trained CNNs significantly while maintaining robust accuracy.
  2. We show that the error-optimal and complexity-optimal GDWS approximations of any pre-trained standard 2D convolution can be obtained via greedy polynomial time algorithms, thus eliminating the need for any expensive training.
  3. We apply GDWS to a variety of networks on CIFAR-10, SVHN, and ImageNet to simultaneously achieve higher robustness and higher FPS than existing robust complexity reduction techniques, while incurring no extra training cost.
  4. We demonstrate the versatility of GDWS by using it to design efficient CNNs that are robust to union of (`∞, `2, `1) perturbation models. To the best of our knowledge, this is the first work that proposes efficient and robust networks to the union of norm-bounded perturbation models.
sections.heading: 2 Background and Related Work
sections.text: The problem of designing efficient and robust CNNs, though crucial for safety-critical Edge applications, is not yet well understood. Very few recent works have addressed this problem [19, 35, 42, 6, 34, 7]. We cluster prior works into the following categories:
  Quantization Reducing the complexity of CNNs via model quantization in the absence of any adversary is a well studied problem in the deep learning literature [30, 31, 2, 44, 14, 26, 3]. The role of quantization on adversarial robustness was studied in Defensive Quantization (DQ) [19] where it was observed that conventional post-training fixed-point quantization makes networks more vulnerable to adversarial perturbations than their full-precision counterparts. EMPIR [35] also leverages extreme model quantization (up to 2-bits) to build an ensemble of efficient and robust networks. However, [40] broke EMPIR by constructing attacks that fully leverage the model structure, i.e., adaptive attacks. In contrast, GDWS is an orthogonal complexity reduction technique that preserves the base model’s adversarial robustness and can be applied in conjunction with model quantization.
  Pruning The goal of pruning is to compress neural networks by zeroing out unimportant weights [10, 8, 46, 41]. The structured pruning method in [42] combines the alternating direction method of multipliers (ADMM) [46] for parameter pruning within the AT framework [21] to design pruned and robust networks. The flexibility of ADMM enables it to achieve a high FPS on Jetson (as seen in Fig. 1a) but suffers from a significant drop in robustness. ATMC [6] augments the ADMM framework [42] with model quantization and matrix factorization to further boost the compression ratio. On the other hand, unstructured pruning methods such as HYDRA [34] prunes models via important score optimization [25]. However, HYDRA’s high pruning ratios (> 90%) doesn’t translate into real-time FPS improvements on off-the-shelf hardware and often requires custom hardware design to fully leverage their capabilities [9]. GDWS is complementary to unstructured pruning methods, e.g., when applied to HYDRA, GDWS boosts the achievable FPS and achieves much higher robustness at iso-FPS when compared to structured (filter) pruning ADMM.
  Neural Architecture Search Resource-efficient CNNs can be designed by exploiting design intuitions such as depthwise separable (DWS) convolutions [12, 32, 13, 47, 15, 38]. While neural architecture search (NAS) [48, 27] automates the process, it requires massive compute resources, e.g., thousands of GPU hours for a single network. Differentiable NAS [20] and one-shot NAS [1] drastically reduce the cost of this search. In [7], a one-shot NAS framework [1] is combined with the AT framework [21] to search for robust network architectures, called RobNets. RobNets achieve slightly higher robustness than existing networks with less storage requirements. In this work, we show that applying GDWS to existing architectures, e.g., WideResNet-28-4, achieves significantly higher FPS than RobNet, at iso-robustness and model size.
sections.heading: 3 Generalized Depthwise-Separable Convolutions
sections.text: In this section, we introduce GDWS convolutions and develop error-optimal and complexity-optimal GDWS approximations of standard 2D convolution. These optimal approximations are then employed to construct GDWS networks from any pre-trained robust CNN built from standard 2D convolutions.
  Notation: A (C,K,M) standard 2D convolution operates on an input feature map X ∈ RC×H×W via M filters (also referred to as kernels or output channels) each consisting of C channels each of dimension K ×K to generate an output feature map Y ∈ RM×H′×W ′ . 2D Convolution as Matrix Multiplication: The M filters can be viewed as vectors {wi}Mi=1 ∈ RCK2 obtained by vectorizing the K2 elements within a channel and then across the C channels. The resulting weight matrix W ∈ RM×CK2 is constructed by stacking these filter vectors, i.e., W = [w1|w2|...|wM ]T. From an operational viewpoint, the matrix W can be used to compute the 2D convolution via Matrix Multiplication (MM) with the input matrix X = Ψ(X) ∈ RCK2×H′W ′ :
  Y = WX = WΨ(X) (1)
  where Ψ is an unrolling operator that generates all H ′W ′ input feature map slices and stacks them in matrix format. The resultant output matrix Y ∈ RM×H′W ′ can be reshaped via the operator Φ to retrieve Y = Φ(Y). The computational complexity of (1) in terms of multiply-accumulate (MAC) operations is given by:
  H ′W ′MCK2 (2)
  The reshaping operators Φ and Ψ are only for notational convenience and are computation-free.
sections.heading: 3.1 GDWS Formulation
sections.text: Definition: A (C,K,g,M) GDWS convolution is parameterized by the channel distribution vector g ∈ ZC+ in addition to the parameters (C,K,M) of a standard 2D convolution. A GDWS convolution is composed of a (C,K,g) Generalized Depthwise (GDW) convolution and a (G, 1,M) standard pointwise (PW) convolution where G = ∑ gc with c ∈ [C]1.
  A (C,K,g) GDW convolutional layer (Fig. 2) operates on an input feature map X ∈ RC×H×W by convolving the cth channel with gc ∈ Z+ depthwiseK×K filters to produce a total ofG intermediate output channels each of size H ′ ×W ′. The (G, 1,M) PW layer operates on the intermediate output feature map by convolving it with M filters of size 1 × 1, thus producing the output feature map Y ∈ RM×H′×W ′ . Relation to DWS: Setting gc = 1 ∀c ∈ [C] reduces the GDWS convolution to the standard DWS convolution popularized by [12]. Thus, GDWS generalizes DWS by allowing for more than one (gc ≥ 1) depthwise filters per channel. This simple generalization relaxes DWS’s highly constrained structure enabling accurate approximations of the 2D convolution. Thus, GDWS when applied to pre-trained models preserves its original behavior and therefore its natural and robust accuracy. Furthermore, GDWS achieves high throughput since it exploits the same hardware features that enable networks with DWS to be implemented efficiently. One might ask: why not use DWS on pre-trained models? Doing so will result in very high approximation errors. In fact, in Section 4.2, we show that applying GDWS to a pre-trained complex network such as ResNet-18 achieves better robust accuracy than MobileNet trained from scratch, while achieving similar FPS.
  GDWS Complexity: The total number of MAC operations required by GDWS convolutions is:
  H ′W ′ ( C∑ c=1 gc(K 2 +M) ) = H ′W ′G(K2 +M) (3)
  Thus, replacing standard 2D convolutions with GDWS convolutions results in a complexity reduction by a factor of G(K
  2+M) CK2M .
sections.heading: 3.2 Properties of GDWS Convolutions
sections.text: We present properties of the GDWS weight matrix W that will be vital for developing the optimal approximation procedures.
  Property 1. The weight matrix W ∈ RM×CK2 of a (C,K,g,M) GDWS convolution can be expressed as:
  W = WPW D (4)
  where WP ∈ RM×G and WD ∈ RG×CK 2
  are the weight matrices of the PW and GDW convolutions, respectively.
  Property 1 implies that any GDWS convolution has an equivalent 2D convolution whose weight matrix is the product of WP and WD, where WP is a regular convolution weight matrix with K = 1 and WD has the following property:
  1we use the notation [C] = {1, 2, ..., C} for brevity.
  Property 2. The weight matrix WD ∈ RG×CK 2
  of a (C,K,g) GDW convolution has a blockdiagonal structure. Specifically, WD is a concatenation of C sub-matrices where each sub-matrix WD,c ∈ RG×K 2 has at most gc non-zero consecutive rows, starting at row index 1 + ∑c−1 k=1 gk as shown in Fig. 3.
  This structure is due to the fact that input channels are convolved independently with at most gc depthwise filters per channel. Finally, let W = [W1|W2|...|WC ] be represented as the concatenation of C sub-matrices, then combining Properties 1 & 2 establishes the following lemma:
  Lemma 1. The weight matrix of a (C,K,g,M) GDWS convolution can be expressed as the concatenation of C sub-matrices Wc where rank(Wc) ≤ min(gc,K2) ∀c ∈ [C].
  The reason for this is that each sub-matrix Wc can be expressed as the sum of gc rank 1 matrices of size M ×K2. A detailed proof of Lemma 1 can be found in the Appendix. A major implication of Lemma 1 is that any 2D standard convolution is equivalent to a GDWS convolution with gc = K2 ∀c2. Hence, in the rest of this paper we will assume gc ≤ K2 when we approximate 2D convolutions with GDWS.
sections.heading: 3.3 Optimal GDWS Approximation Methods
sections.text: We wish to approximate a standard 2D convolution with weight matrix W ∈ RM×CK2 with a GDWS convolution with weight matrix Q ∈ RM×CK2 to minimize the weighted approximation error defined as:
  e(W,Q,α) = √√√√ C∑ c=1 αc||Wc −Qc||2F (5)
  where ||.||F denotes the Frobenius norm of a matrix and α ∈ RC+ is a vector of positive weights. Setting αc = 1 ∀c simplifies (5) to the Frobenius norm of the error matrix W −Q. Furthermore, from (3), one can upper bound the complexity of the GDWS approximation Q via an upper bound on G = ∑ gc where the gc’s are obtained from Lemma 1.
  Based on the GDWS properties and Lemma 1, we state the following error-optimal approximation theorem:
  Theorem 1. Given a (C,K,M) standard 2D convolution with weight matrix W, the (C,K,g,M) GDWS approximation with weight matrix Ŵ that minimizes the error in (5) subject to ∑ gc = G ≤ γ (for some γ ∈ Z+), can be obtained in polynomial time via Algorithm 1.
  That is: Ŵ = arg min
  Q: G≤γ e(W,Q,α) (6)
  can be solved for any weight error vector α ∈ RC+ in polynomial time. While Theorem 1 shows that the optimal GDWS approximation under a complexity constraint can be solved efficiently, a similar result can be obtained for the reverse setting shown next.
  2Typical CNNs satisfy K2 < M , e.g., K = 3 and M ≥ 16
  Theorem 2. Given a (C,K,M) standard 2D convolution with weight matrix W, the (C,K,g,M) GDWS approximation with weight matrix Ŵ that minimizes the complexity in (3) subject to e(W,Q,α) ≤ β (for some β ≥ 0), can be constructed in polynomial time via Algorithm 2.
  That is:
  Ŵ = arg min Q: e(W,Q,α)≤β C∑ c=1 gc (7)
  can be solved for any weight error vector α ∈ RC+ in polynomial time. Proofs of Theorems 1 & 2 can be found in the Appendix.
  Algorithm 1: (MEGO) Minimum Error Complexity-constrained GDWS Optimal Approximation Input: A (C,K,M) convolution W,
  weight error vector α, and constraint γ ∈ Z+.
  Output: A (C,K,g,M) GDWS convolution Ŵ, satisfying∑ gc ≤ γ.
  1 Compute SVDs of Wc = ∑rc i=1 σi,cui,cv T i,c 2 Initialize g = 0 3 while ∑ gc < γ do 4 c′ = arg maxc αcσ 2 gc+1,c // gc < rc 5 gc′ ← gc′ + 1 6 Compute Ŵc via truncated SVD of Wc
  with rank gc: Ŵc = ∑gc i=1 σi,cui,cv T i,c
  7 Construct Ŵ = [Ŵ1|...|ŴC ]
  Algorithm 2: (LEGO) Least Complex Errorconstrained GDWS Optimal Approximation Input: A (C,K,M) convolution W,
  weight error vector α, and constraint β ≥ 0.
  Output: A (C,K,g,M) GDWS convolution Ŵ, satisfying e ≤ β.
  1 Compute SVDs of Wc = ∑rc i=1 σi,cui,cv T i,c 2 Initialize gc = rc, b = 0, c′ = arg minc αcσ 2 rc,c, h = αc′σ 2 rc′ ,c ′ 3 while b+ h < β do 4 b← b+ h and gc′ ← gc′ − 1 5 c′ = arg minc αcσ 2 gc,c // gc > 1 6 h = αc′σ 2 rc′ ,c ′
  7 Compute Ŵc via truncated SVD of Wc with rank gc: Ŵc = ∑gc i=1 σi,cui,cv T i,c 8 Construct Ŵ = [Ŵ1|...|ŴC ]
sections.heading: 3.4 Constructing GDWS Networks
sections.text: When confronted with a CNN with L convolutional layers, the question arises: How to assign resources (in terms of complexity) amongst the L layers such that the robustness of the CNN is minimally compromised? To answer this question, we compute per-layer weight error vectors αl, such that the computed error in (5) weighs how different sub-matrices affect the final output of the CNN.
  Let f : RD → RN be a pre-trained CNN for an N -way classification problem with L convolutional layers parameterizd by weight matrices W(l) ∈ RMl×ClK2l . The CNN f operates on aD-dimensional input vector x to produce a vector z = f(x) of soft outputs or logits. Denote by nx ∈ [N ] the predicted class label associated with x, and define δx,j = zj − znx to be the soft output differences ∀j ∈ [N ] \ {nx}. Inspired by [30, 31], we propose a simple yet effective method for computing the per-layer weight error vectors as follows:
  αc,l = 1
  MlK2l E  N∑ j=1 j 6=nx ||D(c,l)x,j ||2F 2δ2x,j  ∀l ∈ [L], ∀c ∈ [Cl] (8) where D(c,l)x,j ∈ RMl×K 2 l is the derivative of δx,j w.r.t. the sub-matrix W (l) c . The expectation is taken over the input vector x. Equation (8) can be thought of as the expected noise gain from a particular channel in a particular layer to the network output required to flip its decision. The Appendix provides a detailed rationale underlying (8).
  Computation of (8) can be simplified by obtaining an estimate of the mean over a small batch of inputs sampled from the training set and by leveraging software frameworks such as PyTorch
  [24] that automatically take care of computing D(c,l)x,j . Algorithm 3 summarizes the steps required to approximate any pre-trained CNN with an equivalent CNN utilizing GDWS convolutions. Unless specified otherwise, all the results in this paper are obtained via Algorithm 3.
  Algorithm 3: Constructing GDWS networks Input: CNN f with convolutional layers {W(l)}, {αl} computed via (8), and constraint β ≥ 0. Output: CNN f̂ with GDWS convolutions {Ŵ(l)}
  1 f̂ = f // Initialize f̂ 2 for l ∈ {1, ..., L} do 3 Ŵ(l) = LEGO(W(l),αl, β) // solve via Algorithm 2 4 Decompose Ŵ(l) into GDW and PW convolutions via Property 1 and Lemma 1 5 Replace the lth convolution layer in f̂ with GDW and PW convolutions
sections.heading: 4 Experiments
sections.heading: 4.1 Evaluation Setup
sections.text: We measure the throughput in FPS by mapping the networks onto an NVIDIA Jetson Xavier via native PyTorch [24] commands. We experiment with VGG-16 [36], ResNet-183 [11], ResNet-50, and WideResNet-28-4 [43] network architectures, and report both natural accuracy (Anat) and robust accuracy (Arob). Following standard procedure, we report Arob against `∞ bounded perturbations generated via PGD [21] with standard attack strengths: = 8/255 with PGD-100 for both CIFAR-10 [17] and SVHN [23] datasets, and = 4/255 with PGD-50 for the ImageNet [29] dataset. Section 4.3 studies union of multiple perturbation models (`∞, `2, `1). In the absence of publicly released pretrained models, we establish strong baselines using AT [21] following the approach of [28] which utilizes early stopping to avoid robust over-fitting. Details on the training/evaluation setup can be found in the Appendix.
sections.heading: 4.2 Results
sections.text: Ablation Study: We first show the effectiveness of GDWS on the CIFAR-10 datasets using four network architectures. Table 1 summarizesAnat andArob as well as FPS and model size. It is clear that GDWS networks preserve robustness as bothAnat andArob are always within∼1% of their respective baselines. The striking conclusion is that in spite of GDWS offering modest reductions in model size, it drastically improves the FPS of the base network across diverse architectures. For instance, a ResNet-18 utilizing GDWS convolutions is able to run at 104 FPS compared to the baseline’s 28 FPS (>250% improvement) without additional training and without compromising on robust accuracy. In the Appendix, we explore the benefits of applying GDWS using both Algorithms 1 & 2, provide more detailed results on CIFAR-10 and show that similar gains are observed with SVHN dataset.
  3For CIFAR-10 and SVHN, we use the standard pre-activation version of ResNets.
  GDWS vs. RobNet: In Table 1, we also compare GDWS networks (obtained from standard networks) with a publicly available pre-trained RobNet model, the robust network architecture designed via the NAS framework in [7]. Note that RobNet utilizes DWS convolutions which precludes the use of GDWS. However, despite the efficiency of DWS convolutions in RobNet, its irregular cell structure leads to extremely poor mapping on the Jetson as seen by its low 5 FPS. For reference, a standard WideResNet-28-4 (WRN-28-4) runs at 17 FPS with similar robustness and model size. Applying GDWS to the WideResNet-28-4 further increases the throughput to 65 FPS which is a 1200% improvement compared to RobNet while maintaining robustness. This further supports our assertion that model compression alone does not lead to enhanced performance on real hardware.
  GDWS vs. Lightweight Networks: A natural question that might arise from this work: why not train lightweight networks utilizing DWS convolutions from scratch instead of approximating pre-traind complex networks with GDWS? In Table 2, we compare the performance of GDWS networks (obtained from Table 1) vs. standard lightweight networks: MobileNetV1 [12], MobileNetV2 [32], and ResNet-20 [11], as well as a DWS-version of the standard ResNet18 trained from scratch on the CIFAR-
  10 dataset. We find that applying GDWS to a pre-trained complex network such as ResNet-18 achieves better Anat and Arob than all lightweight networks, while achieving DWS-like FPS and requiring no extra training despite offering modest reductions in model size. The only benefit of using lightweight networks is the much smaller model size compared to GDWS networks.
  GDWS vs. Structured Pruning: In Table 3, we compare GDWS with the robust structured pruning method ADMM [42] on CIFAR-10, using two networks: VGG-16 and ResNet-18. Due to the lack of publicly available pre-trained models, we use their released code to reproduce both the AT baselines, and the corresponding pruned models at different pruning ratios. The nature of structured pruning allows ADMM pruned networks (p ≥ 50%) to achieve both high compression ratios and significant improvement in FPS over their un-pruned baselines but at the expense of robust-
  ness and accuracy. For instance, a ResNet-18 with 75% of its channels pruned results in a massive 7% (4%) drop in Anat (Arob) compared to the baseline even though it achieves a 160% improvement in FPS. In contrast, a post-training application of GDWS to the same ResNet-18 baseline results in a massive 275% improvement in FPS while preserving both Anat and Arob within 1% of their baseline values. Thus, despite achieving modest compression ratios compared to ADMM, GDWS achieves comparable improvements in FPS without compromising robustness.
  GDWS vs. Unstructured Pruning: We compare GDWS with HYDRA [34] which is an unstructured robust pruning method, on both CIFAR-10 and ImageNet datasets. We use the publicly released HYDRA models as well as their AT baselines, and apply GDWS to both the un-pruned and pruned models. Table 4 summarizes the robustness and FPS of HYDRA and GDWS networks on CIFAR-10. HYDRA pruned models have arbitrarily sparse weight matrices that cannot be leveraged by off-theshelf hardware platforms immediately. Instead, we rely on the extremely high sparsity (99%) of these matrices to emulate channel pruning whereby channels are discarded only if all filter weights are zero.
  This explains why, despite their high compression ratios, HYDRA models do not achieve significant improvements in FPS compared to their baselines.
  For instance, a 99% HYDRA pruned WideResNet model achieves a massive∼100× compression ratio and improves the FPS from 17 to 28, but suffers from a large ∼10% drop in both Anat and Arob. In contrast, GDWS applied to the same un-pruned baseline preserves robustness and achieves significantly better throughput of 68 FPS, even though the model size reduction is negligible. Interestingly, we find that applying GDWS directly to HYDRA pruned models results in networks with high compression ratios with no robustness degradation and massive improvements in FPS compared to the pruned baseline. For example, applying GDWS to the same 99% HYDRA pruned WideResNet achieves a ∼20× compression ratio and improves the throughput from 28 FPS to 68 FPS while preserving Anat and Arob of the pruned baseline. This
  synergy between HYDRA and GDWS is due to the fact that highly sparse convolution weight matrices are more likely to have low-rank and sparse sub-matrices. This implies that, using Lemma 1, sparse convolutions can be transformed to sparse GDWS versions with negligible approximation error. We explore this synergy in detail in the Appendix. Table 5 shows that GDWS benefits also show up in ImageNet using ResNet-50.
sections.heading: 4.3 Defending against Union of Perturbation Models
sections.text: Recent work has shown that adversarial training with a single perturbation model leads to classifiers vulnerable to the union of (`∞, `2, `1)-bounded perturbations [33, 39, 22]. The method of multi steepest descent (MSD) [22] achieves state-of-the-art union robust accuracy (AUrob) against the union of (`∞, `2, `1)-bounded perturbations. We demonstrate the versatility of GDWS by applying it to a publicly available [22] robust pre(MSD)-trained ResNet-18 model on CIFAR-10. Following the setup in [22], all attacks were run on a subset of the first 1000 test images with 10 random restarts with the following attack configurations: ∞ = 0.03 with PGD-100 , 2 = 0.5 with PGD-500, and 1 = 12 with PGD-100. Table 6 shows that applying GDWS with β = 0.01 to the pre-trained ResNet-18 incurs a negligible (<∼1%) drop in Anat and AUrob while improving the throughput from 28 FPS to 101 FPS (> 250% improvement).
sections.heading: 5 Discussion
sections.text: We have established that the proposed GDWS convolutions are universal and efficient approximations of standard 2D convolutions that are able to accelerate any pre-trained CNN utilizing standard 2D convolution while preserving its accuracy and robustness. This facilitates the deployment of CNNs in safety critical edge applications where real-time decision making is crucial and robustness cannot be compromised. One limitation of this work is that GDWS alone does not achieve high compression ratios compared to pruning. Combining unstructured pruning with GDWS alleviates this problem to some extent. Furthermore, GDWS cannot be applied to CNNs utilizing DWS convolutions, such as RobNet for instance. An interesting question is to explore the possibility of training GDWS-structured networks from scratch. Another possible direction is fine-tuning post GDWS approximation to recover robustness, which we explore in the Appendix.
  In summary, a GDWS approximated network inherits all the properties, e.g., accuracy, robustness, compression and others, of the baseline CNN while significantly enhancing its throughput (FPS) on real hardware. Therefore, the societal impact of GDWS approximated networks are also inherited from those of the baseline CNNs.
sections.heading: Acknowledgments
sections.text: This work was supported by the Center for Brain-Inspired Computing (C-BRIC) and the Artificial Intelligence Hardware (AIHW) program funded by the Semiconductor Research Corporation (SRC) and the Defense Advanced Research Projects Agency (DARPA).
title: Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks