summaries: In this papers, the authors learn a Once-for-all net. This starts as a big neural network which is trained normally (albeit with input images of different resolutions). It is then fine-tuned while sampling sub-networks with progressively smaller kernels, then lower depth, then width (while still sampling larger networks occasionally, as it reads). This results in a network from which one can extract sub-networks for various resource constraints (latency, memory etc.) that perform well without a need for retraining. This paper tries to tackle the problem of searching best architectures for specialized resource constraint deployment scenarios. The authors basically take a two-step approach: First train a large network including all the small networks with weight sharing and some specially designed trick (e.g., progressive shrinking). Second, use prediction based NAS method to learn the performance/inference prediction module, from which the good sub architecture corresponding to a particular scenario is obtained. The experiments show that the proposed method is promising.
abstractText: We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing CO2 emission as much as 5 cars’ lifetime Strubell et al. (2019)) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of subnetworks (> 10) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5× faster than MobileNetV3, 2.6× faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and CO2 emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting (<600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.
authors.name: Han Cai
authors.name: Chuang Gan
authors.name: Tianzhe Wang
authors.name: Zhekai Zhang
authors.name: Song Han
references.authors: Anubhav Ashok Nicholas Rhinehart Fares Beainy Kris M Kitani
references.title: N2n learning: Network to network compression via policy gradient reinforcement learning
references.venue: In ICLR,
references.authors: Han Cai Tianyao Chen Weinan Zhang Yong Yu Jun Wang
references.title: Efficient architecture search by network transformation
references.venue: In AAAI,
references.authors: Han Cai Jiacheng Yang Weinan Zhang Song Han Yong Yu
references.title: Path-level network transformation for efficient architecture search
references.venue: In ICML,
references.authors: Han Cai Ligeng Zhu Song Han
references.title: ProxylessNAS: Direct neural architecture search on target task and hardware
references.venue: In ICLR,
references.authors: Brian Cheung Alex Terekhov Yubei Chen Pulkit Agrawal Bruno Olshausen
references.title: Superposition of many models into one
references.venue: In NeurIPS,
references.authors: Matthieu Courbariaux Yoshua Bengio Jean-Pierre David
references.title: Binaryconnect: Training deep neural networks with binary weights during propagations
references.venue: In NeurIPS,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: In CVPR,
references.authors: Zichao Guo Xiangyu Zhang Haoyuan Mu Wen Heng Zechun Liu Yichen Wei Jian Sun
references.title: Single path one-shot neural architecture search with uniform sampling
references.venue: arXiv preprint arXiv:1904.00420,
references.authors: Song Han Jeff Pool John Tran William Dally
references.title: Learning both weights and connections for efficient neural network
references.venue: In NeurIPS,
references.authors: Song Han Huizi Mao William J Dally
references.title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
references.venue: In ICLR,
references.authors: Cong Hao Xiaofan Zhang Yuhong Li Sitao Huang Jinjun Xiong Kyle Rupnow Wen-mei Hwu Deming Chen
references.title: Fpga/dnn co-design: An efficient design methodology for 1ot intelligence on the edge
references.venue: 56th ACM/IEEE Design Automation Conference (DAC),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In CVPR,
references.authors: Yihui He Ji Lin Zhijian Liu Hanrui Wang Li-Jia Li Song Han
references.title: Amc: Automl for model compression and acceleration on mobile devices
references.venue: In ECCV,
references.authors: Geoffrey Hinton Oriol Vinyals Jeff Dean
references.title: Distilling the knowledge in a neural network
references.venue: arXiv preprint arXiv:1503.02531,
references.authors: Andrew Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen Mingxing Tan Weijun Wang Yukun Zhu Ruoming Pang Vijay Vasudevan
references.title: Searching for mobilenetv3
references.venue: In ICCV 2019,
references.authors: Andrew G Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision applications
references.venue: arXiv preprint arXiv:1704.04861,
references.authors: Gao Huang Zhuang Liu Laurens Van Der Maaten Kilian Q Weinberger
references.title: Densely connected convolutional networks
references.venue: In CVPR,
references.authors: Gao Huang Danlu Chen Tianhong Li Felix Wu Laurens van der Maaten Kilian Q Weinberger
references.title: Multi-scale dense networks for resource efficient image classification
references.venue: In ICLR,
references.authors: Forrest N Iandola Song Han Matthew W Moskewicz Khalid Ashraf William J Dally Kurt Keutzer
references.title: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size
references.venue: arXiv preprint arXiv:1602.07360,
references.authors: Weiwen Jiang Lei Yang Edwin Sha Qingfeng Zhuge Shouzhen Gu Yiyu Shi Jingtong Hu
references.title: Hardware/software co-exploration of neural architectures
references.venue: arXiv preprint arXiv:1907.04650,
references.authors: Weiwen Jiang Xinyi Zhang Edwin H-M Sha Lei Yang Qingfeng Zhuge Yiyu Shi Jingtong Hu
references.title: Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search
references.venue: In Proceedings of the 56th Annual Design Automation Conference
references.authors: Jason Kuen Xiangfei Kong Zhe Lin Gang Wang Jianxiong Yin Simon See Yap-Peng Tan
references.title: Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks
references.venue: In CVPR,
references.authors: Ji Lin Yongming Rao Jiwen Lu Jie Zhou
references.title: Runtime neural pruning
references.venue: In NeurIPS,
references.authors: Chenxi Liu Barret Zoph Maxim Neumann Jonathon Shlens Wei Hua Li-Jia Li Li Fei-Fei Alan Yuille Jonathan Huang Kevin Murphy
references.title: Progressive neural architecture search
references.venue: In ECCV,
references.authors: Hanxiao Liu Karen Simonyan Yiming Yang
references.title: Darts: Differentiable architecture search
references.venue: In ICLR,
references.authors: Lanlan Liu Jia Deng
references.title: Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution
references.venue: In AAAI,
references.authors: Zhuang Liu Jianguo Li Zhiqiang Shen Gao Huang Shoumeng Yan Changshui Zhang
references.title: Learning efficient convolutional networks through network slimming
references.venue: In ICCV,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Sgdr: Stochastic gradient descent with warm restarts
references.venue: arXiv preprint arXiv:1608.03983,
references.authors: Ningning Ma Xiangyu Zhang Hai-Tao Zheng Jian Sun
references.title: Shufflenet v2: Practical guidelines for efficient cnn architecture design
references.venue: In ECCV,
references.authors: Esteban Real Alok Aggarwal Yanping Huang Quoc V Le
references.title: Regularized evolution for image classifier architecture search
references.venue: In AAAI,
references.authors: Mark Sandler Andrew Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen
references.title: Mobilenetv2: Inverted residuals and linear bottlenecks
references.venue: In CVPR, 2018
references.authors: Emma Strubell Ananya Ganesh Andrew McCallum
references.title: Energy and policy considerations for deep learning in nlp
references.venue: In ACL,
references.authors: Mingxing Tan Bo Chen Ruoming Pang Vijay Vasudevan Mark Sandler Andrew Howard Quoc V Le
references.title: Mnasnet: Platform-aware neural architecture search for mobile
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Xin Wang Fisher Yu Zi-Yi Dou Trevor Darrell Joseph E Gonzalez
references.title: Skipnet: Learning dynamic routing in convolutional networks
references.venue: In ECCV,
references.authors: Samuel Williams Andrew Waterman David Patterson
references.title: Roofline: An insightful visual performance model for floating-point programs and multicore architectures
references.venue: Technical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States),
references.authors: Bichen Wu Xiaoliang Dai Peizhao Zhang Yanghan Wang Fei Sun Yiming Wu Yuandong Tian Peter Vajda Yangqing Jia Kurt Keutzer
references.title: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
references.venue: In CVPR,
references.authors: Zuxuan Wu Tushar Nagarajan Abhishek Kumar Steven Rennie Larry S Davis Kristen Grauman Rogerio Feris
references.title: Blockdrop: Dynamic inference paths in residual networks
references.venue: In CVPR,
references.authors: Jiahui Yu Thomas Huang
references.title: Autoslim: Towards one-shot architecture search for channel numbers
references.venue: arXiv preprint arXiv:1903.11728,
references.authors: Jiahui Yu Thomas Huang
references.title: Universally slimmable networks and improved training techniques
references.venue: In ICCV,
references.authors: Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun
references.title: Shufflenet: An extremely efficient convolutional neural network for mobile devices
references.venue: In CVPR,
references.authors: Chenzhuo Zhu Song Han Huizi Mao William J Dally
references.title: Trained ternary quantization
references.venue: In ICLR,
references.authors: Barret Zoph Quoc V Le
references.title: Neural architecture search with reinforcement learning
references.venue: In ICLR,
references.authors: Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V Le
references.title: Learning transferable architectures for scalable image recognition
references.venue: In CVPR,
sections.heading: 1 INTRODUCTION
sections.text: Deep Neural Networks (DNNs) deliver state-of-the-art accuracy in many machine learning applications. However, the explosive growth in model size and computation cost gives rise to new challenges on how to efficiently deploy these deep learning models on diverse hardware platforms, since they have to meet different hardware efficiency constraints (e.g., latency, energy). For instance, one mobile application on App Store has to support a diverse range of hardware devices, from a high-end Samsung Note10 with a dedicated neural network accelerator to a 5-year-old Samsung S6 with a much slower processor. With different hardware resources (e.g., on-chip memory size, #arithmetic units), the optimal neural network architecture varies significantly. Even running on the same hardware, under different battery conditions or workloads, the best model architecture also differs a lot.
  Given different hardware platforms and efficiency constraints (defined as deployment scenarios), researchers either design compact models specialized for mobile (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2018) or accelerate the existing models by compression (Han et al., 2016; He et al., 2018) for efficient deployment. However, designing specialized DNNs for every scenario is engineer-expensive and computationally expensive, either with human-based methods or NAS. Since such methods need to repeat the network design process and retrain the designed network from scratch for each case. Their total cost grows linearly as the number of deployment scenarios increases, which will result in excessive energy consumption and CO2 emission (Strubell et al., 2019). It makes them unable to handle the vast amount of hardware devices (23.14 billion IoT devices till
  Published as a conference paper at ICLR 2020
  Number of Deployment Scenarios 0 20 40 60 80
  16x~1300x reduction
  direct deploy (no retrain)
  train a once-for-all network
  specialized sub-nets
  cpu F P
  G A
  Different Hardware / Constraint
  De sig
  n Co
  st
  Previous: O(N) design cost
  Ours: O(1) design cost
  Mobile AI Tiny AI (AIoT)Cloud AI MCU
  1
  20181) and highly dynamic deployment environments (different battery conditions, different latency requirements, etc.).
  This paper introduces a new solution to tackle this challenge – designing a once-for-all network that can be directly deployed under diverse architectural configurations, amortizing the training cost. The inference is performed by selecting only part of the nce-for-all network. It flexibly supports different depths, widths, kernel sizes, and resolutions without retraining. A simple example of Once-for-All (OFA) is illustrated in Figure 1 (left). Specifically, we decouple the model training stage and the neural architecture search stage. In the model training stage, we focus on improving the accuracy of all sub-networks that are derived by selecting different parts of the once-for-all network. In the model specialization stage, we sample a subset of sub-networks to train an accuracy predictor and latency predictors. Given the target hardware and constraint, a predictor-guided architecture search (Liu et al., 2018) is conducted to get a specialized sub-network, and the cost is negligible. As such, we reduce the total cost of specialized neural network design from O(N) to O(1) (Figure 1 middle).
  However, training the once-for-all network is a non-trivial task, since it requires joint optimization of the weights to maintain the accuracy of a large number of sub-networks (more than 1019 in our experiments). It is computationally prohibitive to enumerate all sub-networks to get the exact gradient in each update step, while randomly sampling a few sub-networks in each step will lead to significant accuracy drops. The challenge is that different sub-networks are interfering with each other, making the training process of the whole once-for-all network inefficient. To address this challenge, we propose a progressive shrinking algorithm for training the once-for-all network. Instead of directly optimizing the once-for-all network from scratch, we propose to first train the largest neural network with maximum depth, width, and kernel size, then progressively fine-tune the once-for-all network to support smaller sub-networks that share weights with the larger ones. As such, it provides better initialization by selecting the most important weights of larger sub-networks, and the opportunity to distill smaller sub-networks, which greatly improves the training efficiency. From this perspective, progressive shrinking can be viewed as a generalized network pruning method that shrinks multiple dimensions (depth, width, kernel size, and resolution) of the full network rather than only the width dimension. Besides, it targets on maintaining the accuracy of all sub-networks rather than a single pruned network.
  We extensively evaluated the effectiveness of OFA on ImageNet with many hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) and efficiency constraints. Under all deployment scenarios, OFA consistently improves the ImageNet accuracy by a significant margin compared to SOTA hardware-aware NAS methods while saving the GPU hours, dollars, and CO2 emission by orders of magnitude. On the ImageNet mobile setting (less than 600M MACs), OFA achieves a new SOTA 80.0% top1 accuracy with 595M MACs (Figure 2). To the best of our knowledge, this is the first time that the SOTA ImageNet top1 accuracy reaches 80% under the mobile setting.
  1https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide/
  Published as a conference paper at ICLR 2020
  1
sections.heading: 2 RELATED WORK
sections.text: Efficient Deep Learning. Many efficient neural network architectures are proposed to improve the hardware efficiency, such as SqueezeNet (Iandola et al., 2016), MobileNets (Howard et al., 2017; Sandler et al., 2018), ShuffleNets (Ma et al., 2018; Zhang et al., 2018), etc. Orthogonal to architecting efficient neural networks, model compression (Han et al., 2016) is another very effective technique for efficient deep learning, including network pruning that removes redundant units (Han et al., 2015) or redundant channels (He et al., 2018; Liu et al., 2017), and quantization that reduces the bit width for the weights and activations (Han et al., 2016; Courbariaux et al., 2015; Zhu et al., 2017).
  Neural Architecture Search. Neural architecture search (NAS) focuses on automating the architecture design process (Zoph & Le, 2017; Zoph et al., 2018; Real et al., 2019; Cai et al., 2018a; Liu et al., 2019). Early NAS methods (Zoph et al., 2018; Real et al., 2019; Cai et al., 2018b) search for highaccuracy architectures without taking hardware efficiency into consideration. Therefore, the produced architectures (e.g., NASNet, AmoebaNet) are not efficient for inference. Recent hardware-aware NAS methods (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019) directly incorporate the hardware feedback into architecture search. Hardware-DNN co-design techniques (Jiang et al., 2019b;a; Hao et al., 2019) jointly optimize neural network architectures and hardware architectures. As a result, they can improve inference efficiency. However, given new inference hardware platforms, these methods need to repeat the architecture search process and retrain the model, leading to prohibitive GPU hours, dollars, and CO2 emission. They are not scalable to a large number of deployment scenarios. The individually trained models do not share any weight, leading to large total model size and high downloading bandwidth.
  Dynamic Neural Networks. To improve the efficiency of a given neural network, some work explored skipping part of the model based on the input image. For example, Wu et al. (2018); Liu & Deng (2018); Wang et al. (2018) learn a controller or gating modules to adaptively drop layers; Huang et al. (2018) introduce early-exit branches in the computation graph; Lin et al. (2017) adaptively prune channels based on the input feature map; Kuen et al. (2018) introduce stochastic downsampling point to reduce the feature map size adaptively. Recently, Slimmable Nets (Yu et al., 2019; Yu & Huang, 2019b) propose to train a model to support multiple width multipliers (e.g., 4 different global width multipliers), building upon existing human-designed neural networks (e.g., MobileNetV2 0.35, 0.5, 0.75, 1.0). Such methods can adaptively fit different efficiency constraints at runtime, however, still inherit a pre-designed neural network (e.g., MobileNet-v2), which limits the degree of flexibility (e.g., only width multiplier can adapt) and the ability in handling new deployment scenarios where the pre-designed neural network is not optimal. In this work, in contrast, we enable a much more diverse architecture space (depth, width, kernel size, and resolution) and a significantly larger number of architectural settings (1019 v.s. 4 (Yu et al., 2019)). Thanks to the diversity and the large design
  Published as a conference paper at ICLR 2020
  39
  train with full width
  channel importance
  0.02
  0.15
  0.85
  0.63
  channel sorting
  progressively shrink the width
  channel importance
  0.82
  0.11
  0.46 reorg.
  reorg.
  progressively shrink the width
  channel sorting
  O1 O2
  O3
  O1
  O2
  O1
  Gradually shrink the width Keep the most important channels when shrinking via channel sorting
  Progressive Shrinking
  space, we can derive new specialized neural networks for many different deployment scenarios rather than working on top of an existing neural network that limits the optimization headroom. However, it is more challenging to train the network to achieve this flexibility, which motivates us to design the progressive shrinking algorithm to tackle this challenge.
sections.heading: 3 METHOD
sections.heading: 3.1 PROBLEM FORMALIZATION
sections.text: Assuming the weights of the once-for-all network as Wo and the architectural configurations as {archi}, we then can formalize the problem as
  min Wo ∑ archi Lval ( C(Wo, archi) ) , (1)
  where C(Wo, archi) denotes a selection scheme that selects part of the model from the once-for-all network Wo to form a sub-network with architectural configuration archi. The overall training objective is to optimize Wo to make each supported sub-network maintain the same level of accuracy as independently training a network with the same architectural configuration.
sections.heading: 3.2 ARCHITECTURE SPACE
sections.text: Our once-for-all network provides one model but supports many sub-networks of different sizes, covering four important dimensions of the convolutional neural networks (CNNs) architectures, i.e., depth, width, kernel size, and resolution. Following the common practice of many CNN models (He et al., 2016; Sandler et al., 2018; Huang et al., 2017), we divide a CNN model into a sequence of units with gradually reduced feature map size and increased channel numbers. Each unit consists of a sequence of layers where only the first layer has stride 2 if the feature map size decreases (Sandler et al., 2018). All the other layers in the units have stride 1.
  We allow each unit to use arbitrary numbers of layers (denoted as elastic depth); For each layer, we allow to use arbitrary numbers of channels (denoted as elastic width) and arbitrary kernel sizes (denoted as elastic kernel size). In addition, we also allow the CNN model to take arbitrary input image sizes (denoted as elastic resolution). For example, in our experiments, the input image size ranges from 128 to 224 with a stride 4; the depth of each unit is chosen from {2, 3, 4}; the width expansion ratio in each layer is chosen from {3, 4, 6}; the kernel size is chosen from {3, 5, 7}. Therefore, with 5 units, we have roughly ((3× 3)2 + (3× 3)3 + (3× 3)4)5 ≈ 2× 1019 different neural network architectures and each of them can be used under 25 different input resolutions. Since all of these sub-networks share the same weights (i.e., Wo) (Cheung et al., 2019), we only require 7.7M parameters to store all of them. Without sharing, the total model size will be prohibitive.
sections.heading: 3.3 TRAINING THE ONCE-FOR-ALL NETWORK
sections.text: Naı̈ve Approach. Training the once-for-all network can be cast as a multi-objective problem, where each objective corresponds to one sub-network. From this perspective, a naı̈ve training approach is to directly optimize the once-for-all network from scratch using the exact gradient of the overall objective, which is derived by enumerating all sub-networks in each update step, as shown in Eq. (1). The cost of this approach is linear to the number of sub-networks. Therefore, it is only applicable to scenarios where a limited number of sub-networks are supported (Yu et al., 2019), while in our case, it is computationally prohibitive to adopt this approach.
  Another naı̈ve training approach is to sample a few sub-networks in each update step rather than enumerate all of them, which does not have the issue of prohibitive cost. However, with such a large number of sub-networks that share weights, thus interfere with each other, we find it suffers from
  Published as a conference paper at ICLR 2020 Connection to Network Pruning
  • Progressive shrinking can be viewed as a generalized network pruning with much higher flexibility across 4 dimensions.
  train with full width
  channel
  importance
  0.02 0.15 0.85 0.63 channel sorting
  progressively shrink the width
  channel importance
  0.82 0.11 0.46 reorg. channel sorting reorg.
  progressively shrink the width
  channel sorting .
  + + + p1 p2 p3O1 O2
  O3O1
  O2
  O1
  significant accuracy drop. In the following section, we introduce a solution to address this challenge, i.e., progressive shrinking.
  Progressive Shrinking. The once-for-all network comprises many sub-networks of different sizes where small sub-networks are nested in large sub-networks. To prevent interference between the sub-networks, we propose to enforce a training order from large sub-networks to small sub-networks in a progressive manner. We name this training scheme as progressive shrinking (PS). An example of the training process with PS is provided in Figure 3 and Figure 4, where we start with training the largest neural network with the maximum kernel size (e.g., 7), depth (e.g., 4), and width (e.g., 6). Next, we progressively fine-tune the network to support smaller sub-networks by gradually adding them into the sampling space (larger sub-networks may also be sampled). Specifically, after training the largest network, we first support elastic kernel size which can choose from {3, 5, 7} at each layer, while the depth and width remain the maximum values. Then, we support elastic depth and elastic width sequentially, as is shown in Figure 3. The resolution is elastic throughout the whole training process, which is implemented by sampling different image sizes for each batch of training data. We also use the knowledge distillation technique after training the largest neural network (Hinton et al., 2015; Ashok et al., 2018; Yu & Huang, 2019b). It combines two loss terms using both the soft labels given by the largest neural network and the real labels.
  Compared to the naı̈ve approach, PS prevents small sub-networks from interfering large sub-networks, since large sub-networks are already well-trained when the once-for-all network is fine-tuned to support small sub-networks. Regarding the small sub-networks, they share the weights with the large ones. Therefore, PS allows initializing small sub-networks with the most important weights of well-trained large sub-networks, which expedites the training process. Compared to network pruning (Figure 4), PS also starts with training the full model, but it shrinks not only the width dimension but also the depth, kernel size, and resolution dimensions of the full model. Additionally, PS fine-tunes both large and small sub-networks rather than a single pruned network. As a result, PS provides a much more powerful once-for-all network that can fit diverse hardware platforms and efficiency constraints compared to network pruning. We describe the details of the PS training flow as follows:
  5
  Published as a conference paper at ICLR 2020
  7x7
  Transformation Matrix: 25x25
  5x5
  Transformation Matrix: 9x9
  3x3
  stage i stage i
  train with full depth
  stage i
  progressively shrink the depth progressively shrink the depth
  O1
  O2
  O3
  O1
  O2
  1
  • Elastic Kernel Size (Figure 5 left). We let the center of a 7x7 convolution kernel also serve as a 5x5 kernel, the center of which can also be a 3x3 kernel. Therefore, the kernel size becomes elastic. The challenge is that the centering sub-kernels (e.g., 3x3 and 5x5) are shared and need to pl y multiple roles (independent kernel a d part of a large kernel). The weights of centered sub-kernels may need to have different distribution or magnitude as different roles. Forcing them to be the same degrades the performance of some sub-networks. Therefore, we introduce kernel transformation matrices when sharing the kernel weights. We use separate kernel transformation matrices for different layers. Within each layer, the kernel transformation matrices are shared among different channels. As such, we only need 25× 25+ 9× 9 = 706 extra parameters to store the kernel transformation matrices in each layer, which is negligible.
  • Elastic Depth (Figure 5 right). To derive a sub-network that has D layers in a unit that originally has N layers, we keep the first D layers and skip the last N −D layers, rather than keeping any D layers as done in current NAS methods (Cai et al., 2019; Wu et al., 2019). As such, one depth setting only corresponds to one combination of layers. In the end, the weights of the first D layers are shared between large and small models.
  • Elastic Width (Figure 6). Width means the number of channels. We give each layer the flexibility to choose different channel expansion ratios. Following the progressive shrinking scheme, we first train a full-width model. Then we introduce a channel sorting operation to support partial widths. It reorganizes the channels according to their importance, which is calculated based on the L1 norm of a channel’s weight. Larger L1 norm means more important. For example, when shrinking from a 4-channel-layer to a 3-channel-layer, we select the largest 3 channels, whose weights are shared with the 4-channel-layer (Figure 6 left and middle). Thereby, smaller sub-networks are initialized with the most important channels on the once-for-all network which is already well trained. This channel sorting operation preserves the accuracy of larger sub-networks.
sections.heading: 3.4 SPECIALIZED MODEL DEPLOYMENT WITH ONCE-FOR-ALL NETWORK
sections.text: Having trained a once-for-all network, the next stage is to derive the specialized sub-network for a given deployment scenario. The goal is to search for a neural network that satisfies the efficiency (e.g., latency, energy) constraints on the target hardware while optimizing the accuracy. Since OFA decouples model training from neural architecture search, we do not need any training cost in this stage. Furthermore, we build neural-network-twins to predict the latency and accuracy given a neural network architecture, providing a quick feedback for model quality. It eliminates the repeated search cost by substituting the measured accuracy/latency with predicted accuracy/latency (twins).
  Specifically, we randomly sample 16K sub-networks with different architectures and input image sizes, then measure their accuracy on 10K validation images sampled from the original training set. These [architecture, accuracy] pairs are used to train an accuracy predictor to predict the accuracy of a model given its architecture and input image size2. We also build a latency lookup table (Cai et al., 2019) on each target hardware platform to predict the latency. Given the target hardware and latency constraint, we conduct an evolutionary search (Real et al., 2019) based on the neural-network-twins to get a specialized sub-network. Since the cost of searching with neural-network-twins is negligible,
  2Details of the accuracy predictor is provided in Appendix A.
  Published as a conference paper at ICLR 2020
  40
  Performances of Sub-networks on ImageNet
  Im ag
  eN et
  T op
  -1 A
  cc (%
  )
  67
  70
  73
  75
  78 w/o PS w/ PS
  D=2 W=3 K=3
  D=2 W=3 K=7
  D=2 W=6 K=3
  D=2 W=6 K=7
  D=4 W=3 K=3
  D=4 W=3 K=7
  D=4 W=6 K=3
  D=4 W=6 K=7
  2.5% 2.8%
  3.5% 3.4% 3.3%
  3.4% 3.7%
  3.5%
  Sub-networks under various architecture configurations D: depth, W: width, K: kernel size
  • Progressive shrinking consistently improves accuracy of sub-networks on ImageNet.Figure 7: ImageNet top1 accuracy (%) performances of sub-networks under resolution 224× 224. “(D = d, W = w, K = k)” denotes a sub-network with d layers in each unit, and each layer has an width expansion ratio w and kernel size k.
  we only need 40 GPU hours to collect the data pairs, and the cost stays constant regardless of #deployment scenarios.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we first apply the progressive shrinking algorithm to train the once-for-all network on ImageNet (Deng et al., 2009). Then we demonstrate the effectiveness of our trained once-for-all network on various hardware platforms (Samsung S7 Edge, Note8, Note10, Google Pixel1, Pixel2, LG G8, NVIDIA 1080Ti, V100 GPUs, Jetson TX2, Intel Xeon CPU, Xilinx ZU9EG, and ZU3EG FPGAs) with different latency constraints.
sections.heading: 4.1 TRAINING THE ONCE-FOR-ALL NETWORK ON IMAGENET
sections.text: Training Details. We use the same architecture space as MobileNetV3 (Howard et al., 2019). For training the full network, we use the standard SGD optimizer with Nesterov momentum 0.9 and weight decay 3e−5. The initial learning rate is 2.6, and we use the cosine schedule (Loshchilov & Hutter, 2016) for learning rate decay. The full network is trained for 180 epochs with batch size 2048 on 32 GPUs. Then we follow the schedule described in Figure 3 to further fine-tune the full network3. The whole training process takes around 1,200 GPU hours on V100 GPUs. This is a one-time training cost that can be amortized by many deployment scenarios.
  Results. Figure 7 reports the top1 accuracy of sub-networks derived from the once-for-all networks that are trained with our progressive shrinking (PS) algorithm and without PS respectively. Due to space limits, we take 8 sub-networks for comparison, and each of them is denoted as “(D = d, W = w, K = k)”. It represents a sub-network that has d layers for all units, while the expansion ratio and kernel size are set to w and k for all layers. PS can improve the ImageNet accuracy of sub-networks by a significant margin under all architectural settings. Specifically, without architecture optimization, PS can achieve 74.8% top1 accuracy using 226M MACs under the architecture setting (D=4, W=3, K=3), which is on par with MobileNetV3-Large. In contrast, without PS, it only achieves 71.5%, which is 3.3% lower.
sections.heading: 4.2 SPECIALIZED SUB-NETWORKS FOR DIFFERENT HARDWARE AND CONSTRAINTS
sections.text: We apply our trained once-for-all network to get different specialized sub-networks for diverse hardware platforms: from the cloud to the edge. On cloud devices, the latency for GPU is measured with batch size 64 on NVIDIA 1080Ti and V100 with Pytorch 1.0+cuDNN. The CPU latency is measured with batch size 1 on Intel Xeon E5-2690 v4+MKL-DNN. On edge devices, including mobile phones, we use Samsung, Google and LG phones with TF-Lite, batch size 1; for mobile GPU,
  3Implementation details can be found in Appendix B.
  Published as a conference paper at ICLR 2020
  Model ImageNet MACs Mobile Search cost Training cost Total cost (N = 40)Top1 (%) latency (GPU hours) (GPU hours) GPU hours CO2e (lbs) AWS cost MobileNetV2 [31] 72.0 300M 66ms 0 150N 6k 1.7k $18.4k MobileNetV2 #1200 73.5 300M 66ms 0 1200N 48k 13.6k $146.9k NASNet-A [44] 74.0 564M - 48,000N - 1,920k 544.5k $5875.2k DARTS [25] 73.1 595M - 96N 250N 14k 4.0k $42.8k MnasNet [33] 74.0 317M 70ms 40,000N - 1,600k 453.8k $4896.0k FBNet-C [36] 74.9 375M - 216N 360N 23k 6.5k $70.4k ProxylessNAS [4] 74.6 320M 71ms 200N 300N 20k 5.7k $61.2k SinglePathNAS [8] 74.7 328M - 288 + 24N 384N 17k 4.8k $52.0k AutoSlim [38] 74.2 305M 63ms 180 300N 12k 3.4k $36.7k MobileNetV3-Large [15] 75.2 219M 58ms - 180N 7.2k 1.8k $22.2k OFA w/o PS 72.4 235M 59ms 40 1200 1.2k 0.34k $3.7k OFA w/ PS 76.0 230M 58ms 40 1200 1.2k 0.34k $3.7k OFA w/ PS #25 76.4 230M 58ms 40 1200 + 25N 2.2k 0.62k $6.7k OFA w/ PS #75 76.9 230M 58ms 40 1200 + 75N 4.2k 1.2k $13.0k OFALarge w/ PS #75 80.0 595M - 40 1200 + 75N 4.2k 1.2k $13.0k
  Table 1: Comparison with SOTA hardware-aware NAS methods on Pixel1 phone. OFA decouples model training from neural architecture search. The search cost and training cost both stay constant as the number of deployment scenarios grows. “#25” denotes the specialized sub-networks are fine-tuned for 25 epochs after grabbing weights from the once-for-all network. “CO2e” denotes CO2 emission which is calculated based on Strubell et al. (2019). AWS cost is calculated based on the price of on-demand P3.16xlarge instances.
  High (Low) Workload
  High (Low) Battery
  Specialized sub-network Number of Deployment Scenarios 0 20 40 60 80 16x~1300x reduction direct deploy (no retrain) train a once-for-all network specialized sub-nets Samsung Note10 Latency (ms) 75.4 cpu F P G A Different Hardware / Constraint De sig n Co st Previous: O(N) design cost Ours: O(1) design cost 8. 9% Edge, Full battery Edge, Low battery Cloud
  Number of Deployment Scenarios 0 20 40 60 80
  cpu F P
  G A
  Different Hardware / Constraint
  De sig n Co st Previous: O(N) design costEdge, Full battery Edge, Low battery
  Cloud
  …
  Architecture Design
  Train from Scratch Architecture Design Train from Scratch
  Architecture Design
  Train from Scratch
  Repeated architecture design and model training
  Note10 OFA #5 Acc Loss MobileNetV3Small
  #5 Acc Loss MobileNetV3Large Untitled 1 8.0 70.0 8.0 67.4 15.3 Untitled 2 8.9 71.0 10.8 70.4 22.0 Untitled 3 10.0 71.7 Untitled 4 10.9 72.6 11.7 73.4 13.0 73.8 14.1 74.3 15.1 74.6 15.8 74.8 17.0 75.0 18.5 75.1 19.4 75.5 20.0 75.6 21.3 75.7 21.7 75.9 22.4 76.1 To p1 Im ag eN et A cc (% ) 67 69 71 73 75 77 6 9 12 15 18 21 24 OFA MobileNetV3-Large MobileNetV3-Small 70.0 76.1 Tra in O nce , Ge t M any 75.2 73.3 70.4 67.4 Tra in Fo ur Tim es , Ge t F ou r
  ProxylessNAS
  FBNet
  MnasNet
  0 30000 60000 90000 120000
  N = 40 N = 100
  Table 1
  40 100
  ProxylessNAS 5700 8550
  FBNet 6500 9750
  MnasNet 45380 68070 453800 567250
  14.3k
  16.3k
  1134.5k453.8k
  specialized sub-nets
  Once-for-all network
  Edge, Full battery
  Cloud
  Edge, Full battery
  Edge, Low battery
  Cloud
  …
  Architecture Design
  Train from Scratch
  Architecture Design
  Train from Scratch
  Architecture Design
  Train from Scratch
  Previous Repeated architecture
  design and model training
  direct deploy (no retrain)
  Ours Train once, specialize for
  many deployment scenarios
  …
  Edge, Low battery
  Improved Efficiency
  SOTA Accuracy on ImageNet Mobile Setting
  CO2
  ProxylessNAS
  FBNet
  MnasNet
  OFA 0 12500 25000 37500 50000
  Total cost (lbs of CO2 emission), N = 40 Table 1-1-1
  40 100
  ProxylessNAS 5700 8550
  FBNet 6500 9750
  MnasNet 45380 68070 453800 567250
  OFA 300 300
  5.7k 6.5k
  454k
  0.34k 1300x
  Figure 8: OFA saves orders of magnitude design cost compared to NAS methods.
  we use Jetson TX2 with Pytorch 1.0+cuDNN, batch size of 16; for embedded FPGA, we use Xilinx ZU9EG and ZU3EG FPGAs with Vitis AI4, batch size 1.
  Comparison with NAS on Mobile Devices. Table 1 reports the comparison between OFA and state-of-the-art hardware-aware NAS methods on the mobile phone (Pixel1). OFA is much more efficient than NAS when handling multiple deployment scenarios since the cost of OFA is constant while others are linear to the number of deployment scenarios (N ). With N = 40, the total CO2 emissions of OFA is 16× fewer than ProxylessNAS, 19× fewer than FBNet, and 1,300× fewer than MnasNet (Figure 8). Without retraining, OFA achieves 76.0% top1 accuracy on ImageNet, which is 0.8% higher than MobileNetV3-Large while maintaining similar mobile latency. We can further improve the top1 accuracy to 76.4% by fine-tuning the specialized sub-network for 25 epochs and to 76.9% by fine-tuning for 75 epochs. Besides, we also observe that OFA with PS can achieve 3.6% better accuracy than without PS.
  OFA under Different Computational Resource Constraints. Figure 9 summarizes the results of OFA under different MACs and Pixel1 latency constraints. OFA achieves 79.1% ImageNet top1 accuracy with 389M MACs, being 2.8% more accurate than EfficientNet-B0 that has similar MACs. With 595M MACs, OFA reaches a new SOTA 80.0% ImageNet top1 accuracy under the mobile setting (<600M MACs), which is 0.2% higher than EfficientNet-B2 while using 1.68× fewer MACs. More importantly, OFA runs much faster than EfficientNets on hardware. Specifically, with 143ms Pixel1 latency, OFA achieves 80.1% ImageNet top1 accuracy, being 0.3% more accurate and 2.6× faster than EfficientNet-B2. We also find that training the searched neural architectures from scratch cannot reach the same level of accuracy as OFA, suggesting that not only neural architectures but also pre-trained weights contribute to the superior performances of OFA.
  Figure 10 reports detailed comparisons between OFA and MobileNetV3 on six mobile devices. Remarkably, OFA can produce the entire trade-off curves with many points over a wide range of latency constraints by training only once (green curve). It is impossible for previous NAS methods (Tan et al., 2019; Cai et al., 2019) due to the prohibitive training cost.
  4https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html
  8
  Published as a conference paper at ICLR 2020
  Compared with EfficientNet (FLOPs) OFA - Train From Scratch #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss EfficientNet #5 Acc Loss MnasNet #5 Acc Loss ProxylessNAS Untitled 1 389 77.0 389 79.10 390 76.3 Untitled 2 482.0 77.6 482.0 79.60 700 78.8 Untitled 3 595.0 77.9 595.0 80.00 1000 79.8 Untitled 4
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  76
  77
  78
  79
  80
  81
  200 400 600 800 1,000 1,200
  OFA OFA - Train from scratch EfficientNet
  76.3
  78.8
  79.879.6 79.1
  MACs (M)
  80.0 1.68x MACs reduction
  Compared with EfficientNet (Pixel1) OFA - Train From Scratch #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss EfficientNet #5 Acc Loss MnasNet #5 Acc Loss ProxylessNAS Untitled 1 78.7 76.4 78.7 78.70 163 76.3 Untitled 2 132.0 78.0 132.0 79.80 276 78.8 Untitled 3 143.0 78.2 143.0 80.10 375 79.8 Untitled 4
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  76
  77
  78
  79
  80
  81
  0 50 100 150 200 250 300 350 400 76.3
  78.8
  79.8 79.8
  78.7
  Google Pixel1 Latency (ms)
  80.1 2.6x latency reduction
  3.8% higher accuracy
  2.8% higher accuracy
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  76
  77
  78
  79
  80
  81
  200 400 600 800 1,000 1,200
  OFA EfficientNet
  76.3
  78.8
  79.879.6 79.1
  MACs (M)
  80.0 1.68x MACs reduction
  2.8% higher accuracy
  FLOPs Once for All #5 Acc Loss Once for All #25 Untitled 1 59.0 Untitled 2 90.0 Untitled 3 154.0 Untitled 4 213.0
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  40 80 120 160 200 240
  OFA MobileNetV3
  75.2
  73.3
  70.4
  67.4
  69.8
  72.6
  75.2 76.3
  MACs (M)
  Pixel1
  OFA - Train From Scratch
  Untitled 1 20.0 Untitled 2 28 Untitled 3 40 Untitled 4 58
  Google Pixel1 Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  18 24 30 36 42 48 54 60
  OFA MobileNetV3 OFA - Train from scratch
  75.2
  73.3
  70.4
  67.4
  76.4 74.9
  73.3
  71.4
  4% higher accuracy
  1.5x speedup
  77.0 77.6
  77.9
  76.4
  78.0 78.2
  Figure 9: OFA achieves 80.0% top1 accuracy with 595M MACs and 80.1% top1 accuracy with 143ms Pixel1 latency, setting a new SOTA ImageNet top1 accuracy on the mobile setting.
  Note10 OFA #5 Acc Loss OFA #25 #5 Acc Loss MobileNetV3Small #5 Acc Loss MobileNetV3Large #5 Acc Loss Untitled 1 8.0 70.0 8 71.4 8.0 67.4 Untitled 2 8.9 71.0 10.9 73.6 10.8 70.4 Untitled 3 10.0 71.7 15.8 75.5 15.3 73.3 Untitled 4 10.9 72.6 22.4 76.6 22.0 75.2 11.7 73.4 13.0 73.8 14.1 74.3 15.1 74.6 15.8 74.8 17.0 75.0 18.5 75.1 19.4 75.5 20.0 75.6 21.3 75.7 21.7 75.9 22.4 76.1 Note8 OFA #5 Acc Loss OFA #25 #5 Acc Loss MobileNetV3Small #5 Acc Loss MobileNetV3Large #5 Acc Loss Untitled 1 22.0 68.5 22.0 70.4 22.0 67.4 Untitled 2 24.0 69.4 31.0 72.8 31.0 70.4 Untitled 3 26.0 70.3 49.0 74.9 49.0 73.3 Untitled 4 28.0 70.9 65.0 76.1 65.0 75.2 30.0 71.6 31.0 71.7 34.0 72.3 36.0 72.7 38.0 73.1 39.0 73.4 42.0 73.7 44.0 74.0 46.0 74.3 49.0 74.7 52.0 74.9
  54.0 75.0 56.0 75.2 58.0 75.3 60.0 75.4 62.0 75.6 65.0 75.7
  LG G8
  OFA #5 Acc Loss OFA #25 #5 Acc Loss MobileNetV3Small #5 Acc Loss MobileNetV3Large #5 Acc Loss
  Untitled 1 8.3 69.4 8.3 71.1 8.2 67.4 Untitled 2 8.9 70.0 11.3 73.0 11.0 70.4 Untitled 3 10.0 70.9 16.0 74.7 17.0 73.3 Untitled 4 11.3 72.1 24.0 76.4 24.0 75.2
  13.3 73.0 14.0 73.5 15.3 73.9 16.0 74.3 18.0 75.0 19.0 75.2 20.0 75.3 21.0 75.4 22.0 75.6 23.0 75.8 24.0 76.0
  Samsung S7 Edge Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  25 40 55 70 85 100
  75.2
  73.3
  70.4
  67.4
  70.5
  73.1
  74.7
  76.3
  Samsung Note10 Latency (ms) To
  p1
  Im ag
  eN et
  A cc
  (% )
  67
  69
  71
  73
  75
  77
  7 9 11 13 15 17 19 21 23
  OFA #25 OFA MobileNetV3
  75.2
  73.3
  70.4
  67.4
  76.6 75.5
  73.6
  71.4
  Samsung Note8 Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  20 26 32 38 44 50 56 62 68
  75.2
  73.3
  70.4
  67.4
  76.1 74.9
  72.8
  70.4
  LG G8 Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  7 10 13 16 19 22 25
  75.2
  73.3
  70.4
  67.4
  76.4
  74.7
  73.0
  71.1
  Google Pixel1 Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  18 24 30 36 42 48 54 60
  75.2
  73.3
  70.4
  67.4
  76.4 74.9
  73.3
  71.4
  Google Pixel2 Latency (ms)
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  67
  69
  71
  73
  75
  77
  23 28 33 38 43 48 53 58 63 68
  75.2
  73.3
  70.4
  67.4
  75.8 74.7
  73.4
  71.5
  1
  Figure 10: OFA consistently outperforms MobileNetV3 on mobile platforms.
  OFA for Diverse Hardware Platforms. Besides the mobile platforms, we extensively studied the effectiveness of OFA on six additional hardware platforms (Figure 11) using the ProxylessNAS architecture space (Cai et al., 2019). OFA consistently improves the trade-off between accuracy and latency by a significant margin, especially on GPUs which have more parallelism. With similar latency as MobileNetV2 0.35, “OFA #25” improves the ImageNet top1 accuracy from MobileNetV2’s 60.3% to 72.6% (+12.3% improvement) on the 1080Ti GPU. Detailed architectures of our specialized models are shown in Figure 14. It reveals the insight that using the same model for different deployment scenarios with only the width multiplier modified has a limited impact on efficiency improvement: the accuracy drops quickly as the latency constraint gets tighter.
  OFA for Specialized Hardware Accelerators. There has been plenty of work on NAS for generalpurpose hardware, but little work has been focused on specialized hardware accelerators. We quantitatively analyzed the performance of OFA on two FPGAs accelerators (ZU3EG and ZU9EG) using Xilinx Vitis AI with 8-bit quantization, and discuss two design principles.
  Principle 1: memory access is expensive, computation is cheap. An efficient CNN should do as much as computation with a small amount of memory footprint. The ratio is defined as the arithmetic intensity (OPs/Byte). The higher OPs/Byte, the less memory bounded, the easier to parallelize. Thanks to OFA’s diverse choices of sub-network architectures (1019) (Section 3.3), and the OFA
  9
  Published as a conference paper at ICLR 2020
  2080ti-gpu64 Once for All #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss Untitled 1 5.5 68.3 5.5 70.0 5.9 59.7 Untitled 2 6.8 71.1 6.8 72.2 7.7 64.4 Untitled 3 10.9 73.8 10.9 74.4 12.3 68.9 Untitled 4 13.4 74.6 13.4 75.0 14.3 70.5 59 63 66 70 73 77
  5 7 9 11 13 15
  OFA #25 OFA MnasNet MobileNetV2 Slimmable Nets V100 gpu64 Once for All #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss MobileNetV2 #5 Acc Loss MnasNet #5 Acc Loss Untitled 1 4.8 69.0 4.8 71.6 4.9 59.7 4.9 60.3 4.8 62.4 Untitled 2 6.0 71.2 6.0 73.0 5.8 64.4 5.8 65.4 6.2 67.8 Untitled 3 9.2 74.6 9.2 75.3 9.2 68.9 9.2 69.8 9.3 71.5 Untitled 4 10.6 75.5 10.6 76.1 10.6 70.5 10.6 72.0 11.2 74.0
  58
  62
  66
  69
  73
  77
  4 6 8 10 12
  OFA #25 OFA MnasNet MobileNetV2 Slimmable Nets
  1080ti gpu64 Once for All #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss MobileNetV2 #5 Acc Loss MnasNet #5 Acc Loss Untitled 1 11.6 70.1 11.6 72.6 12.2 59.7 12.2 60.3 11.9 62.4 Untitled 2 14.9 72.1 14.9 73.8 15.1 64.4 15.1 65.4 15.4 67.8 Untitled 3 22.0 74.6 22.0 75.3 24.0 68.9 24.0 69.8 23.4 71.5 Untitled 4 27.3 75.9 27.3 76.4 27.9 70.5 27.9 72.0 28.5 74.0
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  58
  62
  66
  69
  73
  77
  10 14 18 22 26 30
  2080Ti Latency (ms)
  NVIDIA V100 Latency (ms) Batch Size = 64
  60.3 65.4
  69.8 72.0
  60.3
  65.4
  69.8
  72.0
  NVIDIA 1080Ti Latency (ms) Batch Size = 64
  60.3
  65.4
  69.8 72.0
  72.6 73.8
  75.3 76.4
  70.0 72.2
  74.4 75.0
  71.6 73.0
  75.3
  Intel Xeon CPU Once for All #5 Acc Loss Once for All #25 Slimmable Nets #5 Acc Loss MobileNetV2 #5 Acc Loss MnasNet #5 Acc Loss Untitled 1 9.7 68.8 9.7 71.1 9.8 59.7 9.8 60.3 9.8 62.4 Untitled 2 10.9 70.2 10.9 72.0 10.6 64.4 10.6 65.4 11.0 67.8 Untitled 3 14.8 73.7 14.8 74.6 15.5 68.9 15.5 69.8 14.5 71.5 Untitled 4 16.8 75.0 16.8 75.7 17.8 70.5 17.8 72.0 17.5 74.0
  58
  62
  66
  69
  73
  77
  9 11 13 15 17 19 Intel Xeon CPU Latency (ms)
  Batch Size = 1
  60.3
  65.4
  69.8
  72.071.1
  74.6 75.7
  72.0
  76.1
  Jetson TX2 Latency (ms) Batch Size = 16
  To p-
  1 Im
  ag eN
  et A
  cc (%
  )
  58
  62
  66
  69
  73
  77
  30 45 60 75 90 105 58
  62
  66
  69
  73
  77
  1.5 2.0 2.5 3.0 3.5 4.0 58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0 59.1
  63.3
  69.0 71.5 67.0 69.4
  72.8 73.6
  Xilinx ZU9EG FPGA Latency (ms) Batch Size = 1 (Quantized)
  Xilinx ZU3EG FPGA Latency (ms) Batch Size = 1 (Quantized)
  59.1
  63.3
  69.0 71.5
  67.0
  69.6
  72.8 73.7
  72.0 69.8
  65.4
  60.3
  75.875.4
  72.9
  70.3
  58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0
  58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0
  58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0 58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0 58
  62
  66
  69
  73
  77
  3.0 4.0 5.0 6.0 7.0 8.0
  1
  Figure 11: Specialized OFA models consistently achieve significantly higher ImageNet accuracy with similar latency than non-specialized neural networks on CPU, GPU, mGPU, and FPGA. More remarkably, specializing for a new hardware platform does not add training cost using OFA.
  FPGA Arithmetic Intensity (op/B) MobileNetV2 MnasNet OFA
  0.35 27 27.6 39.4
  0.5 35.3 37.1 49.4
  0.75 51.6 51.9 54.4
  1.0 61 61.2 63.9
  Ar ith
  m et
  ic In
  te ns
  ity
  (O PS
  /B yt
  e)
  0
  18
  35
  53
  70
  1.0x
  FPGA UltraZed-EG GOPS/s
  MobileNetV2 MnasNet OFA
  0.35 36 31.8 61.2
  0.5 48.1 44.0 75.5
  0.75 67.8 81.3
  1.0 79 83.7
  ZU 3E
  G F
  PG A
  (G O
  PS /s
  )
  0
  23
  45
  68
  90
  FPGA ZCU102 GOPS/s-1
  MobileNetV2 MnasNet OFA
  0.35 77 67.6 126.8
  0.5 102.6 94.4 155.3
  0.75 150.6 135.4 164.6
  1.0 185 167.3 186.3
  ZU 9E
  G F
  PG A
  (G O
  PS /s
  )
  0
  50
  100
  150
  200 MobileNet-v2 MnasNet OFA (Ours)
  (under different latency constraints)
  on Xilinx ZU3EG FPGAon Xilinx ZU9EG FPGA
  0.75x0.5x0.35x1.0x0.75x0.5x0.35x1.0x0.75x0.5x0.35x (under different latency constraints)(under different latency co straints)
  O ut
  o f B
  RA M
  O ut
  o f B
  RA M
  Ar ith
  m et
  ic In
  te ns
  ity
  (O PS
  /B yt
  e)
  0.0
  12.5
  25.0
  37.5
  50.0
  ZU 9E
  G F
  PG A
  (G O
  PS /s
  )
  0.0
  40.0
  80.0
  120.0
  160.0 MobileNet-v2 MnasNet OFA (Ours)
  ZU 3E
  G F
  PG A
  (G O
  PS /s
  )
  0.0
  20.0
  .0
  60.0
  80.0
  Figure 12: OFA models improve the arithmetic intensity (OPS/Byte) and utilization (GOPS/s) compared with the MobileNetV2 and MnasNet (measured results on Xilinx ZU9EG and ZU3EG FPGA).
  model twin that can quickly give the accuracy/latency feedback (Section 3.4), the evolutionary search can automatically find a CNN architecture that has higher arithmetic intensity. As shown in Figure 12, OFA’s arithmetic intensity is 48%/43% higher than MobileNetV2 and MnasNet (MobileNetV3 is not supported by Xilinx Vitis AI). Removing the memory bottleneck results in higher utilization and GOPS/s by 70%-90%, pushing the operation point to the upper-right in the roofline model (Williams et al., 2009), as shown in Figure 13. (70%-90% looks small in the log scale but that is significant).
  Principle 2: the CNN architecture should be co-designed with the hardware accelerator’s cost model. The FPGA accelerator has a specialized depth-wise engine that is pipelined with the point-wise engine. The pipeline throughput is perfectly matched for 3x3 kernels. As a result, OFA’s searched model only has 3x3 kernel (Figure 14, a) on FPGA, despite 5x5 and 7x7 kernels are also in the search space. Additionally, large kernels sometimes cause “out of BRAM” error on FPGA, giving high cost. On Intel Xeon CPU, however, more than 50% operations are large kernels. Both FPGA and GPU models are wider than CPU, due to the large parallelism of the computation array.
sections.heading: 5 CONCLUSION
sections.text: We proposed Once-for-All (OFA), a new methodology that decouples model training from architecture search for efficient deep learning deployment under a large number of hardware platforms. Unlike
  10
  Published as a conference paper at ICLR 2020
  FPGA Arithmetic Intensity (op/B) MobileNetV2 MnasNet OFA 0.35 27 27.6 39.4 0.5 35.3 37.1 49.4 0.75 51.6 51.9 54.4 1.0 61 61.2 63.9 Ar ith m et ic In te ns ity (O PS /B yt e) 0 18 35 53 70
  1.0x FPGA UltraZed-EG GOPS/s MobileNetV2 MnasNet OFA 0.35 36 31.8 61.2 0.5 48.1 44.0 75.5 0.75 67.8 81.3 1.0 79 83.7 ZU 3E G F PG A (G O PS /s ) 0 23 45 68 90 FPGA ZCU102 GOPS/s-1 MobileNetV2 MnasNet OFA 0.35 77 67.6 126.8 0.5 102.6 94.4 155.3 0.75 150.6 135.4 164.6 1.0 185 167.3 186.3 ZU 9E G F PG A (G O PS /s ) 0 50 100 150 200 MobileNet-v2 MnasNet OFA (Ours) (under different latency constraints)
  0.75x0.5x0.35x1.0x0.75x0.5x0.35x1.0x0.75x0.5x0.35x (under different latency constraints)(under different latency constraints)
  M B
  1 3x
  3
  C on
  v 3x
  3
  P oo
  lin g
  FC
  164x164
  M B
  4 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  ZU3EG 4.1ms (R = 164) (3x3_MBConv1_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6
  M B 4 3x 3 M B 4 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 4 3x 3
  CPU 10.9ms (R = 144) 3x3_Conv_O40 (3x3_MBConv1_RELU6_O24 (5x5_MBConv4_RELU6_O32 (5x5_MBConv3_RELU6_O32 (3x3_MBConv3_RELU6_O56 (7x7_MBConv3_RELU6_O56 (5x5_MBConv4_RELU6_O104 (3x3_MBConv4_RELU6_O104 (7x7_MBConv3_RELU6_O104 (5x5_MBConv4_RELU6_O128 (7x7_MBConv4_RELU6_O128 (3x3_MBConv6_RELU6_O248 (3x3_MBConv4_RELU6_O248 (5x5_MBConv4_RELU6_O416 1x1_Conv_O1664 1664x1000_Linear M B 1 3x 3 C on v 3x 3 144x144 M B 4 5x 5 M B 3 5x 5 M B 3 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 7x 7 M B 6 3x 3 M B 4 3x 3 M B 4 5x 5 P oo lin g FC M B 1 3x 3 C on v 3x 3 144x144 M B 4 3x 3 M B 4 3x 3 M B 6 3x 3 M B 6 3x 3 M B 4 3x 3 M B 3 3x 3 M B 3 5x 5 M B 3 3x 3 M B 4 5x 5 M B 3 3x 3 M B 6 3x 3 M B 6 3x 3 M B 6 7x 7 M B 4 7x 7 M B 6 3x 3 M B 3 5x 5 P oo lin g FC
  1
  (a) 4.1ms latency on Xilinx ZU3EG (batch size = 1).
  M B
  1 3x
  3
  C on
  v 3x
  3
  P oo
  lin g
  FC
  164x164
  M B
  4 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  ZU3EG 4.1ms (R = 164) (3x3_MBConv1_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6
  M B 4 3x 3 M B 4 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 4 3x 3
  CPU 10.9ms (R = 144) 3x3_Conv_O40 (3x3_MBConv1_RELU6_O24 (5x5_MBConv4_RELU6_O32 (5x5_MBConv3_RELU6_O32 (3x3_MBConv3_RELU6_O56 (7x7_MBConv3_RELU6_O56 (5x5_MBConv4_RELU6_O104 (3x3_MBConv4_RELU6_O104 (7x7_MBConv3_RELU6_O104 (5x5_MBConv4_RELU6_O128 (7x7_MBConv4_RELU6_O128 (3x3_MBConv6_RELU6_O248 (3x3_MBConv4_RELU6_O248 (5x5_MBConv4_RELU6_O416 1x1_Conv_O1664 1664x1000_Linear M B 1 3x 3 C on v 3x 3 144x144 M B 4 5x 5 M B 3 5x 5 M B 3 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 7x 7 M B 6 3x 3 M B 4 3x 3 M B 4 5x 5 P oo lin g FC M B 1 3x 3 C on v 3x 3 144x144 M B 4 3x 3 M B 4 3x 3 M B 6 3x 3 M B 6 3x 3 M B 4 3x 3 M B 3 3x 3 M B 3 5x 5 M B 3 3x 3 M B 4 5x 5 M B 3 3x 3 M B 6 3x 3 M B 6 3x 3 M B 6 7x 7 M B 4 7x 7 M B 6 3x 3 M B 3 5x 5 P oo lin g FC (b) 10.9ms latency on Intel Xeon CPU (batch size = 1).
  M B
  1 3x
  3
  C on
  v 3x
  3
  P oo
  lin g
  FC
  164x164
  M B
  4 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  M B
  5 3x
  3
  M B
  6 3x
  3
  ZU3EG 4.1ms (R = 164) (3x3_MBConv1_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv4_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6 (3x3_MBConv5_RELU6 (3x3_MBConv6_RELU6
  M B 4 3x 3 M B 4 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 5 3x 3 M B 4 3x 3
  CPU 10.9ms (R = 144) 3x3_Conv_O40 (3x3_MBConv1_RELU6_O24 (5x5_MBConv4_RELU6_O32 (5x5_MBConv3_RELU6_O32 (3x3_MBConv3_RELU6_O56 (7x7_MBConv3_RELU6_O56 (5x5_MBConv4_RELU6_O104 (3x3_MBConv4_RELU6_O104 (7x7_MBConv3_RELU6_O104 (5x5_MBConv4_RELU6_O128 (7x7_MBConv4_RELU6_O128 (3x3_MBConv _RELU6_O248 (3x3_MBConv4_RELU6_O248 (5x5_MBConv4_RELU6_O416 1x1_Conv_O1664 1664x1000_Linear M B 1 3x 3 C on v 3x 3 144x144 M B 4 5x 5 M B 3 5x 5 M B 3 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 3x 3 M B 3 7x 7 M B 4 5x 5 M B 4 7x 7 M B 6 3x 3 M B 4 3x 3 M B 4 5x 5 P oo lin g FC M B 1 3x 3 C on v 3x 3 144x144 M B 4 3x 3 M B 4 3x 3 M B 6 3x 3 M B 6 3x 3 M B 4 3x 3 M B 3 3x 3 M B 3 5x 5 M B 3 3x 3 M B 4 5x 5 M B 3 3x 3 M B 6 3x 3 M B 6 3x 3 M B 6 7x 7 M B 4 7x 7 M B 6 3x 3 M B 3 5x 5 P oo lin g FC
  1
  (c) 14.9ms latency on NVIDIA 1080Ti (batch size = 64).
  Figure 14: OFA can design specialized models for different hardware and different latency constraint. “MB4 3x3” means “mobile block with expansion ratio 4, kernel size 3x3”. FPGA and GPU models are wider than CPU model due to larger parallelism. Different hardware has different cost model, leading to different optimal CNN architectures. OFA provides a unified and efficient design methodology.
  previous approaches that design and train a neural network for each deployment scenario, we designed a once-for-all network that supports different architectural configurations, including elastic depth, width, kernel size, and resolution. It reduces the training cost (GPU hours, energy consumption, and CO2 emission) by orders of magnitude compared to conventional methods. To prevent sub-networks of different sizes from interference, we proposed a progressive shrinking algorithm that enables a large number of sub-network to achieve the same level of accuracy compared to training them independently. Experiments on a diverse range of hardware platforms and efficiency constraints demonstrated the effectiveness of our approach. OFA provides an automated ecosystem to efficiently design efficient neural networks with the hardware cost model in the loop.
sections.heading: ACKNOWLEDGMENTS
sections.text: We thank NSF Career Award #1943349, MIT-IBM Watson AI Lab, Google-Daydream Research Award, Samsung, Intel, Xilinx, SONY, AWS Machine Learning Research Award for supporting this
  11
  Published as a conference paper at ICLR 2020
  research. We thank Samsung, Google and LG for donating mobile phones. We thank Shuang Wu and Lei Deng for drawing the Figure 2.
sections.heading: A DETAILS OF THE ACCURACY PREDICTOR
sections.text: We use a three-layer feedforward neural network that has 400 hidden units in each layer as the accuracy predictor. Given a model, we encode each layer in the neural network into a one-hot vector based on its kernel size and expand ratio, and we assign zero vectors to layers that are skipped. Besides, we have an additional one-hot vector that represents the input image size. We concatenate these vectors into a large vector that represents the whole neural network architecture and input image size, which is then fed to the three-layer feedforward neural network to get the predicted accuracy. In our experiments, this simple accuracy prediction model can provide very accurate predictions. At convergence, the root-mean-square error (RMSE) between predicted accuracy and estimated accuracy on the test set is only 0.21%. Figure 15 shows the relationship between the RMSE of the accuracy prediction model and the final results (i.e., the accuracy of selected sub-networks). We can find that lower RMSE typically leads to better final results. O ce for All #25 Untitled 1 16.3 72.4 Untitled 2 8.7 72.2 Untitled 3 4.5 72.7 2.3 72.8 1.0 74.1 0.5 74.7 0.2 75.1
  B IMPLEMENTATION DETAILS OF PROGRESSIVE SHRINKING
  After training the full network, we first have one stage of fine-tuning to incorporate elastic kernel size. In this stage (i.e., K ∈ [7, 5, 3]), we sample one sub-network in each update step. The network is fine-tuned for 125 epochs with an initial learning rate of 0.96. All other training settings are the same as training the full network.
  14
  Published as a conference paper at ICLR 2020
  Next, we have two stages of fine-tuning to incorporate elastic depth. We sample two sub-networks and aggregate their gradients in each update step. The first stage (i.e., D ∈ [4, 3]) takes 25 epochs with an initial learning rate of 0.08 while the second stage (i.e., D ∈ [4, 3, 2]) takes 125 epochs with an initial learning rate of 0.24.
  Finally, we have two stages of fine-tuning to incorporate elastic width. We sample four sub-networks and aggregate their gradients in each update step. The first stage (i.e., W ∈ [6, 4]) takes 25 epochs with an initial learning rate of 0.08 while the second stage (i.e., W ∈ [6, 4, 3]) takes 125 epochs with an initial learning rate of 0.24.
title: CIALIZE IT FOR EFFICIENT DEPLOYMENT