The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block configurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. 8 minutes on a single CPU for CIFAR-10).
This paper introduces an approach to compressing a deep network so as to satisfy a given budget. In contrast to methods that replace all convolutional blocks with the same cheaper alternative, the authors argue that mixing different types of such cheap alternatives should be effective. To achieve this in a fast manner, they propose to randomly sample architectures with mixed blocktypes that satisfy the given budget and rank these architectures using Fisher information.
Originality:
- The approach is simple, but seems effective. Although the different components that make up the final method are known, they are put together so as to address a different task, in a way that I find interesting.
Methodology:
- From the paper, it is not entirely clear how sampling under budget constraints is achieved. I imagine that one could do it in a naive way, by choosing the first block uniformly randomly, and then removing the candidates that do not fit the budget for the subsequent blocks. This, however, would seem to give a different importance to the early blocks than to the later ones, since the former would essentially not have any budget constraints. I would appreciate it is the authors could explain their strategy in more detail.
- The assumption that the number of blocks in the student is the same as in the teacher seems constraining. A better architecture might be obtained by having fewer, wider blocks, or more, thinner blocks. Do the authors see a potential way to address this limitation?
- In principle, it seems that the proposed Fisher Information-based strategy could be used for general NAS, not just for compact architectures. Have the authors investigated this direction?
Related work:
- It seems to me that the literature review on compression/pruning is a bit shallow. I acknowledge, however, that most works do not tackle the scenario where a budget is given. However, Chen et al., "Constraint-aware Deep Neural Network Compression", ECCV 2018, do, and it would be interesting to discuss and provide comparisons with this work.
Experiments:
- I appreciate the ablation study, which answered several of my questions.
- In Table 1, it seems counterintuitive that the (negative) correlation becomes smaller as the number of mini-batches increases. Do the authors have an explanation for this?
- In Section 5, are the baseline compact networks all trained using the same Attention Transfer algorithm as for the BlockSwap ones?
- In Table 2, the budget values (P. (K)) seem fairly arbitrary? How were they obtained? They seem to match those of SNIP. Is this because the authors ran SNIP, and then set their budget accordingly?
- Below Fig. 3, the authors mention that they are generous with sparsity-based methods because they count the number of non-zero parameters. Note that several structured-sparsity compression methods have been proposed (Alvarez & Salzmann, NIPS 2016, 2017; Wen et al., NIPS 2016), which, by contrast with regular sparsity ones, would cancel out entire filters.
Summary:
Overall, I like the simple idea proposed in this paper. I would however appreciate it if the authors could clarify the way they sample the architectures and address my questions about their experiments.