In this paper, we present a deep convolutional neural network (CNN) that performs arbitrary resize operation on intermediate feature map resolution at stage-level. Motivated by weight sharing mechanism in neural architecture search, where a super-network is trained, and sub-networks inherit the weights from the supernetwork, we introduce a novel CNN approach. We present Resizable Neural Networks (ResizableNet), which resize feature map with arbitrary scaling factor at stage level. The inference cost of ResizableNet is equivalent to a network with a single scale because the weights of a convolutional layer are shared across diversified scales. Additionally, we provide essential techniques to build and train our propose ResizableNet. We demonstrate the effectiveness of ResizableNet on large-scale ImageNet classification with various modern network architectures such as MobileNet, ShuffleNet, and ResNet. To go even further, we present three applications of resizable networks: 1) Resizable Augmentation (Resizable-Aug). We present a data augmentation technique based on ResizableNet. It obtains superior performance on ImageNet classification, outperform AutoAugment by 1.2% with ResNet-50. 2) Resizable Network for Neural Architecture Search (Resizable-NAS). We introduce ResizableNet as a single network that can deploy on many different hardware to meet various efficiency constraints. 3) Adaptive Resizable Neural Network (Resizable-Adapt). We introduce the adaptive resizable networks as dynamic networks, which further improve the performance with less testing cost via data-dependent inference.
In a standard multi-stage neural network such as a ResNet, the resize operation between stages typically reduces the spatial resolution by 0.5.  (input ---> stage 1---> 0.5 reduction ---> stage 2 --> 0.5 reduction ---> ....). In this paper the authors apply a variety of reduction factors in between these stages for different training examples (input ---> stage 1---> variable reduction ---> stage 2 --> variable reduction ---> ....). They demonstrate that simply training in this way is a powerful form of augmentation. It also produces a network which may be scaled depending on a FLOP budget.
I think this is a really neat idea, and as far as I'm aware it is novel.  It is similar in spirit to EfficientNet, although more flexible. The experimental results are good. However, the paper is let down by poor writing and a lack of detail.
The paper needs a rewrite as there are many grammatical errors, which cause a bad impression:
- "performs arbitrary resize operation" ->  "performs arbitrary resize operations"
- "Scale is the fundamental component of the physical world" ---> *a* fundamental component!
- "i.e the acuracy of NISP" --> e.g.
- "compare with baseline" -> "compared to the baseline"
- There are several instances of a space missed out between a letter and an open bracket 
- "The Figure 2(a)" --> "Figure 2(a)"
The comparison to weight-sharing NAS methods is unnecessary. Those entail searching for architectures and sharing weights through the search process, whereas in this work it is having a network that can take different sized inputs at different stages. On that note, it doesn't feel right to me to refer to there being many "subnetworks". What there really is, is just a single network that is robust to multi-scaled inputs (which is a good thing!)
Figure 1 is nice!
EfficientNet-B0 is a base-network that can be scaled up with a compound scaling approach found through grid search. What happens if you scale up your Resizeable net to the same FLOPs as e.g. EfficientNet-B7?
I like the old-school vision citations, although referring to object detection makes me wonder why there are no experiments on it. For distillation, I recommend you cite https://arxiv.org/abs/1312.6184, as the Hinton paper is really just an extension of this.
The fair sampling seems important to the method. Could a detailed explanation be included in the appendix? Do you given the performance as a result of naive sampling? What do you mean by "Some convolutional layers can go through more training issues than others"? 
A big problem in this paper is that (as far as I can tell) the scaling factors considered aren't given (but we are told that they lie between 0 and 1). It isn't possible to sample these arbitrarily as you indicate that a batch-norm layer is needed for each one, so these must be discrete (because of this, it isn't correct to say that you have infinite networks contained within).  It therefore isn't clear e.g. in Table 1 which permutation of scaling factors were used in upsampling the networks. Are the results given representative of the best-case selection of these scaling? I hope this isn't the case. I am assuming it is uniform scaling in the case of the "individually trained counterparts". It would be interesting to know more generally which combinations worked well.
In Section 3.5 I'm not sure what is meant by "It is inefficient to process all images to one-subnetwork, as the algorithm spends equal energy at each sample".  I assume energy use is proportional to the number of FLOPS, which in turn depends on spatial resolution.
The legend in Figure 2 is close-to unreadable and needs changing.
The results are impressive, but error bars would be appreciated if possible. As ImageNet is the only dataset considered, this would give some needed clout. 
Pros
-------
- Nice, novel method
- Good experimental results
Cons
--------
- Paper is poorly written
- Very few details of the scaling factor variations
- Only one dataset considered
Although the paper is written badly and the narrative is muddled, the underlying idea is a nice one, which is executed well experimentally. Because of this I would like to recommend a Weak Accept, subject to the authors (i) doing a rewrite and (ii) including more information regarding the scaling permutations.