  Automatic designing computationally efficient neural networks has received
much attention in recent years. Existing approaches either utilize network
pruning or leverage the network architecture search methods. This paper
presents a new framework named network adjustment, which considers network
accuracy as a function of FLOPs, so that under each network configuration, one
can estimate the FLOPs utilization ratio (FUR) for each layer and use it to
determine whether to increase or decrease the number of channels on the layer.
Note that FUR, like the gradient of a non-linear function, is accurate only in
a small neighborhood of the current network. Hence, we design an iterative
mechanism so that the initial network undergoes a number of steps, each of
which has a small `adjusting rate' to control the changes to the network. The
computational overhead of the entire search process is reasonable, i.e.,
comparable to that of re-training the final model from scratch. Experiments on
standard image classification datasets and a wide range of base networks
demonstrate the effectiveness of our approach, which consistently outperforms
the pruning counterpart. The code is available at
https://github.com/danczs/NetworkAdjustment.

Computer Vision and Pattern Recognition