Network structures are important to learning good representations of many tasks in computer vision and machine learning communities. These structures are either manually designed, or searched by Neural Architecture Search (NAS) in previous works, which however requires either expert-level efforts, or prohibitive computational cost. In practice, it is desirable to efficiently and simultaneously learn both the structures and parameters of a network from arbitrary classes with budgeted computational cost. We identify it as a new learning paradigm -Boosting Network, where one starts from simple models, delving into complex trained models progressively. In this paper, by virtue of an iterative sparse regularization path --Split Linearized Bregman Iteration (SplitLBI), we propose a simple yet effective boosting network method that can simultaneously grow and train a network by progressively adding both convolutional filters and layers. Extensive experiments with VGG and ResNets validate the effectiveness of our proposed algorithms.
This paper proposes an architecture search method for deep convolutional neural network models that progressively increases the number of filters per layer as well as the number of layers, and the authors refer to this general approach as boosting networks. The algorithm for increasing the number of filters is based on split linear Bregman iteration, and the algorithm for increasing the number of layers proceeds block by block, increasing the layers per block until the accuracy does not increase. The experiments convincingly demonstrate gains in performance and smaller network sizes compared to baseline models, naive boosting methods, and a related method called Autogrow.
In my view, there are two main areas for further improvement for this work. First, the GT-Layers algorithm can be better motivated. Why is GT-Filters only run once at the beginning, rather than iteratively as the number of layers increases? Why does the procedure go from bottom blocks to up blocks (and, by the way, what are bottom blocks and up blocks)? Why measure training accuracy to determine when to add layers? I understand that these are all fairly heuristic choices, but nevertheless there needs to be proper motivation for all of the above.
Second, an additional effort should be made to compare to additional prior work in architecture search. It seems like the authors are suggesting that this method should be more computationally efficient and find smaller architectures, and demonstrating this empirically would greatly strengthen the paper. In particular, the existing results depicting the final number of parameters in the learned model are particularly striking to me. I appreciate that the authors included an experiment showing that a standard ResNet cannot be trained with the same number of FLOPs as the network found by your method. Can a similar analysis be made for wall clock time, i.e., how long the models actually take to train? A similar study (FLOPs, wall clock time, etc.) would also be very useful for the current comparison to Autogrow, as these metrics are often just as important, if not more important, than the number of parameters of the final model.
A thorough pass through the paper for spelling and grammar would be very useful.