Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.
This paper proposes a variant of neural architecture search.  It uses established work on network morphisms as a basis for defining a search space.  Experiments search for effective CNN architectures for the CIFAR image classification task.
Positives:
(1) The approach is straightforward to implement and trains networks in a reasonable amount of time.
(2) An advantage over prior work, this approach integrates architectural evolution with the training procedure.  Networks are incrementally grown; child networks are initialized with learned parameters from their parents.  This eliminates the need to restart training when making an architectural change, and drastically speeds the search.
Negatives:
(1) The state-of-the-art CNN architectures are not mysterious or difficult to find, despite the paper's characterization of them being so.  Indeed, ResNet and DenseNet designs are both guided by extremely simple principles: stack a series of convolutional layers, pool occasionally, and use some form of skip-connection throughout.  The need for architectural search is unclear.
(2) The proposed search space is boring.  As described in Section 4, the possibly evolutionary changes are limited to deepening the network, widening the network, and adding a skip connection.  But these are precisely the design aspects that have been well-explored by human trial and error and for which good rules of thumb are already available.
(3) As a consequence of (1) and (2), the result is essentially rigged.  Since only depth, width, and skip connections are considered, the end network must end up looking like a ResNet or DenseNet, but with some connections pruned.  There is no way to discover a network outside of the principled design space articulated in point (1) above.  Indeed, the discovered network diagrams (Figures 4 and 5) fall in this space.
(4) Performance is worse than the best hand-designed baselines.  One would hope that, even if the search space is limited, the discovered networks might be more efficient or higher performing in comparison to the human designs which fall within that same space.  However, the results in Tables 3 and 4 show this not to be the case.  The best human designs outperform the evolved networks.  Moreover, the evolved networks are woefully inefficient in terms of parameter count.
Together, these negatives imply the proposed approach is not yet at the point of being useful in practice.  I think further work is required (perhaps expanding the search space) to resolve the current limitations of automated architecture search.
Misc:
Tables 3 and 4 would be easier to parse if resources were simply reported in terms of total GPU hours.