SMASH: ONE-SHOT MODEL ARCHITECTURE SEARCH THROUGH HYPERNETWORKS
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized handdesigned networks.
Summary of paper - This paper presents SMASH (or the one-Shot Model Architecture Search through Hypernetworks) which has two training phases (one to quickly train a random sample of network architectures and one to train the best architecture from the first stage). The paper presents a number of interesting experiments and discussions about those experiments, but offers more exciting ideas about training neural nets than experimental successes. 
Review - The paper is very well written with clear examples and an excellent contextualization of the work among current work in the field. The introduction and related work are excellently written providing both context for the paper and a preview of the rest of the paper. The clear writing make the paper easy to read, which also makes clear the various weaknesses and pitfalls of SMASH. 
The SMASH framework appears to provide more interesting contributions to the theory of training Neural Nets than the application of said training. While in some experiments SMASH offers excellent results, in others the results are lackluster (which the authors admit, offering possible explanations). 
It is a shame that the authors chose to push their section on future work to the appendices. The glimmers of future research directions (such as the end of the last paragraph in section 4.2) were some of the most intellectually exciting parts of the paper. This choice may be a reflection of preferring to highlight the experimental results over possible contributions to theory of neural nets.  
Pros - 
* Strong related work section that contextualizes this paper among current work
* Very interesting idea to more efficiently find and train best architectures 
* Excellent and thought provoking discussions of middle steps and mediocre results on some experiments (i.e. last paragraph of section 4.1, and last paragraph of section 4.2)
* Publicly available code 
Cons - 
* Some very strong experimental results contrasted with some mediocre results
* The balance of the paper seems off, using more text on experiments than the contributions to theory. 
* (Minor) - The citation style is inconsistent in places. 
=-=-=-= Response to the authors
I thank the authors for their thoughtful responses and for the new draft of their paper. The new draft laid plain the contribution of the memory bank which I had missed in the first version. As expected, the addition of the future work section added further intellectual excitement to the paper. 
The expansion of section 4.1 addressed and resolved my concerns about the balance of the paper by effortless intertwining theory and application. I do have one question from this section -  In table 1, the authors report p-values but fail to include them in their interpretation; what is purpose of including these p-values, especially noting that only one falls under the typical threshold for significance?