This work provides an automatic machine learning (AutoML) modelling architecture called Autostacker. Autostacker improves the prediction accuracy of machine learning baselines by utilizing an innovative hierarchical stacking architecture and an efficient parameter search algorithm. Neither prior domain knowledge about the data nor feature preprocessing is needed. We significantly reduce the time of AutoML with a naturally inspired algorithm -Parallel Hill Climbing (PHC). By parallelizing PHC, Autostacker can provide candidate pipelines with sufficient prediction accuracy within a short amount of time. These pipelines can be used as is or as a starting point for human experts to build on. By focusing on the modelling process, Autostacker breaks the tradition of following fixed order pipelines by exploring not only single model pipeline but also innovative combinations and structures. As we will show in the experiment section, Autostacker achieves significantly better performance both in terms of test accuracy and time cost comparing with human initial trials and recent popular AutoML system.
The authors introduce a simple hill climbing approach to (very roughly) search in the space of cascades of classifiers.
They first reinvent the concept of cascades of classifiers as an extension of stacking (https://en.wikipedia.org/wiki/Cascading_classifiers). Cascading is like stacking but carries over all original model inputs to the next classifier.
The authors cast this nicely into a network view with nodes that are classifiers and layers that use the outputs from previous layers. However, other than relating this line of work to the ICLR community, this interpretation of cascading is not put to any use. 
The paper incorrectly claims that existing AutoML frameworks only allow using a specific single model. In fact, Auto-sklearn (Feurer et al, 2015) automatically constructs ensembles of up to 50 models, helping it to achieve more robust performance.
I have some questions about the hillclimbing approach: 
- How is the "one change" implemented in the hill climber? Does this evaluate results for each of several single changes and pick the best one? Or does it simply change one classifier and continue? Or does it evaluate all possible individual changes and pick the best one? I note that the term "HillClimber" would suggest that some sort of improvement has to be made in each step, but the algorithm description does not show any evaluation step at this point. The hill climbing described in the text seems to make sense, but the pseudocode appears broken.
Section 4.2: I am surprised that there is only a comparison to TPOT, not one to Auto-sklearn. Especially since Auto-sklearn constructs ensembles posthoc this would be an interesting comparison.
As the maximum range of number of layers is 5, I assume that scaling is actually an issue in practice after all, and the use of hundreds of primitive models alluded to in the introduction are not a reality at this point.
The paper mentions guarantees twice:
- "This kind of guarantee of not being worse on average comes from the the characteristic of AutoStacked"
- "can be guaranteed to do better on average"
I am confident that this is a mistake / an error in choosing the right expression in English. I cannot see why there should be a guarantee of any sort.
Empirically, Autostacker appears better than RandomForest, but that is not a big feat. The improvements vs. TPOT are more relevant. One question: the data sets used in Olson et al are very small. Does TPOT overfit on these? Since AutoStacker does not search as exhaustively, could this explain part of the performance difference? How many models are evaluated in total by each of the methods?
I am unsure about the domain for the HillClimber. Does it also a search over which classifiers are used where in the pipeline, or only about their hyperparameters?
Minor issues:
- The authors systematically use citations wrongly, apparently never using citep but only having inline citations.
- Some parts of the paper feel unscientific, such as using phrases like "giant possible search space". 
- There are also several English grammar mistakes (e.g., see the paragraph containing "for the discover") and typos.
- Why exactly would a small amount of data be more likely to be unbalanced?
- The data "cleaning" method of throwing out data with missing values is very unclean. I hope this has only been applied to the training set and that no test set data points have been dropped?
- Line 27 of Algorithm 1: sel_pip has not been defined here
Overall, this is an interesting line of work, but it does not seem quite ready for publication.
Pros: 
- AutoML is a topic of high importance to both academia and industry
- Good empirical results 
Cons: 
- Cascading is not new
- Unclear algorithm: what exactly does the Hillclimber function do?
- Missing baseline comparison to Auto-sklearn
- Incorrect statements about guarantees