Modern learning models are characterized by large hyperparameter spaces. In order to adequately explore these large spaces, we must evaluate a large number of configurations, typically orders of magnitude more configurations than available parallel workers. Given the growing costs of model training, we would ideally like to perform this search in roughly the same wall-clock time needed to train a single model. In this work, we tackle this challenge by introducing ASHA, a simple and robust hyperparameter tuning algorithm with solid theoretical underpinnings that exploits parallelism and aggressive early-stopping. Our extensive empirical results show that ASHA outperforms state-of-the-art hyperparameter tuning methods; scales linearly with the number of workers in distributed settings; converges to a high quality configuration in half the time taken by Vizier (Google's internal hyperparameter tuning service) in an experiment with 500 workers; and beats the published result for a near state-of-the-art LSTM architecture in under 2× the time to train a single model.
In this paper, the authors propose an extension of the Successive Halving Algorithm (SHA) allowing its deployment in a distributed setting, a so-called massively parallel setting. The proposed algorithm is relatively straightforward and the main contribution appears to be in the experimental validation of the method. 
Quality: The work appears to be of a good quality. The experiments could benefit from some more rigorous statistical analysis, but that would most likely require a higher number of repetitions, which is understandably hard to do in a large-scale setting. 
Clarity: In general, the paper is well written. However, the presentation of Algorithm 3.2 was confusing. More comments on this below. 
Originality: The contribution is incremental, but important.
Justification: The problem is an important one. There is room for more research exploring the optimization of hyperparameters for large architectures such as deep networks.
Overall I think the paper is good enough for acceptation, but I found some elements that deserve attention. The experimental section is a bit perplexing, mostly in the first experiments on the sequential approaches. The final experiment on the large-scale setting is disappointing because it only compares ASHA with an underpowered version of Vizier, so the demonstration is not as impressive as it could be. Furthermore, PBT was discarded based on results from a small-scale benchmark, which were not very convincing either (possibly significantly better on one of two versions of the CIFAR-10 benchmark). If the authors have other reasons as to why PBT was not a good candidate for comparison, they should bring them forth. 
Find below some more comments and suggestions:
Regarding Algorithm 3.2, promotability (line 12) is never defined explicitly, so this can lead to confusion (it did in my case). I think promotability also requires that at least eta jobs are finished in the rung k, am I correct? If so it is missing from the definition. Perhaps a subroutine should be defined?
It feels odd to me that the first experiment mentioned in the paper is tucked away in Appendix 1. I find it breaks the flow of the paper. The fact that SHA outperforms Fabolas I believe is one of the important claims of the paper. Hence, the result should probably not be in an Appendix. I would suggest putting Figure 3 in the Appendix instead, or removing/condensing Figure 2, which is nice but wastes a lot of space. I also fail to grasp the difference between the CIFAR-10 benchmarks in Appendix 1 and those in Section 4.2. It seems they could be joined in one single experiment comprising SHA v Hyperband v Fabolas v PBT (perhaps removing a variant of Hyperband to reduce clutter). 
I also do not think the comparison between SHA and ASHA in a sequential case is relevant. The behavior of ASHA in the distributed case will be different than in the sequential case, so the comparison between the two variants of SHA does not bring any useful information. If I followed the method correctly, in the 9 worker example with eta=3, the first round of jobs would be all jobs at the lowest bracket (s=0), which would be followed by a round of jobs at the next bracket (3 jobs at s=1), and so on. Hence, the scheduling behavior would be exactly the same as SHA (albeit distributed instead of sequential). Am I correct in my assessment? If so, perhaps ASHA should just be removed from the sequential experiments. 
As a point of sale, it might be interesting to provide the performance of models with manually tuned hyperparameters as a reference (i.e., the recommended hyperparameters in the reference implementations of those works that were cited).
Appendix A.2 serves no purpose and should probably be removed.
Section 4.3 & 4.3.1: In both cases, what is the running time R for a single model?