DEEPOBS: A DEEP LEARNING OPTIMIZER BENCHMARK SUITE
There is significant past and ongoing research on optimization methods for deep learning. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of such optimizers. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DEEPOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on IMAGENET or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce L A T E X code for inclusion in academic publications. It supports TENSORFLOW and is available open source.
This paper presents a new benchmark suite to compare optimizer on deep neural networks. It provides a pipeline to help streamlining the analysis of new optimizers which would favor easily reproducible results and fair comparisons.
Quality
The paper covers well the problems underlying the construction of such a benchmark, discussing the problems and models selection, runtime estimation, hyper-parameter selection and visualizations. It falls short however in some cases:
1. Hyper-parameter optimization
    While they mention the importance of hyper-parameter tuning for the benchmark, they leave it to the user to tune them without providing any standard procedure. Furthermore, they use grid search to build the baselines while this is known to be a poor optimizer [1].
2. Estimated runtime
    Runtime is estimated for a single set of hyper-parameters of the optimizer, but some optimizer may have similar or roughly similar results for a large set of hyper-parameters that widely affects the runtime. The effect of the hyper-parameters should be taken into account for this part of the benchmark.
3. Interpretation
    Such a benchmark should makes it easier for interpretation of results as the authors suggests. However, the paper does not convey much interpretation in section 4, beside the fact that results are not conclusive for any baseline. Results of the paper seem low, but they are difficult to verify since the plots are not very precise. For instance Wide ResNet-18-8 reports 1.54% test accuracy on SVHN [6] while this paper reports ~ 15% for the Wide ResNet 18-4 version. Figure 2 is a good attempt at making interpretations of sensitivity of optimizers' hyper-parameters but has limited interpretability compared to what can be found in the literature [2].
4. Problems
    There is an effort to provide varied types of problem, including classical optimization functions, image classification, image generation and language modeling. The number of problems consists mostly of image classification however and is very limited for image generation and language modeling.
Clarity
The paper is well written and easy to understand in general. 
On a minor note, most figures are difficult to read. Side nodes on figure 1 does not divide clearly without any capital letter or punctuation at the end of sentence. Figure 2 should be self contained with its own legend. Figure 3 is useful for a visual impression of the speed of convergence but a histogram would be necessary for a better visual comparison of the different performances.
Section 2.2 has a confusing terminology for the "train valid set". Is it a standard validation set? 
Originality
There is virtually no benchmarks for optimizers available for the community. I believe a standardized procedure for comparing optimizers can be viewed as an original contribution. 
Significance
Reproducibility is a problem in machine learning [3, 4] and optimizers' efficiency on deep neural networks generalization performance is still not very well understood [5]. Therefore, there is a strong need for a benchmark for sound comparisons and to favor better reproducibility.
Conclusion
The benchmark presented in this paper would be an important contribution to the community but lacks a few important features in my opinion, in particular, sound hyper-parameter optimization procedure and sound interpretation tools. On a skeptical note, I doubt the benchmark will be used extensively if the results it provides yield no conclusive interpretation as reported for the baselines. As I feel there is more work needed to support the goals of the paper, I would suggest this paper for a workshop. Nevertheless, I would not be upset if it was accepted because of the importance of the subject and the originality of this work.
[1] Bergstra, James, and Yoshua Bengio. "Random search for hyper-parameter optimization." Journal of Machine Learning Research 13, no. Feb (2012): 281-305.
[2] Biedenkapp, Andre, Joshua Marben, Marius Lindauer and Frank Hutter. “CAVE : Configuration Assessment , Visualization and Evaluation.” In International Conference on Learning and Intelligent Optimization (2018).
[3] Lucic, Mario, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. “Are GANs Created Equal? A Large-Scale Study.” arXiv preprint arXiv:1711.10337 (2017).
[4] Melis, Gábor, Chris Dyer, and Phil Blunsom. “On the state of the art of evaluation in neural language models.” arXiv preprint arXiv:1707.05589 (2017).
[5] Wilson, Ashia C., Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. "The marginal value of adaptive gradient methods in machine learning." In Advances in Neural Information Processing Systems, pp. 4148-4158. 2017.
[6] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016
-----------
Revision
-----------
In light of the discussion with the authors, the revision made to chapter 4 and in particular the proposed modifications to section 2.4 for a camera-ready paper, I revise my score to 6.