Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.
First, I would like to note that the claim that SGD with momentum is a special case of Adam with large epsilon is technically wrong because Adam also includes the bias-corrected momentum estimates which SGD with momentum does not consider. It might seem like a small difference, however it is a form of learning rate schedule which most users of Adam are not aware of. In practice, however, Adam with large epsilon can approximate SGD with momentum. Just don't claim the equivalent since it is not there. 
I have some difficulties understanding the contribution of the paper. For example 
"When tuning all available metaparameters under a realistic protocol at scales common in deep learning,
we find that more general update rules never underperform their special cases."
In practice you do adjust hyperparameter search spaces to fit your conclusions, e.g., "We found that searching over (epsilon, alpha0/epsilon) was more efficient than searching over (epsilon, alpha)." Again, this alone invalidates your experimental setup since you biased it in order to fit your conclusion: "was more efficient" was found after running some prior experiments. 
Another situation where your experimental setup is unfairly tuned is when you used different hyperparameter ranges for similar hyperparameter, e.g. see D.2 for ResNet-32 on CIFAR-10 where 6 orders of magnitute difference was used for the initial learning of Momentum and 3 orders of magnitude difference for the initial learning rate of Adam. Similarly, there is a difference of 10x for ImageNet experiments. 
The paper suggests that 16 experiments is enough to produce good results. First, one should not forget the  special arrangements (see above) done for hyperparameter search space. Second, for any person working in black-box optimization it is clear that 16 experiments is next to nothing. It should give you something good in 1D, possibly in 2D if your search range is narrow. This is absolutely nothing in larger dimensions (providing that your benchmark in not super trivial and your hyperparameter search space is not absolutely boring when you already narrowed it around the optimum). After 16 evaluations you get pretty bad settings for most algorithms. 
Update: 
The paper uses a naive hyperparameter optimizer and runs it for a very small budget. The latter likely affects the conclusion of the paper that different training algorithms perform similarly. The authors seem to accept it by mentioning that this is the case for their tuning protocol/budget. 
If we would like to compare different training algorithms, we should optimize them on a set of problems using 2-3 state-of-the-art hyperparameter optimizers. Then, we should study how the best seen solutions so far and their robustness  change as a function of the computation budget (the maximum budget should be large enough). Then, one would see that the results are not that different for small budgets (a boring result) and somewhat different for larger budgets. Showing only the boring part seems more misleading than useful. 
Update#2:
As I mentioned in my review, Adam with large epsilon is not equivalent to momentum SGD but only approximates the latter. This is because the original Adam has a bias correction term and even if the same *global* learning rate schedule is used both for Adam with large epsilon and momentum SGD, they are not equivalent. In order to obtain the exact equivalence, one would need to either
1) drop the bias correction term of Adam and thus modify the algorithm in order to satisfy the claimed equivalence
or 
2) set a particular learning rate *for each batch pass* of Adam to simulate the effect of the bias correction term, this leads to a large number of hyperparameters - as many as the number of batch passes, this is intractable (the setup of the authors does not optimize such batch-wise hyperparameters, they are defined by a global scheduler as a function of batch/epoch index).  
If you avoid these modifications, then you can't claim the equivalence but only an approximation. If you don't have the equivalence of the two approaches and so momentum SGD is not a particular case of Adam, then the following sentence from the abstract is false: " As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum)". Again, strictly speaking, it is false that "Adam should never perform worse than momentum" because momentum SGD is not a particular case of the original Adam *unless* you drop the bias correction term or simulate it with tons of hyperparameters, one learning rate value per batch pass. Any global learning rate schedule *used for both* algorithms will not solve the issue because the bias correction term will remain. If you don't modify the learning rate schedule of Adam but only of momentum SGD, then you basically adjust your SGD by moving some part of Adam in it to claim the equivalence of the two, such actions can make pretty much every second algorithm equivalent to another. 
My main concern is described in the first Update. It is trivial that a more general optimizer is capable to perform at least as good as its particular case. What is not trivial is to clarify the interplay of computational budgets spent on hyperparameter tuning vs number of hyperparameters vs performance over time.