On the Tunability of Optimizers in Deep Learning
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability: How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.
The main contributions of the submission are: 1. A comprehensive empirical comparison of deep learning optimizers, with their performance compared under different amount of hyper-parameter tuning (they perform hyper-parameter tuning using random search). 2. The introduction of a novel metric that tries to capture the *tunability* of an optimizer. This metric attempts to trade off the performance of an optimizer when tuned only with a small number of hyper-parameter trials, and its performance when carefully tuned. The metric is defined as a weighted average of the performance after tuning with i random trials, with i that goes from 1 to K. The weights of this weighted average and K are *hyper-parameters* of the metric itself. They use K=100 and suggest 3 possible choices of weights. The paper appears to treat 2. as the main contribution. However, I do not think the metric they introduce is good enough to be recommended in future work, when comparing tunability of optimizers (or other algorithms with hyperparameters). The reason is that simpler methods provide just as much information, and do not rely on the need of interpreting the choice of the weights and K. This point is proven in the paper itself, where for example Figure 2 provides a more concrete and easier to interpret information than the tunability metric, similar graphs could be easily provided per dataset. Similarly, figure 3 as well as figures 5-7 and 8 in the appendix provide very good information about the tunability of the various optimizers without using the introduced metrics. Information similar (although not identical) to that summarized in table 5 could be captured by substituting the 3 metrics with the best performance after tuning for 4, 16 and 64 iterations respectively (just as examples). A stronger contribution is 1., which however is somewhat incremental compared to similar comparisons made in the past. Comparisons which, while mentioned, should perhaps have been discussed and compared more in detail in this work. Overall, I do not feel the comparisons dramatically change the qualitative understanding the field has of the different optimizers and their tunability. They also suggest that when the tuning budget is low, using Adam but tuning only the learning rate is beneficial, which could be a valuable and practical suggestion. I enjoyed reading the submission, which is very clearly written, but due to the relatively limited value of the contributions, and excessive focus on the tunability metric which I do not feel is giustified, I slightly lean against acceptance here at ICLR. I do think, however, that it would make a great submission to a smaller venue or workshop. Other comments/notes: * One aspects that is mostly left out of the discussion (except from one side comment) is the wallclock time, as some optimizers might be on average quicker to train (for example due to quicker convergence), this can easily lead it to be quicker to tune even though it requires a higher budget of trials. I think it would be worth discussing this more. * minor: in figure 8 in the appendix, the results after 100 iterations is, as far as I understand, over a single replication, so is not particularly reliable (and will always be 100% of a single optimizer) * similarly to the above, if the configurations are always sampled from the same 100, confidence intervals in the graphs become less reliable as the budget increases.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On the Tunability of Optimizers in Deep Learning
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability: How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The main contributions of the submission are: 1. A comprehensive empirical comparison of deep learning optimizers, with their performance compared under different amount of hyper-parameter tuning (they perform hyper-parameter tuning using random search). 2. The introduction of a novel metric that tries to capture the *tunability* of an optimizer. This metric attempts to trade off the performance of an optimizer when tuned only with a small number of hyper-parameter trials, and its performance when carefully tuned. The metric is defined as a weighted average of the performance after tuning with i random trials, with i that goes from 1 to K. The weights of this weighted average and K are *hyper-parameters* of the metric itself. They use K=100 and suggest 3 possible choices of weights. The paper appears to treat 2. as the main contribution. However, I do not think the metric they introduce is good enough to be recommended in future work, when comparing tunability of optimizers (or other algorithms with hyperparameters). The reason is that simpler methods provide just as much information, and do not rely on the need of interpreting the choice of the weights and K. This point is proven in the paper itself, where for example Figure 2 provides a more concrete and easier to interpret information than the tunability metric, similar graphs could be easily provided per dataset. Similarly, figure 3 as well as figures 5-7 and 8 in the appendix provide very good information about the tunability of the various optimizers without using the introduced metrics. Information similar (although not identical) to that summarized in table 5 could be captured by substituting the 3 metrics with the best performance after tuning for 4, 16 and 64 iterations respectively (just as examples). A stronger contribution is 1., which however is somewhat incremental compared to similar comparisons made in the past. Comparisons which, while mentioned, should perhaps have been discussed and compared more in detail in this work. Overall, I do not feel the comparisons dramatically change the qualitative understanding the field has of the different optimizers and their tunability. They also suggest that when the tuning budget is low, using Adam but tuning only the learning rate is beneficial, which could be a valuable and practical suggestion. I enjoyed reading the submission, which is very clearly written, but due to the relatively limited value of the contributions, and excessive focus on the tunability metric which I do not feel is giustified, I slightly lean against acceptance here at ICLR. I do think, however, that it would make a great submission to a smaller venue or workshop. Other comments/notes: * One aspects that is mostly left out of the discussion (except from one side comment) is the wallclock time, as some optimizers might be on average quicker to train (for example due to quicker convergence), this can easily lead it to be quicker to tune even though it requires a higher budget of trials. I think it would be worth discussing this more. * minor: in figure 8 in the appendix, the results after 100 iterations is, as far as I understand, over a single replication, so is not particularly reliable (and will always be 100% of a single optimizer) * similarly to the above, if the configurations are always sampled from the same 100, confidence intervals in the graphs become less reliable as the budget increases.
assistant
The main contributions of the submission are: 1. A comprehensive empirical comparison of deep learning optimizers, with their performance compared under different amount of hyper-parameter tuning (they perform hyper-parameter tuning using random search). 2. The introduction of a novel metric that tries to capture the *tunability* of an optimizer. This metric attempts to trade off the performance of an optimizer when tuned only with a small number of hyper-parameter trials, and its performance when carefully tuned. The metric is defined as a weighted average of the performance after tuning with i random trials, with i that goes from 1 to K. The weights of this weighted average and K are *hyper-parameters* of the metric itself. They use K=100 and suggest 3 possible choices of weights. The paper appears to treat 2. as the main contribution. However, I do not think the metric they introduce is good enough to be recommended in future work, when comparing tunability of optimizers (or other algorithms with hyperparameters). The reason is that simpler methods provide just as much information, and do not rely on the need of interpreting the choice of the weights and K. This point is proven in the paper itself, where for example Figure 2 provides a more concrete and easier to interpret information than the tunability metric, similar graphs could be easily provided per dataset. Similarly, figure 3 as well as figures 5-7 and 8 in the appendix provide very good information about the tunability of the various optimizers without using the introduced metrics. Information similar (although not identical) to that summarized in table 5 could be captured by substituting the 3 metrics with the best performance after tuning for 4, 16 and 64 iterations respectively (just as examples). A stronger contribution is 1., which however is somewhat incremental compared to similar comparisons made in the past. Comparisons which, while mentioned, should perhaps have been discussed and compared more in detail in this work. Overall, I do not feel the comparisons dramatically change the qualitative understanding the field has of the different optimizers and their tunability. They also suggest that when the tuning budget is low, using Adam but tuning only the learning rate is beneficial, which could be a valuable and practical suggestion. I enjoyed reading the submission, which is very clearly written, but due to the relatively limited value of the contributions, and excessive focus on the tunability metric which I do not feel is giustified, I slightly lean against acceptance here at ICLR. I do think, however, that it would make a great submission to a smaller venue or workshop. Other comments/notes: * One aspects that is mostly left out of the discussion (except from one side comment) is the wallclock time, as some optimizers might be on average quicker to train (for example due to quicker convergence), this can easily lead it to be quicker to tune even though it requires a higher budget of trials. I think it would be worth discussing this more. * minor: in figure 8 in the appendix, the results after 100 iterations is, as far as I understand, over a single replication, so is not particularly reliable (and will always be 100% of a single optimizer) * similarly to the above, if the configurations are always sampled from the same 100, confidence intervals in the graphs become less reliable as the budget increases.