We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters. While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks. Finally, we show its effectiveness on two distinct tasks.
Summary of paper:
This work proposes an extension to an existing method (Franceschi 2017) to optimize regularization hyperparameters. Their method claims increased stability in contrast to the existing one.
Summary of review:
This is an incremental change of an existing method. This is acceptable as long as the incremental change significantly improves results or the paper presents some convincing theoretical arguments. I did not find either to be the case. The theoretical arguments are interesting but lacking in rigor. The proposed method introduces hyper-hyperparameters which may be hard to tune. The experiments are small scale and it is unclear how much the method improves random grid search. For these reasons, I cannot recommend this paper for acceptance.
Comments:
1. Paper should cite Domke 2012 in related work section.
2. Should state and verify conditions for application of implicit function theorem on page 2.
3. Fix notation on page 3. Dot is used on the right hand side to indicate an argument but not left hand side for equation after "with respect to \lambda".
4. I would like to see more explanation for the figure in Appendix A. What specific optimization is being depicted? This figure could be moved into the paper's main body with some additional clarification.
5. I did not understand the paragraph beginning with "This poor estimation". Is this just a restatement of the previous paragraph, which concluded convergence will be slow if \eta is too small?
6. I do understand the notation used in equation (8) on page 4. Are <, > meant to denote less than/greater than or something else?
7. Discussion of weight decay on page 5 seems tangential to main point of the paper. Could be reduced to a sentence or two.
8. I would like to see some experimental verification that the proposed method significantly reduces the dropout gradient variance (page 6), if the authors claim that tuning dropout probabilities is an area they succeed where others don't.
9. Experiments are unconvincing. First, only one hyperparameter is being optimized and random search/grid search are sufficient for this. Second, it is unclear how close the proposed method is to finding the optimal regularization parameter \lambda. All one can conclude is that it performs slightly better than grid search with a small number of runs. I would have preferred to see an extensive grid search done to find the best possible \lambda, then seen how well the proposed method does compared to this.
10. I would have liked to see a plot of how the value of lambda changes throughout optimization. If one can initialize lambda arbitrarily and have this method find the optimal lambda, that is more impressive than a method that works simply because of a fortunate initialization.
Typos:
1. Optimization -> optimize (bottom of page 2)
2. Should be a period after sentence starting "Several algorithms" on page 2.
3. In algorithm box on page 5, enable_projection is never used. Seems like warmup_time should also be an input to the algorithm.