Well-performing deep learning models have enormous impact, but getting them to perform well is complicated, as the model architecture must be chosen and a number of hyperparameters tuned. This requires experimentation, which is timeconsuming and costly. We propose to address the problem of hyperparameter tuning by learning to forecast the training behaviour of deep learning architectures. Concretely, we introduce a forecasting model that, given a hyperparameter schedule (e.g., learning rate, weight decay) and a history of training observations (such as loss and accuracy), predicts how the training will continue. Naturally, forecasting is much faster and less expensive than running actual deep learning experiments. The main question we study is whether the forecasting model is good enough to be of use -can it indeed replace real experiments? We answer this affirmatively in two ways. For one, we show that the forecasted curves are close to real ones. On the practical side, we apply our forecaster to learn hyperparameter tuning policies. We experiment on a version of ResNet on CIFAR10 and on Transformer in a language modeling task. The policies learned using our forecaster match or exceed the ones learned in real experiments and in one case even the default schedules discovered by researchers. We study the learning rate schedules created using the forecaster are find that they are not only effective, but also lead to interesting insights.
The paper investigates the possibility of learning a model to predict the training behaviour of deep learning architectures from hyperparameter information and a history of training observations. The model can then be used by researchers or a reinforcement learning agent to make better hyperparameter choices. The paper first adapts the Transformer model to be suitable to this prediction task by introducing a discretization scheme that prevents the transformer decoder's predictions from collapsing to a single curve. Next, the problem is formalized as a partially-observable MDP with a discrete action set, and PPO and SimPLe are introduced. The proposed model-based method is compared against a human and a model-free baseline training a Wide ResNet on CIFAR-10. The model-based method achieves better validation error than the other baselines that use actual data. Next, the method is compared against a human and a model-free baseline training Transformer models on the Penn Treebank dataset. While the human achieves the best performance at the end of the run, the proposed method appears to learn more quickly than the others and finishes with performance comparable to the model-free baseline.
Currently I lean towards accepting this paper for publication, despite a few issues. It asks an interesting question: can we learn a model of the training dynamics to avoid actually having to do the training? This could potentially prevent a lot of unnecessary computation and also lead to better-performing models. It then shows some experimental evidence suggesting that this is possible.
Most importantly, I would like to see a measure of variance/uncertainty like confidence intervals included in the results; otherwise it's impossible to assess whether the results are likely to be significant or not. Other questions:
1. In the PTB experiment, it looks like the human only adapts the learning rate and leaves the rest of the hyperparameters alone. Why was this policy used as the baseline? It seems extremely basic and unlikely to truly lead to optimal performance.
2. Why were more baselines from the related work not included? I understand the experiments are a proof of concept, but it would be nice to get a feeling for what some of the other methods do.
3. How do PPO and SimPLe handle partial observability? Is it principled to apply them to partially-observable environments?
4. Why not use continuous actions with a parameterized policy (e.g. Gaussian)?
5. Is it reasonable to assume that the learning dynamics of all deep learning architectures are similar enough that a model trained on one set of deep learning architectures and problems will generalize to new architectures and problems?