Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -it can guide and improve the learning of a new task model with different specifications, or on different datasets.
Summary: This paper proposes a meta-learning solution for problems involving optimizing multiple loss values. They use a simple (small mlp), discrete, stochastic controller to control applications of updates among a finite number of different update procedures. This controller is a function of heuristic features derived from the optimization problem, and is optimized using policy gradient either exactly in toy settings or in a online / truncated manor on larger problems. They present results on 4 settings: quadratic regression, MLP classification, GAN, and multi-task MNT. They show promising performance on a number of tasks as well as show the controllers ability to generalize to novel tasks.
This is an interesting method and tackles a impactful problem. The setup and formulation (using PG to meta-optimize a hyper parameter controller) is not extremely novel (there have been similar work learning hyper parameter controllers), but the structure, the problem domain, and applications are. The experimental results are through, and provide compelling proof that this method works as well as exploration as to why the method works (analyzing output softmax). Additionally the "transfer to different models" experiment is compelling.
Comments vaguely in order of importance:
1. I am a little surprised that this training strategy works. In the online setting for larger scale problems, your gradients are highly correlated and highly biased. As far as I can tell, you are performing something akin to truncated back back prop through time with policy gradients. The biased introduced via this truncation has been studied in great depth in [3] and shown to be harmful. As of now, the greedy nature of the algorithm is hidden across a number of sections (not introduced when presenting the main algorithm). Some comment as to this bias -- or even suggesting that it might exist would be useful. As of now, it is implied that the gradient estimator is unbiased.
2. Second, even ignoring this bias, the resulting gradients are heavily correlated. Algorithm 1 shows no sign of performing batched updates on \phi or anything to remove these corrections. Despite these concerns, your results seem solid. Nevertheless, further understanding as to this would be useful.
3. The structure of the meta-training loop was unclear to me. Algorithm 1 states S=1 for all tasks while the body -- the overhead section -- you suggest multiple trainings are required ( S>1?).
4. If the appendix is correct and learning is done entirely online, I believe the initialization of the meta-parameters would matter greatly -- if the default task performed poorly with a uniform distribution for sampling losses, performance would be horrible. This seems like a limitation of the method if this is the case.
5. Clarity: The first half of this paper was easy to follow and clear. The experimental section had a couple of areas that left me confused. In particular:
5.1/Figure 1: I think there is an overloaded use of lambda? My understanding as written that lambda is both used in the grid search (table 1) to find the best loss l_1 and then used a second location, as a modification of l_2 and completely separate from the grid search?
6. Validation data / test sets: Throughout this work, it is unclear what / how validation is performed. It seems you performing controller optimization (optimizing phi), on the validation set loss, while also reporting scores on this validation set. This should most likely instead be a 3rd dataset. You have 3 datasets worth of data for the regression task (it is still unclear, however, what is being used for evaluation), but it doesn't look like this is addressed in the larger scale experiments at all. Given the low meta-parameter count of the I don't think this represents a huge risk, and baselines also suffer from this issue (hyper parameter search on validation set) so I expect results to be similar. 
7. Page 4: "When ever applicable, the final reward $$ is clipped to a given range to avoid exploding or vanishing gradients". It is unclear to me how this will avoid these. In particular, the "exploding" will come from the \nabla log p term, not from the reward (unless you have reason to believe the rewards will grow exponentially). Additionally, it is unclear how you will have vanishing rewards given the structure of the learned controller. This clipping will also introduce bias, this is not discussed, and will probably lower variance. This is a trade off made in a number of RL papers so it seems reasonable, but not for this reason.
8. "Beyond fixed schedules, automatically adjusting the training of G and D remains untacked" -- this is not 100% true. While not a published paper, some early gan work [2] does contains a dynamic schedule but you are correct that this family of methods are not commonplace in modern gan research.
9. Related work: While not exactly the same setting, I think [1] is worth looking at. This is quite similar causing me pause at this comment: "first framework that tries to learn the optimization schedule in a data-driven way". Like this work, they also lean a controller over hyper-parameters (in there case learning rate), with RL, using hand designed features.
10. There seem to be a fair number of heuristic choices throughout. Why is IS squared in the reward for GAN training for example? Why is the scaling term required on all rewards? Having some guiding idea or theory for these choices or rational would be appreciated.
11. Why is PPO introduced? In algorithm 1, it is unclear how PPO would fit into this? More details or an alternative algorithm in the appendix would be useful. Why wasn't PPO used on all larger scale models? Does the training / performance of the meta-optimizer (policy gradient  vs ppo) matter? I would expect it would. This detail is not discussed in this paper, and some details -- such as the learning rate for the meta-optimizer I was unable to find.
12. "It is worth noting that all GAN K:1 baselines perform worse than the rest and are skipped in Figure 2, echoing statements (Arjovsky, Gulrajani, Deng) that more updates of G than D might be preferable in GAN training." I disagree with this statement. The WGAN framework is built upon a loss that can be optimized, and should be optimized, until convergence (the discriminator loss is non-saturating) -- not the reverse (more G steps than D steps) as suggested here. Arjovsky does discuss issues with training D to convergence, but I don't believe there is any exploration into multiple G steps per D step as a solution.
13. Reproducibility seems like it would be hard. There are a few parameters (meta-learning rates, meta-optimizers) that I could not find for example and there is a lot of complexity.
14: Claims in paper seem a little bold / overstating. The inception gain is marginal to previous methods, and trains slower than other baselines. This is also true of MNT section -- there, the best baseline model is not even given equal training time! There are highly positive points here, such as requiring less hyperparameter search / model evaluations to find performant models.
15. Figure 4a. Consider reformatting data (maybe histogram of differences? Or scatter plot). Current representation is difficult to read / parse.
Typos:
page 2, "objective term. on GANs, the AutoLoss: Capital o is needed.
Page 3: Parameter Learning heading the period is not bolded.
[1] Learning step size controllers for robust neural network training. Christian Daniel et. al.
[2]http://torch.ch/blog/2015/11/13/gan.html
[3] Understanding Short-Horizon Bias in Stochastic Meta-Optimization, Wu et.al.
Given the positives, and in-spite of the negatives, I would recommend to accept this paper as it discusses an interesting and novel approach when controlling multiple loss values.