Learning to Teach with Dynamic Loss Functions
Teaching is critical to human society: it is with teaching that prospective students are educated and human civilization can be inherited and advanced. A good teacher not only provides his/her students with qualified teaching materials (e.g., textbooks), but also sets up appropriate learning objectives (e.g., course projects and exams) considering different situations of a student. When it comes to artificial intelligence, treating machine learning models as students, the loss functions that are optimized act as perfect counterparts of the learning objective set by the teacher. In this work, we explore the possibility of imitating human teaching behaviors by dynamically and automatically outputting appropriate loss functions to train machine learning models. Different from typical learning settings in which the loss function of a machine learning model is predefined and fixed, in our framework, the loss function of a machine learning model (we call it student) is defined by another machine learning model (we call it teacher). The ultimate goal of teacher model is cultivating the student to have better performance measured on development dataset. Towards that end, similar to human teaching, the teacher, a parametric model, dynamically outputs different loss functions that will be used and optimized by its student model at different training stages. We develop an efficient learning method for the teacher model that makes gradient based optimization possible, exempt of the ineffective solutions such as policy optimization. We name our method as "learning to teach with dynamic loss functions" (L2T-DLF for short). Extensive experiments on real world tasks including image classification and neural machine translation demonstrate that our method significantly improves the quality of various student models.
** Post Response ** A thorough discussion regarding the connections to meta-learning techniques [3-7] should be added to the paper (and would significantly strengthen the paper, as it would make it more accessible and more likely to have impact within the meta-learning community). I hope that the authors include this discussion in the final version, as they say they will in the author response.  The new experiments and results address my other primary concerns, demonstrating that the method performs very strongly on a number of challenging, large-scale problems and generalizes to new datasets and architectures. I am raising my score from a 7 to an 8.  ** Original Review ** This paper aims to learn a differentiable loss function that leads to good performance on the "true" objective for a particular task (e.g. accuracy or BLEU score). In particular, the loss function is optimized with gradient descent with reverse mode autodiff w.r.t. the true objective on held-out data. The experiments include comparisons to prior work on MNIST, CIFAR-10, CIFAR-100, and De-En NMT, showing a small improvement.  Pros: The idea is interesting and motivated well, from the perspective of a teacher adapting how it teaches a student. The experiments include a number of different datasets, many different model architectures, and comparisons to multiple previous methods. Cons: The improvement over prior work seems to be relatively small (but it is consistent across tasks). There are important aspects of the experimental set-up that are unclear (details below). Finally, the proposed approach seems to be very computationally intensive, since it requires training the teacher on the task and then training the student using the teacher. After being trained, can the teacher generalize to new tasks?  Method: As mentioned above, the method is nicely motivated, from the perspective of a teacher. But, the method in its current form seems somewhat impractical, as it requires training a teacher from scratch for a particular task and then training the student. It seems like it would be much more practical to train the teacher across multiple tasks and measure generalization to new tasks. Can the teacher generalize to new settings? Regarding reproducibility, will code be released?  Experiments: How does the proposed approach compare to reward-augmented max likelihood [1] and distillation [2]? [1] considered the same problem of optimizing for a non-differentiable reward function. Model distillation [2] also has a similar flavor of a teacher/student where the teacher is an ensemble of models and the student is a single model. One crucial piece of information that is missing from the experiments is -- what data is used for the development set, and how was this data used for the comparisons? The concern is that, perhaps the proposed method is seeing a benefit because it is using the validation set in a more direct way, rather than simply being used to find good hyperparameters. If the dev set is only used by the prior methods for choosing hyperparameters, how do these prior methods perform when training with both the training and development data (after finding good hyperparameters by holding out the dev set)? Additionally, when experimenting with multiple student architectures, was a single teacher trained across all student architectures? Or was it trained individually for each architecture? Or was the teacher trained on one student architecture and evaluated on another? This bit of information doesn't seem to be stated in the paper. Showing generalization to new student architectures would be nice.  Related Work: It would be nice (but not crucial) to discuss the relationship with works on learning to optimize [3,4]. These works are related in that they learn a meta-learner/teacher that learns how to update learner/student. Unlike this work, that learns a loss function, these prior works learn to directly output the weight update to be applied to learner/student. There is also prior work in learning-to-learn that learns critics or loss functions across multiple tasks [5,6,7]. Of course, one big difference with this work and these prior works is that these prior works typically train over multiple tasks or multiple architectures, evaluating generalization to new settings. This paper also seems to be related to automatic curriculum generation, e.g. [8].  Minor comments: - The notation for y in Section 4.1 is inconsistent. What is the difference between y, y with an arrow, and y' with an arrow?  [1] Reward Augmented Maximum Likelihood for Neural Structured Prediction https://arxiv.org/abs/1609.00150 [2] Distilling the Knowledge of a Neural Network https://arxiv.org/abs/1503.02531 [3] Learning to Learn by Gradient Descent by Gradient Descent https://arxiv.org/abs/1606.04474 [4] Learning to Optimize https://arxiv.org/abs/1606.01885 [5] Meta-critic Networks https://arxiv.org/abs/1706.09529 [6] Domain Adaptive Meta-Learning https://arxiv.org/abs/1802.01557 [7] Evolved Policy Gradients https://arxiv.org/abs/1802.04821 [8] Automatic Curricula via Asymmetric Self-Play https://arxiv.org/abs/1703.05407