Transfer Learning with Neural AutoML
We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural AutoML that uses knowledge from prior tasks to speed up network design. We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks. On language and image classification tasks, Transfer Neural AutoML reduces convergence time over single-task training by over an order of magnitude on many tasks.
This paper applies both multi-task training and transfer learning to AutoML. The paper extends the ideas presented in the Neural Architectura Search (NAS) technique (Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017). The authors maintain the two-layer solution, with one network "the controller" choosing the architectural parameters for the "child" network which is used to solve the targeted task. The performance of the child network is fed back to the controller network to influence its results.   The novelty of this paper is in the way this two-layer solution is used. Instead of using the controller to decide the network for one task, multiple tasks are considered at once during the search phase (multi-task training). In addition, once the network is trained for a series of tasks, transfer learning is used to train the controller network to guide the design of the child network. With this technique in place, the accuracy performance of the child network achieves close to state-of-the-art results in a lot fewer iterations/trails in the search space, considerably reducing the time it takes to find the design of a performant network.  The results of the paper discuss two different types of tasks: NLP text classification and image classification. The NLP side of the analysis is a lot richer than the image classification. There are 21 tasks that are split randomly into two groups: 8 tasks used for the multi-task pre-training, while the rest of the tasks are used in the transfer learning setting. The 8-task pre-trained model is transferred to each of the remaining tasks individually.  The results show that the number of iterations for finding the top-10 best models is drastically reduced even when doing multi-task transfer learning for the neural architecture search. The performance of the top-10 models is on-par with state of the art numbers achieved by manually-designed network, which is a really nice result.  There are several way in which the paper could improve. I will list them in the order of importance:  1. My main concern with the paper is in the discussion of the time cost and savings achieved by this technique. I would have liked to see a discussion on how long it takes to pre-train the network on the 8 tasks before actually transfering it to a new task. I understand that this time is amortized across training networks for a lot of tasks, but I think it is an important detail of the technique that shouldn't be ignored.  2. The image classification part of the paper, while it shows that the technique works for a different domain, hence hopefully it generalizes, is rather weak. I'm guessing the results are not as interesting also due to the smaller dataset and simpler task.  3. There are few parts in the paper that could be explained a bit better. In particular, it's not entirely clear to me what the NAML approach really is. I'm guessing in the NAML case, instead of a random search, the two-layer controller-child network is used with feedback for only one task being fed from the child to the controller. Is NAML really the approach in NAS? If not, what are the differences?  Nit-picks: The arrangement of the figures is a bit strange. In particular, there doesn't seem to be a relationship among the graphs in Figure 4. I would try to rearrange them without prioritizing on esthetics, but rather on flow and understanding. Same goes for Figure 1. which is discussed towards the end of the paper.  Overall, I find the main idea of the paper simple and neat and I enjoyed reading the paper.  -------------------------- AFTER REBUTTAL: I really like the idea of applying Transfer Learning to AutoML. I think both areas are hot right now and having one example of transfer learning applied to AutoML is valuable to the community. That being said, for this paper to be referred to in the future, it needs to have clear writing for the Experimental Section. Most of the questions that were posed in the reviews stem from not understanding the experimental section and that's a pity. I hope the authors will take the feedback from these reviews (including the clarifications provided in the author response) and include them in the paper.