Recently, Neural Architecture Search (NAS) has been successfully applied to multiple artificial intelligence areas and shows better performance compared with hand-designed networks. However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. Generally, the architecture for a new task is either searched from scratch, which is neither efficient nor flexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal. In order to tackle the transferability of NAS and conduct fast adaptation of neural architectures, we propose a novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS. T-NAS learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task. Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.  2019). However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. As shown in Figure 1, we get the architecture-0 on given dataset using a NAS method. Now what if there exist a new task? This drives us to ask: how to get a suitable architecture for a new task in NAS? Generally, there exist two naive solutions in handling multiple tasks. One of them (S1) is to search an architecture for a new task from scratch but it is inefficient and not flexible for practical application scenarios. Another solution (S2) is to borrow architecture from the ones searched on other tasks but it might be not optimal for the new task. Therefore, it is urgently needed to study the transferability of NAS for large-scale model deployment in practical application. It should be more desirable to learn a transferable architecture that can adapt to some new unseen tasks easily and quickly according to the previous knowledge.
To this end, we propose a novel Transferable Neural Architecture Search (T-NAS) method (the bottom of Figure 1  Finn et al. (2017), where a model learns the meta-weights that are able to adapt to a new task through a few gradient steps. Push it forward, it is also possible to find a good initial point of network architecture for NAS. Therefore, the T-NAS learns a meta-architecture (transferable 1 Under review as a conference paper at ICLR 2020 Dataset NAS Architecture-0
Summary:
Current neural architecture search (NAS) methods work in the mode of what is called S1 in this paper: given a dataset, search for a new architecture from scratch for that particular dataset. This paper proposes using MAML-style metalearning for learning meta-architectures across many meta-training tasks so that given a new test task (dataset) the meta-architecture is a few search steps away from a near-optimal one. Hence the name Transferable-NAS (T-NAS). 
The main method pretty much follows what one would expect for a MAML-style approach. During meta-training a number of tasks (datasets) are used to learn a meta-architecture and corresponding meta-weights (Algorithm 1) and then during meta-testing time, given a new task (dataset) use any NAS search technique starting from the meta-architecture and corresponding meta-weights. (The authors use DARTS as the NAS technique which by itself is the same bilevel optimization as MAML, but for NAS. This sets up a 4-level optimization problem during meta-training time!).
Experiments on two settings are presented: Few-shot learning and the more traditional supervised learning in NAS literature. 
Comments:
- I really like the premise of the paper. While there have been papers trying to leverage dataset-level features for NAS transfer via bayesian approaches, computing dataset level features is always a bit difficult. The nice part about the MAML is that the transfer is embedded into the representation itself. But I have a bunch of clarification questions which is quite possible is mostly due to my misunderstandings. So please bear with me:
1. Section 5.2.1: "On the Mini-imagenet
dataset, One{normal + reduction} cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16." What are the 5000 independent tasks? In 5.1 it is said that 64 training classes, 16 validation classes and 20 test classes are present. I took that to mean that tasks are different classes of images in the few-shot setting. Clearly that is not the case. What precisely is a task in the few-shot learning case?
2. Secton 5.3: "...we choose 10 tasks with 200-shot, 50-query, 10-way for each task...." Again, what precisely is a task in the supervised learning setting?
3. One curious question I had is how much does architecture search matter as opposed to just taking the meta-architecture found by Alg 1 and just pretraining it on large image datasets like ImageNet or bigger and then just finetuning on downstream tasks. Specifically if you look at Table 3 ResNet12 pretrained has really good performance (although its size is ~100 times that of the others in the table). Perhaps S2 or AutoMAML with pretraining is the best option?