summaries: Motivated by exploring the ranking correlations of the existing RandomNAS in NASBench-201, this paper proposes EPS to improve the search efficiency and keep good ranking correlations by evolving the proxy search space (PS) in RandomNAS. Specially, EPS contains three stages: 1) training the supernet in PS, 2) validating the architectures among the PS and 3) evolving the PS by tournament selection with the aging mechanism. Furthermore, a model-size-based regularization is introduced in the selection stage. Experiments on some popular benchmarks demonstrate the effectiveness of the method.

summaries: This paper claims that random search-based NAS methods show a low ranking correlation among top-20% candidate architectures in the search phase. To address this issue, this paper proposes to introduce a proxy search space consisting of good architectures and evolve it using evolutionary algorithms. This paper also proposes a simple size regularization to help the NAS algorithm escape from the small architecture traps. The experimental results show that the proposed approach achieves competitive performance with baseline methods.

summaries: This paper proposes Evolving the Proxy Search Space (EPS) as a new RandomNAS-based approach. The goal is to find an effective proxy search space (PS) that is only a small subset of GS to dramatically improve RandomNAS’s search efficiency while at the same time keeping a good correlation for the top-performing architectures. EPS runs in three stages iteratively: Training the supernet by randomly sampling from a PS; Validating the architectures among the PS on a subset of the validation dataset in the training interval; Evolving the PS by a tournament selection evolutionary algorithm with the aging mechanism.

abstractText: Random-sampling Neural Architecture Search (RandomNAS) has recently become a prevailing NAS approach because of its search efficiency and simplicity. There are two main steps in RandomNAS: the training step that randomly samples the weight-sharing architectures from a supernet and iteratively updates their weights, and the search step that ranks architectures by their respective validation performance. Key to both steps is the assumption of a high correlation between estimated performance(i.e., accuracy) for weight-sharing architectures and their respective achievable accuracy (i.e., ground truth) when trained from scratch. We examine such a phenomenon via NASBench-201, whose ground truth is known for its entire NAS search space. We observe that existing RandomNAS can rank a set of architectures uniformly sampled from the entire global search space(GS), that correlates well with its ground-truth ranking. However, if we only focus on the top-performing architectures (such as top 20% according to the ground truth) in the GS, such a correlation drops dramatically. This raises the question of whether we can find an effective proxy search space (PS) that is only a small subset of GS to dramatically improve RandomNAS’s search efficiency while at the same time keeping a good correlation for the top-performing architectures. This paper proposes a new RandomNAS-based approach called EPS (Evolving the Proxy Search Space) to address this problem. We show that, when applied to NASBench-201, EPS can achieve near-optimal NAS performance and surpass all existing state-of-the-art. When applied to different-variants of DARTS-like search spaces for tasks such as image classification and natural language processing, EPS is able to robustly achieve superior performance with shorter or similar search time compared to some leading NAS works. Our code is available at https://github.com/IcLr2020SuBmIsSiOn/EPS

title: Understanding and simplifying one-shot architecture

authors:
Gabriel Bender
Pieter-Jan Kindermans
Barret Zoph
Vijay Vasudevan
Quoc Le

title: Understanding and simplifying one-shot architecture search

venue: In International Conference on Machine Learning,

authors:
Han Cai
Ligeng Zhu
Song Han

title: Proxylessnas: Direct neural architecture search on target task and hardware

venue: arXiv preprint arXiv:1812.00332,

authors:
Han Cai
Chuang Gan
Tianzhe Wang
Zhekai Zhang
Song Han

title: Once-for-all: Train one network and specialize it for efficient deployment

authors:
Xin Chen
Lingxi Xie
Jun Wu
Qi Tian

title: Progressive differentiable architecture search: Bridging the depth gap between search and evaluation

venue: In Proceedings of the IEEE International Conference on Computer Vision, pp. 1294–1303,

authors:
Yukang Chen
Tong Yang
Xiangyu Zhang
Gaofeng Meng
Xinyu Xiao
Jian Sun

title: Detnas: Backbone search for object detection

authors:
Xuanyi Dong
Yi Yang

title: One-shot neural architecture search via self-evaluated template network

title: Searching for a robust neural architecture in four gpu hours

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

title: Nas-bench-201: Extending the scope of reproducible neural architecture search

venue: arXiv preprint arXiv:2001.00326,

authors:
Zichao Guo
Xiangyu Zhang
Haoyuan Mu
Wen Heng
Zechun Liu
Yichen Wei
Jian Sun

title: Single path one-shot neural architecture search with uniform sampling

authors:
Liam Li
Ameet Talwalkar

title: Random search and reproducibility for neural architecture search

venue: arXiv preprint arXiv:1902.07638,

authors:
Chenxi Liu
Liang-Chieh Chen
Florian Schroff
Hartwig Adam
Wei Hua
Alan L Yuille
Li FeiFei

title: Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: Darts: Differentiable architecture search

venue: In Proc. of ICLR,

authors:
Zhichao Lu
Ian Whalen
Vishnu Boddeti
Yashesh Dhebar
Kalyanmoy Deb
Erik Goodman
Wolfgang Banzhaf

title: Nsga-net: neural architecture search using multi-objective genetic algorithm

venue: In Proceedings of the Genetic and Evolutionary Computation Conference,

authors:
Junran Peng
Ming Sun
ZHAO-XIANG ZHANG
Tieniu Tan
Junjie Yan

title: Efficient neural architecture transformation search in channel-level for object detection

authors:
Esteban Real
Alok Aggarwal
Yanping Huang
Quoc V Le

title: Regularized evolution for image classifier architecture search

venue: In Proc. of AAAI,

authors:
Tonmoy Saikia
Yassine Marrakchi
Arber Zela
Frank Hutter
Thomas Brox

title: Autodispnet: Improving disparity estimation with automl

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jon Shlens
Zbigniew Wojna

title: Rethinking the inception architecture for computer vision

authors:
Sirui Xie
Hehui Zheng
Chunxiao Liu
Liang Lin

title: Snas: stochastic neural architecture search

venue: arXiv preprint arXiv:1812.09926,

authors:
Yuhui Xu
Lingxi Xie
Xiaopeng Zhang
Xin Chen
Guo-Jun Qi
Qi Tian
Hongkai Xiong

title: Pcdarts: Partial channel connections for memory-efficient differentiable architecture

authors:
Antoine Yang
Pedro M Esperança
Fabio M Carlucci

title: Nas evaluation is frustratingly hard

venue: arXiv preprint arXiv:1912.12522,

authors:
Zhaohui Yang
Yunhe Wang
Xinghao Chen
Boxin Shi
Chao Xu
Chunjing Xu
Qi Tian
Chang Xu

title: Cars: Continuous evolution for efficient neural architecture search

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Chris Ying
Aaron Klein
Eric Christiansen
Esteban Real
Kevin Murphy
Frank Hutter

title: Nasbench-101: Towards reproducible neural architecture search

authors:
Jiahui Yu
Pengchong Jin
Hanxiao Liu
Gabriel Bender
Pieter-Jan Kindermans
Mingxing Tan
Thomas Huang
Xiaodan Song
Ruoming Pang
Quoc Le

title: Bignas: Scaling up neural architecture search with big single-stage models

venue: arXiv preprint arXiv:2003.11142,

authors:
Kaicheng Yu
Christian Sciuto
Martin Jaggi
Claudiu Musat
Mathieu Salzmann

title: Evaluating the search phase of neural architecture search

authors:
Arber Zela
Thomas Elsken
Tonmoy Saikia
Yassine Marrakchi
Thomas Brox
Frank Hutter

title: Understanding and robustifying differentiable architecture search

authors:
Arber Zela
Julien Siems
Frank Hutter

title: Nas-bench-1shot1: Benchmarking and dissecting oneshot neural architecture search

venue: arXiv preprint arXiv:2001.10422,

authors:
Miao Zhang
Huiqi Li
Shirui Pan
Xiaojun Chang
Steven Su

title: Overcoming multi-model forgetting in one-shot nas with diversity maximization

venue: Proc. of CVPR,

authors:
Barret Zoph
Quoc V Le

title: Neural architecture search with reinforcement learning

authors:
Barret Zoph
Vijay Vasudevan
Jonathon Shlens
Quoc V Le

title: Learning transferable architectures for scalable image recognition

venue: In Proc. of CVPR,

venue: Other works (Li  Chen et al. (2019b); Zhang et al

venue: Zela et al

venue: Works like Yu et al

title: 2019b), for CIFAR-100 and SVHN we use 16 number

authors: paper(Dong Yang

heading: 1 INTRODUCTION

text:
Neural architecture search (NAS) has been successfully utilized to discover novel DNN architectures in complex search spaces and outperformed human-crafted designs. Early NAS works like NASNet (Zoph et al. (2018)) and AmoebaNet (Real et al. (2019)) used reinforcement learning or evolutionary algorithms to search for the DNN architectures by training a substantial amount of independent network architectures from scratch. Although these searched architectures can deliver high accuracy, they come with tremendous computation and time costs. Therefore, researchers gradually shift their focuses to one-shot NAS, which is more efficient and can deliver satisfying outputs within a few GPU-days. There are two main types of one-shot NAS. One is the differentiable NAS (DNAS), such as Liu et al. (2019b); Cai et al. (2018); Xie et al. (2018); Dong & Yang (2019b); Xu et al. (2019); Chen et al. (2019a), which uses a continuous relaxation of the architecture representations and introduces architecture parameters to distinguish the architectures. The other is Random-Sampling NAS (RandomNAS), such as Li & Talwalkar (2019); Chen et al. (2019b); Zhang et al. (2020); Guo et al. (2019); Bender (2019); Yang et al. (2020). RandomNAS approaches typically have two phases: (1) Training phase: in each iteration, RandomNAS randomly samples one architecture or a set of architectures and updates their shared weights in the supernet; (2) Search phase: after supernet training, the desired architectures are selected based on their performance ranking on the validation dataset
using inherited weights from the supernet, which is called weight-sharing performance. Finally, the selected architectures will be retrained from scratch to get their actual (retrained) performance for deployment. Compared to DNAS, RandomNAS usually consumes less GPU memories by partially updating the weights. Also, it generates multiple target architectures, while DNAS generally retrieves a single architecture based on the maxima of the representation distribution. There are, however, two major drawbacks preventing RandomNAS from achieving higher search efficiency. First, although RandomNAS achieves a promising ranking correlation between the weight-sharing estimation and the retrained performance over all architecture candidates, it delivers a low ranking correlation among “good" architectures (e.g., top-20%-performing architectures among the search space) that researchers are more interested in. Second, by following the RandomNAS approach, smaller network architectures (with less parameters) tend to converge faster than the larger ones, which significantly degrades the ranking correlation.
To address the drawbacks of RandomNAS, we first introduce a proxy search space(PS): a subset of the architectures flexibly sampled from the global search space(GS) to study the features of RandomNAS. We then evaluate the RandomNAS with the proposed PS using the NASBench-201 benchmark (Dong & Yang (2020)) and notice two interesting phenomena: (1) When uniformly sampling a PS from a global search space, RandomNAS in the PS will maintain a similar ranking correlation with the one in the GS, even the size of the PS is extremely small (e.g. 16 architectures). (2) If the PS consists of the “good" architectures, the PS-based search can significantly improve the ranking correlation among the PS compared to the RandomNAS trained in GS and validated in PS.
Based on these observations, the PS constructed from “good” architectures can help overcome the first drawback of RandomNAS to search for more promising architectures. So, the remaining question is how to find a suitable PS containing sufficient “better" architectures? In this paper, we consider it a natural selection problem and solve it by the evolutionary algorithm. The architectures in the initial PS are iteratively evolved and gradually upgraded, while the average ranking of the architectures in the PS is improved. Meanwhile, it also helps improve the ranking correlation of the PS. We propose a new RandomNAS approach, named Evolving the Proxy Search Space (EPS). EPS runs in three stages iteratively: Training the supernet by randomly sampling from a PS; Validating the architectures among the PS on a subset of the validation dataset in the training interval; Evolving the PS by a tournament selection evolutionary algorithm with the aging mechanism. In this way, EPS gradually includes more high quality architectures in the proxy search space, while improves its ranking correlation. To solve the second issue in which smaller architectures converge faster than larger ones, we introduce a simple model-size-based regularization in the final selection stage. Our result on NASBench-201 shows 17.2% improvement in ranking correlation measured by Spearman’s ρ by adding the regularization on NASBench-201.
In the experiments, we demonstrate that EPS delivers a near-optimal performance on NASBench201. We also extend the EPS on DARTS search space. By using the 5-search-runs measurement, EPS demonstrates a robust search ability compared with the previous works in 8 hours search time with little hyper-parameters fine-tuning effort. Also, EPS is evaluated in 4 DARTS sub search space (Zela et al. (2019)) using 3 datasets, on which DARTS easily fails. EPS surpasses the DARTSADA and DARTS-ES on most cases and can often find the global state-of-the-art architectures. EPS also shows a high performance on a language modeling task which consolidates the generalization ability and robustness of EPS.

heading: 2 RANDOMNAS ON NASBENCH-201

text:
In this section, we present two major drawbacks we found in the existing RandomNAS methods and investigate the ranking correlation using a proxy search space. Proxy search space (PS) is a subset of the global search space (GS) and the RandomNAS is more flexible to be analyzed by training in the PS including different architectures from the GS. We run the following experiments on NASBench201, which is a unified and fair benchmark designed for NAS algorithm evaluation and contains the ground-truth architecture accuracy on three datasets. The GS of NASBench-201 contains 15,625 architectures and they are constructed by 4 nodes and 5 associated operation options. (Please refer to Fig. 1 in Dong & Yang (2020) for details.)
2.1 RANKING CORRELATION IN DIFFERENT PROXY SEARCH SPACES
First, we want to study whether RandomNAS in a PS with architectures uniformly sampled from the GS can achieve a higher ranking performance than the RandomNAS in the GS. We design an experiment to train the RandomNAS in the PS with 10 different sizes from 24 to 213 independently and compare the results with the RandomNAS in the GS. The experiment for each scenario is conducted 10 times with different random seeds (10×11 total runs). We calculate the Spearman’s ρ ranking correlation between the RandomNAS validation loss and the ground-truth test accuracy with 16 architectures randomly sampled from each search space. Fig. 1a shows the raw data with ± one standard deviation error bar for different proxy search space. The ranking correlations are close and there is no significant difference among them through statistical testing with a Pearson coefficient as -0.10. We observe that when the architectures are randomly sampled from the global search space, RandomNAS in a PS maintains a similar global-ranking performance with RandomNAS in the GS.
Second, although RandomNAS trained in the GS achieves 0.783 Spearman’s ρ when validating in the GS, it shows a poor Spearman’s ρ (0.347) in top-20% of the GS (which consists of top-20%performing architectures ranked by ground-truth accuracy from the GS). Hence, we’d like to see if RandomNAS in a PS can have a better ranking performance. Fig. 1b shows another six different scenarios, demonstrating the ranking correlation differences when sampling from GS or PS. The first three are: the RandomNAS trained in the GS with Spearman’s ρ calculated for 64 architectures randomly samples from top-100% (global), top-60%, top-20% in the GS independently. The other three are from PS: the RandomNAS trained in a PS with 256 architectures sampled from top100%, top-60%, top-20% the from GS and Spearman’s ρ calculated with 64 architectures randomly sampled from the same PS. It shows that the RandomNAS in the GS achieves 0.783 (the blue line) Spearman’s ρwith the validation architectures sampled from global. However, it only achieves 0.347 (purple) with the validation architectures sampled from top-20%. On the contrast, the RandomNAS in the PS shows that its Spearman’s ρ curve under PS top-100% (orange, 0.785) is similar with the GS top-100%, which is consistent with the Fig. 1b results. However, the Spearman’s ρ curve of the PS top-60% (red, 0.687) and top-20% (brown, 0.607) both surpass the GS top-60% (green, 0.530) and GS top-20%. We suspect that such an improvement may be due to the better constructed PS sampled from top performing GS, an insight we will take advantage of in designing our new solution below.

heading: 2.2 SMALL ARCHITECTURES GAIN LOWER LOSS

text:
We observe another drawback of RandomNAS in that smaller architectures inside the supernet converge faster and tend to gain lower validation loss than the larger ones in the early training phase.
Fig. 2a shows the validation losses of 4000 architectures from RandomNAS in the GS after training. Although the architectures with a model size of 1.05Mb achieves higher accuracy (queried from the ground truth in NASBench201), the Pareto front (shown as orange dots) only reaches the 0.83Mb architecture. One possible explanation is that the light-weight operations converge faster. Fig. 2b shows the average size of the validation Pareto front among 320 architectures in each validation interval. We can see that the average sizes of the Pareto-front architectures are small in the early search stage and will increase after more iterations, meaning that the larger architectures gradually begin to converge.
3 OUR METHOD
Algorithm 1: EPS Initialize a supernet Initialize an empty architecture population queue,Q_pop Initialize an empty history for i = 1, 2, · · · , P do
new_architecture← RandomInitArch(supernet)
Enqueue(Q_pop, new_architecture) end sample_set← RandomSample(Q_pop, S ) for i = 1, 2, · · · , T _iter do
architecture ← RandomSample( sample_set, 1 ) TrainOneBatch(supernet, architecture) if i% I_val = 0 then
for k = 1, 2, · · · , S do architecture← sample_set [k] val_loss← ValOneBatch(supernet, architecture) architecture.loss← val_loss end Sort sample_set by architecture.loss in
ascending order history.append(sample_set) for k in 1,2...M do
new_architecture← Mutate(sample_set[k]) Enqueue(Q_pop, new_architecture )
Dequeue(Q_pop) end sample_set← RandomSample(Q_pop, S
) end
end selected_architectures←
ArchitectureSelection(history,T ) ArchitectureRetrain(selected_architectures)
Motivated by the observations in 2.1, we propose a simple yet effective random sampling approach, called EPS, to iteratively improve the architecture quality within the proxy search space. Our proposed solution can distinguish the “better" architectures with higher ranking correlation compared to the existing solutions using global search. Algorithm 1 shows the overall flow of EPS.
Main Hyper-parameters. (1) T_iter: the number of iterations for supernet training. (2) I_val: validation interval. (3) P : maximum population size, where the population is a proxy search space. (4) S: sample size (the number of sampled architectures from population). (5) M : the number of mutated architectures. The ablation study on the search hyperparameters will be discussed in Section 4.
Overall Flow.
EPS performs three major stages iteratively, as training, validation and evolving, followed by a final architecture selection and retraining. At the beginning, we build a supernet which contains all possible architectures in the search space and initialize Q_pop and sample_set. RandomInitArch uniformly select a architecture from the GS, and then the architecture is enqueued into the Q_pop. Q_pop is the proxy search space and sample_set is randomly sampled from Q_pop for the tournament selection. (1) Training. The supernet is trained by
a total of T_iter iterations. In each iteration, one architecture is sampled from the sample_set and its weights are updated in the weight-sharing supernet by gradient descent. (2) Validation. Every I_val iterations of training, the architectures inside the sample_set will be evaluated sequentially. In each validation iteration, one architecture is evaluated on a batch of the randomly sampled validation data, and its validation loss is recorded. We validate the architecture on a single batch instead of the whole validation dataset to find a trade-off between the search time and completeness of validation. (3) Evolving. The architecture inside sample_set are sorted in ascending order according to their latest loss values, and the top-M architectures are mutated and en-queued. Meanwhile, it removes M oldest architecture from Q_pop. The mutation operation is described in Appendix B. (4) Selection. After supernet training, we revisit the history of evolution and select all the “winner" architectures which get the lowest loss in the Q_pop in the each of last T validation intervals. However, if T is large, it still remains numerous number of “winners". Based on the observation from Section 2.2, we propose a simple size regularization: architecture.loss = architecture.loss+ αe−βsize(architecture)/max_size. max_size is the maximum architecture size in the global search space. The regularization imposes a penalty on small architectures, and the experiment results in Section 4 shows the significant improvement from the regularization. By adding the regularization , we select the one with the lowest regularized loss as the ideal architecture from winners.

heading: 4 ABLATION STUDY ON NASBENCH-201

text:
We first apply EPS on NASBench-201 (Dong & Yang (2020)) for an ablation study. Following the NASBench-201 settings, we construct a supernet contains the same searchable cells. The number of iterations for training is T_iter = 80000. The main hyper-parameters explored are the maximum population size P ∈ {64, 128, 256}, the sample size S ∈ {32, 64}, the mutation number M ∈ {1, 4}, and the val-
idation interval I_val ∈ {50, 100, 200}. The number of total settings we explore is 36. For the selection, we set T = 10000 mod I_val. In the first round, the “winner" architectures which get the lowest loss in each of the last T validation interval are selected. If a “winner" dominates several validation intervals, its last loss will be adopted as the criterion. In the second round, the size regularization is added to all the “winner" architectures and we select the one with the lowest regularized loss.
Size regularization. We use the validation-training split by NASBench-201 and analyze α and β in the size regularization (defined in Sec. 3). We compare the ranking correlation between the architecture’s regularized loss and the validation accuracy in the final validation interval of the 36 settings. Each setting runs five times. The average Spearman’s ρ (in 36 × 5 runs) controlled by α and β is shown in Fig. 3a. We explore the α ∈ [0, 4] and β ∈ [0, 8] with the step 0.1. The average Spearman’s ρ reaches 0.796 when α = 1.4 and β = 5.2. Compared to the original Spearman’s ρ 0.624, it gains 0.172 improvement with the size regularization.
Search results. By adding the regularization, the setting {P = 256, S = 64,M = 4, I_val = 50} achieves the best average validation accuracy of five found architectures. The detailed results of the 36 settings are in the Table 16. We compare the EPS with the RandomNAS (RSPS) and non-aging evolutionary algorithm (EA) in Table 1. RandomNAS follows the same training settings as the EPS and evaluates 4000 architectures on all the validation data after training. Since the RandomNAS in the benchmark used a 256-size data batch for validation of each architecture and only evaluated 100 architectures (please refer to Appendix B, Dong & Yang (2020)), our RandomNAS results higher. EPS surpasses the RandomNAS by a large margin. Our experiment results support the idea that by properly evolving the proxy search space, it can gain a better performance compared to the global RandomNAS. Also, size regularization (SR) alleviates the slower convergence issue of larger architectures and shows the improvements on both EPS and RSPS. For the non-aging EA, architectures with the highest running loss are removed out of Q_pop instead of the oldest ones. The results show that the non-aging EA is trapped by some poor-performance architectures. The underline reasons are: (1) A young architecture with high loss can be removed before well trained;
(2) An aged architecture survives in the population when it performs well in the early stage and produces many mutations, which may dominate the population and mislead the search direction. We also compare EPS with other state-of-the-art NAS algorithms in Table 2. It shows that EPS delivers a near-optimal results and surpasses NAS works referred in the original Benchmark regarding test accuracy on CIFAR-10 and generalized test accuracy on CIFAR-100 and Imagenet-16. We also plot the average ranking of the proxy search space(population) and the Spearman’s ρ of the population in Fig. 3b which shows that EPS successfully improve the average ranking and correlation Spearman’s ρ when training.

heading: 5 EXPERIMENTS ON DARTS SEARCH SPACE AND ITS VARIANTS

text:
To further illustrate the generalizability of EPS, we also evaluate it on the DARTS search space and its variants without finetuning effort. Following the previous works, we build the network by stacking searchable cells, which contain N nodes as a directed acyclic graph. We first apply EPS in the DARTS image classification search space and 4 different DARTS sub search spaces proposed by Zela et al. (2019). Then we extend the EPS to DARTS language modeling search space,
which is completely different from computer vision tasks.

heading: 5.1 DARTS IMAGE CLASSIFICATION SEARCH SPACE

text:
We use the consistent hyper-parameters settings from the NASBench-201 and transfer to the DARTS image search space. Utilizing the random sampling strategy, EPS directly searches on a supernet with 20 cells and 16 number of initial channels with less than 8GB GPU memories consumption. We follow the PC-DARTS evaluations of the searched architectures in five independent search runs, and the results are shown in Table 3. Among the 5 runs, we are able to achieve the lowest average test error compared to DARTS and PC-DARTS. In addition, we make an extensive comparison to more recent NAS works on CIFAR-10 using the best network discovered by EPS within the 5 runs. Also, we do the search follow the same settings on CIFAR-100, and the results are shown in Table 4. It shows that on CIFAR-10/100, EPS is on a par with the recent NAS works and is able to find state-of-the-art architectures.
We design another experiment to compare the EPS to RandomNAS on DARTS search space. The experiment is conducted with 8 cells and 16 number of initial channels for both search and validation. We randomly sample 27 architectures from the last validation interval (after 80,000 training iterations) for EPS. We train the architectures 3 times independently follow the DARTS validation settings. Furthermore, the Spearman’s ρ between the EPS(with size regularization) loss and the ground truth is provided. Follow the same EPS training settings, we train a global RandomNAS, and we show both Spearman’s ρ in Fig. 4. The randomly sampled 10 architectures average accuracy is plotted by the red line (95.15%). Two observations are made: 1) The architectures sampled from the final proxy search space surpass the random sampling baseline by a large margin (the average accuracy of samples 95.54% vs. 95.15%). 2) EPS delivers 0.68 Spearman’s ρ while RandomNAS performs worse (0.41) at distinguishing the difference between them. Note that due to the computational consumption, we could only afford a single repetition for the experiment.

heading: 5.2 DARTS IMAGE CLASSIFICATION SUB SEARCH SPACES

text:
In the work (Zela et al. (2019)), it proposed four sub search spaces from DARTS where original DARTS fails. The four search spaces are: S1: A search space contains the most confident 2 operations DARTS predicted on each edge. S2: A search space contains {3×3 SepConv, SkipConnect} on each edge. S3: A search space contains {3× 3 SepConv, SkipConnect, None} on each edge.
S4: A search space contains {3 × 3 SepConv, Noise} on each edge. Also, we noticed that RandomNAS tend to fail on the S4 which suggests the method is difficult to distinguishes the Noise operation from 3× 3 SepConv. Here, we use similar settings for the EPS. We use the DARTS maximum architecture’s size as max_size for the size regularization and set the size for Noise simply as negative 3× 3 SepConv size since it disrupts the information on its edge. Also, we notice that None operation is as a placeholder for the DNAS training and not in the final architecture. Thus, S3 is the same as S2 for EPS. The results are shown in Table 5. For each setting, we report the 3 found architectures mean and std follow Table 6 in Zela et al. (2019). EPS delivers better performance on the most of them. Even in the S4 where RandomNAS is fragile, EPS still performs robustly and achieves the better performance on 2 settings out of three compared to DARTS-ADA. EPS even find an architectures on CIFAR-10, S3 with 2.55±0.05 testing error, which is the state-of-the-art in the whole search space (In Table 12,S3#1).
5.3 DARTS PTB SEARCH SPACE
We also evaluate EPS on the Penn Treebank (PTB) dataset, which targets natural language processing tasks and is completely different from computer vision tasks. The performance evaluation usually uses the perplexity score, the lower, the better. We use the DARTS PTB search space for the recurrent cells. During the recurrent cell search, the hidden and embedding sizes are both set to be 850, and the hyperparameters settings are similar to the previous tasks. Please refer to the Appendix B.4 for the details. Since operations in the search space are weight-free, we use a simple selection strategy
by training the last 10000 iteration’s “winner" architectures for two epochs and evaluate the top-5 architectures for 20 epochs. So the training and selection take 8+4 hours (0.5 GPU day). After the search, we use the same settings as in DARTS to train the best networks for 3600 epochs. The best model discovered by EPS achieves a validation perplexity of 58.4 and a test perplexity of 56.27, which is on a par with the state-of-the-art approaches. The full comparisons are shown in Table 6.

heading: 6 CONCLUSION

text:
We showed that RandomNAS had performed poorly in ranking good architectures and suffered from the lower validation loss for small architectures. Based on the proxy search space, we observed the ranking correlation between the prediction and the ground truth among good architectures could be improved if using a proxy search space consists of good architectures. Hence, we proposed an efficient way to Evolve the Proxy Search Space (EPS). We also designed a simple size regularization to help RandomNAS-based algorithm jump out of the small architecture traps. In the NASBench201 experiments, EPS delivered a near-optimal solution and surpassed the existing methods. In the
extensive experiments on DARTS search spaces and its variants, EPS outperformed or was tied with the majority of recent NAS works. We believe such observations and insights as presented in this paper can be useful to the community for the future NAS research with better interpretability.

heading: A BACKGROUND AND RELATED WORK

text:
Neural Architecture Search. Neural Architecture Search has demonstrated its capability of automatic DNN architecture design and generated DNNs even better than handcrafted ones in several tasks like object detection (Chen et al. (2019b); Peng et al. (2019)), segmentation (Liu et al. (2019a)) and disparity estimation (Saikia et al. (2019)). Early works, such as Zoph & Le (2017); Real et al. (2019) use reinforcement learning or evolutionary algorithm to deliver optimal network structures in discrete search spaces but spend tremendous time on architecture searching and network training. Recently, more efficient methods, such as one-shot and weight-sharing NAS, are widely adopted by the community to reduce the network search time. Previously published works (Liu et al. (2019b); Cai et al. (2018); Xie et al. (2018); Dong & Yang (2019b); Xu et al. (2019); Chen et al. (2019a)) start to use differentiable representation for the search space and focus on optimizing the architecture search speed. Other works (Li & Talwalkar (2019); Chen et al. (2019b); Zhang et al. (2020); Guo et al. (2019); Bender (2019); Yang et al. (2020)) adopt discrete and random sampling strategies to explore architectures in a weight-sharing supernet. Recently, we have seen a rising concern about the evaluation and effectiveness of NAS (Yang et al. (2019); Yu et al. (2019); Zela et al. (2019)). Therefore, researchers started to launch benchmarks to provide fair and efficient NAS evaluations (Ying et al. (2019); Zela et al. (2020); Dong & Yang (2020)).
Random Sampling NAS. As one of the major approaches, random sampling NAS has been widely studied in recent years. Bender et al. (2018) proposed a one-shot network training strategy with the dropout of operations, which can be considered as an extreme case of the random sampling. Later, Li & Talwalkar (2019) explored the reproducibility issue of existing works and observed that the Random Sampling strategy creates a strong baseline on image classification and language modeling. Guo et al. (2019) proposed a strategy to combine the uniformly random sampling training and post-training evaluation by an evolutionary algorithm to deliver more efficient solutions for image classification. To improve the search latency, Yang et al. (2020) used a modified NSGA-III to generate the Pareto frontier with a fast search time; while Zhang et al. (2020) proposed a general approach to maximize the search diversity and enhance the ranking correlation for both DNAS and RandomNAS. Also, RandomNAS has shown its strength in developing hardware-efficient network designs (Cai et al. (2019); Yu et al. (2020)) Although most of the works have achieved the state-ofthe-art performance in their domains, few of them have investigated the ranking correlation between the prediction of architectures by RandomNAS and the ground truth, which is the keystone for the success of the RandomNAS as a ranking estimator.
Differences between EPS and other works. Although we use the tournament selection EA and aging mechanism for the EPS, which is also adopted by Real et al. (2019) (named regularized evolutionary algorithm), the motivation is different. We find the effectiveness of such a method for updating the proxy search space in a weight-sharing supernet while Real et al. (2019) took it as a start-from-scratch-training selection for independent architectures. Thus, EPS searching time is close to the time to train an architecture from the search space. However, Real et al. (2019) takes more than 3,000 GPU days to find an architecture that leads to a 9,000X consumption than ours. Also, we notice a recent work, the CARS(Yang et al. (2020)) uses a modified NSGA-III for the selection in the training of a supernet. The main differences are 1. EPS motivates by several observations of RandomNAS, which is very different from the CARS. 2. EPS uses a single object EA for minimizing the validation losses while CARS uses an NSGA-III based algorithm for two conflicting objectives (potentially): maximizing the performance while minimizing the architectures’ parameter size. 3. EPS updates a single architecture at one time while CARS updates multiple architectures and leads to higher GPU memories consumption. 4. EPS uses a batch of data for validation of each architecture while CARS uses the whole validation dataset for an architecture. Overall, EPS shows a higher accuracy on DARTS search space compared to CARS, and since CARS isn’t open-sourced we are unable to do the extensive experiments for it.

heading: B DETAILED SETTINGS

text:
B.1 NASBENCH-201
Our works are done on PyTorch(Paszke et al. (2019)). Works like Yu et al. (2018); Liu et al. (2019b) point out the global moving average of the batch normalization leads to fluctuation among the
weight-sharing architectures. In the EPS, we disable the learnable affine parameters in all the batch normalization layers. We strictly follow the NASBench-201 settings to split the CIFAR-10 training dataset into train/validation two parts(1:1). We use the SGD with the momentum as 0.9. The initial learning rate is 0.1. The weight decay is 4 × 10−5. The scheduler is the cosine annealing from 1 to 0. The training batch size is 128 and the validation batch size is 250(1% of the training data). The loss function is a label-smoothing(Szegedy et al. (2016)) cross entropy loss with αsmooth = 0.1 which contains more information of the prediction distribution than the one-hot criterion. The evolutionary algorithm only mutates the cells. The rule for mutation is: The current operation on the edge have P_opwise = 0.5 as the probability to be mutated to a new operation(include itself) on the same edge. For the data augmentation we only use the random cropping with padding 4, the random horizontal flipping and image normalization based on the training dataset statistics.
B.2 DARTS IMAGE CLASSIFICATION SEARCH SPACE
The settings are the same as B.1 except the mutation. Since the DARTS search space is larger than NASBench-201(1018 vs. 15,625), we adopt a simple yet conservative mutation rule. 1. The current operation on the edge have P_opwise = 0.2 as the probability to be mutated to a new operation(include itself) on the same edge. 2. If a node has the unconnected predecessors, one of its edges have P_opwise = 0.1 as the probability to switch to an edge linked the unconnected predecessors, and the operation on the new edge will be chosen randomly. We apply this settings through all DARTS-search-space series. For the search supernet, we choose 16 initial channels and 20 stacked cells. For training architectures from the scratch, we strictly follow Liu et al. (2019b) settings to generate a fairly comparable results.
B.3 DARTS IMAGE CLASSIFICATION SUB SEARCH SPACE
Follow Zela et al. (2019); Liu et al. (2019b), for CIFAR-100 and SVHN we use 16 number of initial channels and 8 cells when training architectures from scratch. For CIFAR-10 S1 and S3, we use 36 initial channels and 20 stacked cell. For S2 and S4 we use 16 initial channels and 20 stacked cell. During the EPS searching, we use 16 number of initial channels and 8 cells for the supernet of CIFAR-100 and SVHN. we use 16 number of initial channels and 20 cells for the supernet of CIFAR-10.
B.4 DARTS PTB SEARCH SPACE
We use the settings for search as Liu et al. (2019b) except: 1. We choose the embedding and hidden dimension of 850 for the RNN. 2. We run the experiment for 80,000 iterations with 128 training batch size and uses the {P :256, S:64, M :4, I_val:50} for EA. 3. The gradient clip is changed to 0.1. 4. We only use 1/4 validation dataset for the architecture validation. The settings for training an architecture from the scratch is exactly the same.

heading: C ADDITIONAL EXPERIMENT

text:
C.1 OPTIONAL APPROACHES ON NASBENCH-201
Here we discuss two optional methods and evaluated them on NASBench-201. The first one is the progressive search. The algorithm is shown in Alg. 2. The supernet is initialized with 3 searchable cells (N = 1 and total searchable cells are 3N ). Please refer to the Figure 1 in the NASBench-201 paper(Dong & Yang (2020) for the defination of N ). We gradually increase N by 1 in each growth interval (Gval = 15000 iterations) until Nmax = 5. The new cells are randomly initialized and stacked into the supernet. Other training settings are adopted from EPS. The second one is the EPS with multi layer perceptron(MLP). The algorithm is shown in Alg. 3. The idea is to use the MLP learn the losses of architectures in the currentQ_pop and to predict architectures in the GS. We use a 3-layer MLP with 100 as the embedding size. The input is the indexes of the operations on 6 edges. In MLPTrain, we use the architectures evaluated in current Q_pop as the input and their losses as the label to finetune the MLP for 20 epochs. In the MLPPredict, we use the current MLP to predict losses of 256 architectures randomly generated. Then it return the best predicted architecture. The five-run experiment results on NASBench-201 are shown in the Table 7. Original EPS outperforms
other two methods on CIFAR-10. Since Progressive EPS speeds up 1.5x, we considered it as a good trade-off between the performance and the speed.
Algorithm 2: Progressive EPS Initialize a supernet withN = 1 Initialize an empty architecture population queue,Q_pop Initialize an empty history for i = 1, 2, · · · , P do
new_architecture← RandomInitArch(supernet) Enqueue(Q_pop, new_architecture)
end sample_set← RandomSample(Q_pop, S ) for i = 1, 2, · · · , T _iter do
architecture ← RandomSample( sample_set, 1 ) TrainOneBatch(supernet, architecture) if i% I_val = 0 then
for k = 1, 2, · · · , S do architecture← sample_set [k] val_loss← ValOneBatch(supernet, architecture) architecture.loss← val_loss end Sort sample_set by architecture.loss in ascending order history.append(sample_set) for k in 1,2...M do
new_architecture←Mutate(sample_set[k]) Enqueue(Q_pop, new_architecture ) Dequeue(Q_pop)
end sample_set← RandomSample(Q_pop, S ) if i%G_val = 0 ANDN < Nmax then
N = N + 1 end
end end selected_architectures← ArchitectureSelection(history,T ) ArchitectureRetrain(selected_architectures)
Algorithm 3: EPS with MLP Initialize a supernet Initialize an empty architecture population queue,Q_pop Initialize an 3-layerMLP Initialize an empty history for i = 1, 2, · · · , P do
new_architecture← RandomInitArch(supernet) Enqueue(Q_pop, new_architecture)
end sample_set← RandomSample(Q_pop, S ) for i = 1, 2, · · · , T _iter do
architecture ← RandomSample( sample_set, 1 ) TrainOneBatch(supernet, architecture) if i% I_val = 0 then
for k = 1, 2, · · · , S do architecture← sample_set [k] val_loss← ValOneBatch(supernet, architecture) architecture.loss← val_loss end MLPTrain(MLP ,Q_pop) Sort sample_set by architecture.loss in ascending order history.append(sample_set) for k in 1,2...M do
new_architecture←MLPPredict(MLP ) Enqueue(Q_pop, new_architecture ) Dequeue(Q_pop)
end sample_set← RandomSample(Q_pop, S )
end end selected_architectures← ArchitectureSelection(history,T ) ArchitectureRetrain(selected_architectures)
C.2 4-RUN TRIAL ON DARTS PTB SEARCH SPACE
Similar to the work Li & Talwalkar (2019), we conduct an experiment to run EPS 4 times and compare the 300-epoch validation perplexity of the selected architectures. The results are shown in Table 8.

heading: D MORE DETAILS


summaries: This work seeks to efficiently learn new tasks by combining meta-RL and imitation learning (IL). Such a combination is a natural thing to try, as both lines of work improve sample complexity of learning a new task: meta-RL by leveraging experience on prior related tasks, and IL by leveraging demonstrations. Demonstrations also form a natural way of specifying a new task to the agent.

summaries: This work proposes PERIL, a method for combined Meta Imitation Learning and Meta Reinforcement Learning using context-based meta-learning. Given a set of demonstrations, a latent variable representing the desired task is inferred, and trajectories are generated conditioned on the inferred latent variable.  The data from the expert demonstrations and trajectories are used for meta-learning updates.

summaries: This paper introduces PERIL, a meta RL method that combines demonstration trajectories and trajectories collected by the policy, in order to adapt to a new task. To this end, the authors combine ideas from metaRL (specifically from PEARL (Rakelly et al. 2019) and Humplik et al (2019)) where a set encoder is used to encode trajectories to a latent vector describing the task, with imitation learning techniques by (a) training this encoder also with demonstrations (b) initialising the latent vector at test time by feeding demonstrations through the encoder, and (c) having additional losses inspired by metaIL techniques. The motivation is that using demonstrations allows us to learn tasks that are difficult otherwise, for example because the rewards (at test time) are sparse.

abstractText: Imitation learning is a natural way for a human to describe a task to an agent, and it can be combined with reinforcement learning to enable the agent to solve that task through exploration. However, traditional methods which combine imitation learning and reinforcement learning require a very large amount of interaction data to learn each new task, even when bootstrapping from a demonstration. One solution to this is to use meta reinforcement learning (meta-RL) to enable an agent to quickly adapt to new tasks at test time. In this work, we introduce a new method to combine imitation learning with meta reinforcement learning, Probabilistic Embeddings for hybrid meta-Reinforcement and Imitation Learning (PERIL). Dual inference strategies allow PERIL to precondition exploration policies on demonstrations, which greatly improves adaptation rates in unseen tasks. In contrast to pure imitation learning, our approach is capable of exploring beyond the demonstration, making it robust to task alterations and uncertainties. By exploiting the flexibility of meta-RL, we show how PERIL is capable of interpolating from within previously learnt dynamics to adapt to unseen tasks, as well as unseen task families, within a set of meta-RL benchmarks under sparse rewards.

authors:
Yan Duan
John Schulman
Xi Chen
Peter L Bartlett
Ilya Sutskever
Pieter Abbeel

title: Under review as a conference paper at ICLR 2017 RL 2 : FAST REINFORCEMENT LEARNING VIA SLOW REINFORCEMENT LEARNING

venue: Technical report

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

venue: Technical report,

authors:
Abhishek Gupta
Russell Mendonca
YuXuan Liu
Pieter Abbeel
Sergey Levine

title: MetaReinforcement Learning of Structured Exploration Strategies

venue: Advances in Neural Information Processing Systems, 2018-December:5302–5311,

authors:
Jan Humplik
Alexandre Galashov
Leonard Hasenclever
Pedro A. Ortega
Yee Whye Teh
Nicolas Heess

title: Meta reinforcement learning as task inference

venue: URL http://arxiv

authors:
Gerrit Schoettler
Ashvin Nair
Juan Aparicio Ojea
Sergey Levine
Eugen

title: Solowjow. MetaReinforcement Learning for Robotic Industrial Insertion Tasks

authors:
Mel Vecerik
Todd Hester
Jonathan Scholz
Fumin Wang
Olivier Pietquin
Bilal Piot
Nicolas Heess
Thomas Rothörl
Thomas Lampe
Martin Riedmiller

title: Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

title: Adaptive Control of Robot Manipulators With Uncertain Kinematics and Dynamics

authors:
Haozhe Wang
Jiale Zhou

title: Learning Context-aware Task Reasoning for Efficient Metareinforcement Learning KEYWORDS Multitask Learning; Deep Reinforcement Learning

authors:
Henry Zhu
Justin Yu
Abhishek Gupta
Dhruv Shah
Kristian Hartikainen
Avi Singh
Vikash Kumar
Sergey Levine

title: The Ingredients of Real-World Robotic Reinforcement Learning

heading: 1 INTRODUCTION

text:
Reinforcement Learning (RL) and Imitation Learning (IL) are two popular approaches for teaching an agent, such as a robot, a new task. However, in their standard form, both require a very large amount of data to learn: exploration in the case of RL, and demonstrations in the case of IL. In recent years, meta-RL and meta-IL have emerged as promising solutions to this, by leveraging a meta-training dataset of tasks to learn representations which can quickly adapt to this new data. However, both these methods have their own limitations. Meta-RL typically requires hand-crafted, shaped reward functions to describe each new task, which is tedious and not practical for nonexperts. A more natural way to describe a task is to provide demonstrations, as with meta-IL. But after adaptation, these methods cannot continue to improve the policy in the way that RL methods can, and are restricted by the similarity between the new task and the meta-training dataset. A third limitation, which both methods can suffer from, is that defining a low-dimensional representation of the environment for efficient learning (as opposed to learning directly from high-dimensional images), requires hand-crafting this representation. As with rewards, this is not practical for nonexperts, but more importantly, it does not allow generalisation across different task families, since each task family would require its own unique representation and dimensionality.
In this work, we propose a new method, PERIL, which addresses all three of these limitations, in a hybrid framework that combines the merits of both RL and IL. Our method allows for tasks to be defined using demonstrations only as with IL, but upon adaptation to the demonstrations, it also allows for continual policy improvement through further exploration of the task. Furthermore, we define the state representation using only the agent’s internal sensors, such as position encoders of a robot arm, and through interaction we implicitly recover the state of external environment, such as the poses of objects which the robot is interacting with. Overall, this framework allows for learning of new tasks without requiring any expert knowledge in the human teacher.
PERIL operates by implicitly representing a task by a latent vector, which is predicted during learning of a new task, through two means. First, during meta-training we encourage high mutual information between the demonstration data and the latent space, which during testing forms a prior for
the latent space and represents the agent’s task belief from demonstrations alone. Second, we allow further exploration to continually update the latent space, by conditioning on the robot’s states and actions during exploration of this new task. We model the latent space via an encoder, from which posterior sampling can be done to encode the agent’s current task belief. In essence, the encoder aims to learn an embedding which can simultaneously (i) infer the task intent, and (ii) output a policy which can solve the inferred task. During meta-training, PERIL is optimised end-to-end, by simultaneously learning both a policy and the embedding function upon which the policy is conditioned.
In our experiments we find PERIL achieves exceptional adaptation rates and is capable of exploiting demonstrations to efficiently explore unseen tasks. Through structured exploration, PERIL outperforms other Meta-RL and Meta-IL baselines and is capable of zero-shot learning. We show how our method is capable of multi-family meta-learning as well as out-of-family meta-learning by clustering distinct meta-trained latent space representations. As an extension, we also show how to use privileged information during training to create an auxiliary loss for training the embedding function, which helps to form a stronger relationship between the latent space and the true underlying state which defines the task. Supplementary videos are available at our anonymous webpage https://sites.google.com/view/peril-iclr-2021

heading: 2 RELATED WORK

text:
Meta-RL was conceptualised as an RNN-based task. Developed by Wang (2016) and Duan et al., the authors use RNNs to feed a history of transitions into the model such that the policy can internalise the dynamics. On another line, Finn et al. (2017) developed a learning-to-learn strategy, model agnostic meta learning (MAML), which meta-learns an initialisation that adapts the parameters of the policy network and fine tunes it during meta-testing. Although promising results in simple goalfinding tasks, MAML-based methods fail to produce efficient stochastic exploration policies and adapt to complex tasks (Gupta et al., 2018). Meta-learning robust exploration strategies is key in order to improve sample efficiency and allow fast adaptation at test time. In light of this, contextbased RL was developed with the aim of reducing the uncertainty of newly explored tasks. These map transitions τ collected from an unseen task into a latent space z via an encoder qφ(z|τ), such that the conditioned policy πθ(τ |z) can efficiently solve said task (Rakelly et al.; Wang & Zhou, 2020). An underlining benefit of decoupling task encoding from the policy is that it disentangles task inference from reward maximisation, whilst gradient-based and RNN-based meta-RL policies do this internally. An important remark regarding training conditions of the discussed meta-RL methods is that they are typically meta-trained using dense reward functions (Rakelly et al.; Wang, 2016; Wang & Zhou, 2020). These dense reward functions provide information-rich contexts of the unseen task. Considering that the ultimate goal is to allow agents to solve new tasks in the real world, adaptation during test time must be robust to sparse reward feedback (Schoettler et al., 2020).
In the context of RL, incorporating demonstrations has proven successful in aiding exploration strategies, stabilising learning and increasing sample efficiency (Vecerik et al., 2017). Learning expressive policies from a set of demonstrations requires a vast amount of expert trajectories (Ross et al.), particularly in high dimensional state-action spaces (Rajeswaran et al.). In contrast, Meta-IL can be implemented in meta-RL by conditioning the agent with expert trajectories. On the another hand, Zhou et al. propose a MAML-based meta-IL approach which averages objective across demonstrations. The latter learns to adapt at test-time by receiving binary rewards. A similar strategy was also developed by and Mendonca et al.. The caveats of these approaches remain that of traditional IL: (i) Imitating expert trajectories hinders the policy from doing better than the demonstrations; (ii) Cloning behaviours reduces flexibility and generalisation capacity.

heading: 3 HYBRID META-REINFORCEMENT AND IMITATION LEARNING

heading: 3.1 PROBLEM STATEMENT

text: We assume access to a set of tasks T ∈ p(T ), with each task represented as a partially observable Markov decision process (POMDP). Observations o are incomplete and only include measurements of the agent’s internal state. For example, a peg-in-hole task would include the robot’s (and therefore the peg’s) pose in the observation, but not the hole’s pose. The full state s contains both the robot’s internal state, and the state of the external environment. Our method involves an agent inferring z, a descriptor of a task which provides the information required to solve that task, such as a representation of s. Each task is then defined as T = {p(o0|z), p(o′, z′|o, a, z), r(o, a|z)}, with an unknown initial observation distribution p(o0|z), transition distribution p(o′, z′|o, a, z), and reward function r(o, a|z), where a is the action taken by the agent. By leveraging task beliefs, PERIL exploits enriched observational spaces (s ∼ {o ∩ z}) with the objective of closing said POMDP into a stable MDP form. With the aim of supporting continuous task inference through interaction, we construct z as a probabilistic embedding conditioned on a set of recently collected transitions, referred to as the context c. We define a contexton cTt = (ot, at, rt, ot+1) as a transition collected on task T at time-step t such that the context cT0:t (what we call c) denotes the set of accumulated contextons. During meta-testing, the agent then attempts to find the true posterior p(z|c) over the task belief z ∼ p(z|c), by conditioning on c. Demonstrations significantly reduce the search space in exploration whilst providing a natural means of communicating the task. Thus, access to expert trajectories provides information-rich context which can be exploited to pre-condition the policy via a prior over z. Subsequently, online adaptation through RL can further disambiguate z. In our approach, we leverage dual meta-learning objectives to perform hybrid adaptation to new tasks, with conditioning on both demonstration contexts and exploratory contexts, where we define a demonstration as a trajectory of T observations and actions d = {(ot, at)}Tt=1, and an adaptation episode τe as a trajectory of N steps τe = {(ot, at, rt, zt)}Nt=1: Primal Inference: The agent observes a set of k = 1, ...,K demonstrations from an expert set Ddemo := {dk} to form a context prior cdemo = {dk}Kk . Exploratory Adaptation: The agent explores with initialised context c ←− cdemo and adapts with sampled trajectories τ such that at time-step t, a hybrid context is formed c←− cdemo ∩ τ0:t. Primal task inference aims to provide long-term inference on a task belief: meta-trained inference from previously solved tasks. Using posterior sampling, exploratory adaptation provides a proxy for short-term memory: as the context is updated with recent transitions, structured exploration can adjust primal inference by reasoning about the uncertainty of z, where c := {cdemo∩(τTe , ..., τ∞Te )}.

heading: 3.2 PROBABILISTIC EMBEDDINGS FOR META-RL

text:
As we do not have access to the posterior p(z|c), we use variational inference methods to produce an approximation qφ(z|c). Through generative processes, we sample z ∼ qφ(z|c) and optimise φ by maximising a meta-objective conditioned on z. Variational Encoders (VEs) are capable of producing latent distributions which optimise an arbitrary task-dependent objective G(T |·). We average the expectancy over a set of tasks from p(T ) to formulate a meta-objective which rewards fast adaptation to unseen tasks (Eq. 1), where β controls the constraint for mutual information between the inferred variable z and context c. This information bottleneck filters down redundant information and compresses the z into a generalisable form. We set the prior p(z) to a unit Gaussian.
φ∗ = argmax φ
ET ∈p(T ) [ Ez∼qφ(z|cT ) [ G(T |z) + βDKL [ qφ(z|cT )||p(z) ]] (1)
Traditional meta-RL methods leverage RNN-based inference systems to produce latent features from a history of recently collected transitions. The problem with this approach is that learning from entire trajectories leads to massive variances which hinder the learning process (Duan et al.; Humplik et al., 2019). In theory recurrence is not strictly required, since any MDP is defined by a distribution of sequentially invariant transitions. In light of this, we exploit task encoders as suggested by Rakelly et al., where q(z|c) is modelled as a product of Gaussian Factors. Each factor ψφ(z|cn) parameterised by φ, is independently computed: ψφ(z|cn) = N (fµψ(cn), fσψ (cn)).
We build on top of probabilistic meta-RL methods derived by Rakelly et al.. Decoupling task inference from task completion allows efficient off-policy objectives to be optimised. Moreover, conditioning a policy with diverse task descriptors aims to generalise behaviours amongst different dynamics and rewards. This is generally not the case when employing MAML-based or RNN-based meta-training kernels to ”learn how to learn” new tasks, since policy gradients modify the policy parameters online which, in turn, can result in highly unstable and inefficient adaptation to new tasks, requiring up to hundreds of sample trajectories in unseen environments.
To that end, we employ meta-adapted maximum entropy RL, where a proxy for the policy π(a|s) is defined by including the task belief z as part of the observational space πθ(a|o, z), resulting in taskconditioned critic LTcritic and actor LTactor losses for the the Q-function Qθ(·) and the policy πθ(·) respectively (see A.1 for more details). As VEs are a form of generative processes, G(T |·) can be defined with the aim of recovering transition dynamics or reconstructing state spaces (Humplik et al., 2019). Instead, we optimise qφ(z|c) in a model-free manner as proposed in Rakelly et al.. Here, parameters φ can be optimised to maximise expected discounted rewards under a policy πθ(·|z) through reconstruction of Qθ(·, z). We fold LTcritic as a proxy for the task-dependent goal G(T |z):
G(T |z)←− LTcritic(z) (2)

heading: 3.3 CONDITIONING ON DEMONSTRATIONS

text:
By leveraging meta-learning, we suggest that the agent can learn to exploit the overlapping statistical information from within heterogeneous demonstrations belonging to different tasks, with the aim of inferring task embeddings z which can help improve exploration during test-time adaptation. In order to link demonstrations into the probabilistic meta-RL framework, we formulate an objective based on mutual information I(z; τ) between the demonstration trajectories and the latent space distribution p(z) (Eq. 3). This objective is similar to that used in Yu et al. with the exception that we adapt this to meta-imitation learning and off-policy RL instead of recovering reward functions.
I(z; τ) = Ez∼p(z),τ∼pθ(τ |z) [ log p(z|τ)− log p(z) ]
(3)
As direct access to the posterior p(z|τ) is not available, I(z; τ) is intractable. However, in order to adjust for this, we leverage a variational approximation to p(z|τ) using our task belief encoder qφ(z|τ). We assume access to a distribution of expert demonstrations pπE (τ |z). Additionally, we include further desiderata over the mutual information objective based on distributional matching:
Learning from demonstrations. Matching generated trajectories to those sampled from expert trajectories: minθ Ep(z)[DKL(pπE (τ |z)||pθ(τ |z))]. This secondary objective can be considered as trying to match the distribution of trajectories generated by the agent’s policy conditioned on z, to those from the expert policy (similar to BC). Since they share the same marginal distribution p(z), matching these distributions also encourages matching of the conditionals pπE (z|τ) and p(z|τ). Linking variational posterior. minθ Epθ(τ)[DKL(p(z|τ)||q(z|τ))]. Encourage qφ(z|τ) to approximate the true posterior p(z|τ) such that, given a new demonstration, the encoder properly infers the task. This objective acts as a regulator as it reduces over-fitting by penalising variance: optimal distributional mismatch is reached when the encoder produces constant information-less representations of z. The distributional matching objective can therefore be defined as in (4).
min φ −I(z; τ) + Ep(z)[DKL(pπE (τ |z)||p(τ |z))] + Ep(τ)[DKL(p(z|τ)||qφ(z|τ))] (4)
After several manipulation steps (see section A.1 for more details), we deconstruct this objective into a differentiable form and define two separate loss components LTbc (Eq. 5) and LTinfo (Eq. 6), where πb is the behavioural policy which collected the transitions (truncated importance sampling).
LTbc = −Eτ∼pTπE (τ),z∼qφ(z|τ)[log πθ(τ |z)] (5)
LTinfo = −Eτ∼pTπE ,z∼qφ(z|τ),τb∼πb(τ |z)
[ min ( log πθ(τb|z) log πb(τb|z) , 1 ) log qφ(z|τb) ] (6)
As LTinfo conditions the encoder, we aggregate it to the task-dependent goal G(T |z) (7).
G(T |z)←− LTcritic(z) + LTinfo(z) (7)

heading: 3.4 AUXILIARY SYSTEMS

text:
An inherent problem is that the agent generates predictions Qθ(o, a, z) based on unsupervised estimates z ∼ qφ(z|cT ). Whilst this is theoretically generalisable to any MDP, it brings large variances and instabilities during training because the agent must simultaneously (i) distinguish one task from another, and (ii) use the task beliefs to solve that task. A plausible solution to mitigate these instabilities is to exploit privileged information (a brief task descriptor such as the position and orientation of a door) during training, allowing the encoder qφ(z|cT ) to produce task beliefs z which provide succinct descriptors of T . Although this information is not available during testing, it could be made available, for example, if training is performed in a full-state simulation, or in the real-world but with extra environment instrumentation. In theory, a perfect task descriptor is that which provides sufficient information to close the POMDP. We model ground truth task descriptor b̂ for T as a vector b̂ ∈ Rv where v is the dimension of the task descriptor. During training we wish to condition the encoder to produce latent spaces z ∼ qφ(z|cT ) which, when mapped into the dimensions of b, can produce approximations of b̂. We use a VAE dλ(b|z) parameterised by λ as our auxiliary module, which produces vectors bµ, bσ ∈ Rv and optimise it via a log-likelihood maximisation objective (8).
LTaux = −E(b̂,c)∼T ,z∼qφ(z|c) [ log dλ(b = b̂|z) ]
(8)
Inference b ∼ dλ(b|z) from the trajectories in cT is decoupled from RL as gradients do not pass through the actor or the critic. This stabilises the encoder by having a fixed supervised target, and allows off-policy training. Leveraging auxiliary objectives for meta-RL was coined by Humplik et al. (2019), where the authors condition task inference solely on LTaux to train RNN-based policies. However, in PERIL we use simple descriptors such that these solely condition the encoder to converge towards a suggested direction. Thus, we extend G(T |z) with the auxiliary loss to produce the ultimate meta-objective (9). Details of the final computational graph is presented in Figure 8. By keeping the critic loss LTcritic in G(T |z), we give freedom for the encoder to perform unsupervised learning so that it can structure representations of z which represent different dynamics and rewards: if we solely base task inference on LTaux, a task such as screwing a bolt would output the same task belief as a task of inserting a peg, even though the tasks are fundamentally different.
G(T |z)←− LTcritic(z) + LTinfo(z) + LTaux(z) (9)

heading: 3.5 IMPLEMENTATION

text: Meta-training requires sampling from a set of transitions (ot, at, rt, ot+1) T t=0 corresponding to a distribution of tasks T ∈ p(T ) and optimising the meta-objective. We use task-specific replay buffers BT to store collected transitions from which we can randomly sample during training. Following primal inference objective outlined in section 3.1, task inference is initially conditioned via contextons retrieved from expert demonstrations. Consequently, we store a set of k demonstrations generated by an expert policy on task-conditioned demonstration buffers DT . This expert policy may consist of a human user or, in our case, a pre-trained SAC agent with access to the full state of the environment. Meta-training algorithms are presented in Algorithms 1 and its process is illustrated in Figure 2. Further details of the meta-training and meta-testing processes in section A.2. We augment DT with imperfect trajectories which lead to demo-like (and increased) sum of rewards. This allows PERIL to: (i) be robust to imperfect demonstrations and surpass performance from the expert; (ii) combat the overfitting nature of BC from a narrow set of k demonstrations. The agent must also exploit posterior sampling to update its task belief in order to accomplish continuous adaptation objectives. To this end, we use exploratory data to update the task belief during adaptation and set additional task-dependent encoder buffers X T which only stores recently collected contextons. This mitigates distributional shift between off-policy training and on-policy adaptation.

heading: 4 EXPERIMENTS

text:
We leverage probabilistic meta-RL methods to find values for z which reconstruct unobserved representations of a new task. To that end, we focus on validating PERIL in environments where the agent must explore in order to understand the dynamics, whilst also utilising sparse rewards to identify the task intent. Furthermore, neural networks, are particularly susceptible to abrupt discontinuities. Thus, it is of particular interest to leverage meta-training in situations as such. We devise
Algorithm 1 PERIL: Meta-training Require: Batch of T training tasks {Ti}i=1,..,T , Ti ∈ p(T )
1: Initialise πθ, Qθ, qφ, bλ and replay BTi , encoder X Ti and demo buffersDTi for T ∈ {Ti}i=1,..,T 2: while not done do 3: for each Ti, i ∈ {1, ..., T} do . Collect data 4: Initialise context from demonstration ci = {τE}, τE ∼ Di and clear encoder buffer 5: for k = 1,...,K do 6: Sample from context z ∼ qφ(z|ci) 7: Roll out policy τ ∼ πθ(τ |z). Add to Bi and Xi. Add to Di if ER[τ ] > ER[τE ] 8: Update context ci ←− {ci ∪ xi}, xi ∈ Xi 9: for n = 1, ..., Nt do . Training for Nt steps
10: for each Ti, i ∈ {1, ..., T} do 11: Sample buffers xi ∼ Xi, τE ∼ Di, bi ∼ Bi to form hybrid context ci = {τE ∪ xi} 12: Sample z ∼ qφ(z|ci) and compute losses Limi, Libc, Liactor Licritic, LiDKL , L i aux 13: Update parameters θπ, θQ, φ, λ via gradient descent
tasks which involve contact-rich interactions and boundaries, factors which are considered to require a higher level of perception of the environment (Zhu et al., 2020). More details on A.3.
The different task families tested in this study are illustrated in Figure 3. Within a task family, one single task is defined by the configuration of the environment, such as the pose of the object or goal. When leveraging privileged information, this pose is used as the task descriptor. For each task, demonstrations are provided from different robot initial poses, and the agent is then required to perform that task, from any new robot initial pose. We compare our proposed methods to other meta-RL and IL baselines. To ensure fairness in all tests, we used the same number of parameters during training. Specifically, MetaIL (Yu et al.), extends PEARL by conditioning exploration on demonstrations. We also use Noisy-BC as a baseline where the agent clones a demonstration with additional noise (20% of p(o0|·)) . We also consider the case where privileged information is unavailable, thus compare these baselines to both PERIL and PERIL-A. In PERIL we omit Laux.

heading: 4.1 PERFORMANCE

text: We evaluate the performance of each method over 5 task families (Figure 4) using k = 3 demonstrations per task. Since meta-learning requires trajectory-based adaptation, we record the averaged trajectory return after 3 policy roll-outs. The results indicate that, in contrast to other baselines, meta-learning without demonstrations through sparse reward feedback is particularly ineffective (PEARL). On the other hand, PERIL-based methods significantly outperform MetaIL and NoisyBC, especially in tasks from Peg2D or Reach2D/3D where the agent must explore efficiently. It is in these cases where the auxiliary module contributes greatest. On the other hand, in task families such as Stick2D and Key2D, a few contact-rich interactions with the environment can quickly help decipher the task. In this case, goal-oriented auxiliary targets are not as helpful.

heading: 4.2 INCREASING GENERALISATION

text:
Most meta-RL research involves adaptation to tasks within a single task family (Rakelly et al.; Duan et al.; Yu et al.). But in practice, we seek agents capable of interpolating from a diverse set of dynamics. To test the representational power behind PERIL we evaluate training and adaptation performance in multi-task family settings (Figure 5). Here, we train a single agent along each task from from within the 2D task families. TSNE plots of the latent space distribution created during metatesting reveals PERIL-based methods produce structured task beliefs during hybrid inference of new tasks. Moreover, this verifies PERIL’s long-term memory contribution. Last, it showcases PERIL performs structured exploration at the multi-task level, allowing the agent to efficiently switch from one macro policy to another and robustly inherit a diverse set of behaviours. Despite the effectiveness of privileged information in complex exploration tasks in single task family adaptation, auxiliary systems do not generally contribute as much during meta-multi-tasking. We believe that task inference is substantially more convoluted in the multi-task family case as discerning dynamics and intent requires more complex embeddings in the latent space than the position of the goal. We also test generalisation by adapting PERIL in out-of-task distributions (Figure 6), where PERIL interpolates from within meta-trained latent space representations to adapt to unseen dynamics.
Metrics during adaptation of unseen tasks reveal PERIL is substantially more efficient in exploration with respect to other baselines, and is capable of zero-shot learning (Table 1), where the context from the demonstration is sufficient to condition the agent to adapt to an unseen task. Note we record the K-shot as the average kth adaptation trajectory until successful completion of the new task.

heading: 4.3 ALTERATIONS ON K

text: We study the inherent meta-IL dependency over the number of per-task expert demonstrations k. We find that increasing the number of demonstrations per task slightly improves sample efficiency but does not alter the asymptotic capacity of PERIL (Figure 7). Improvement is small since, by augmenting the demonstrations online, PERIL is capable of self-inducing additional demos which improves sample efficiency.

heading: 5 CONCLUSION

text: We have introduced PERIL, a new method for meta imitation and meta reinforcement learning which builds a representation of a new task by conditioning on both demonstrations, and further exploration. This provides a framework where new tasks can be defined naturally by a non-expert, without requiring reward shaping or state-space engineering. Experiments across a range of tasks show our method is able to adapt not only to novel instances within a task family, but also to entirely novel task families, whilst doing so with superior data efficiency to a range of baselines. This provides the foundation for further theoretical extensions of this framework, such as learning from imperfect demonstrations, as well as further applications, such as real-world contact-rich robot manipulation.

heading: A APPENDIX

heading: A.1 MATHEMATICAL EXTENSIONS

heading: Probabilistic Kernel

text:
We adopt a MaxEnt RL formulation to define a meta-objective for task T ∈ p(T ) (10), where H(·|o) represents the observational entropy, and the temperature α controls the weighting term for the entropy-based reward.
π∗ = argmax π ∑ T ∈p(T ) Ec∼T [∑ t∈T Ez∼p(z|c),(ot,at)∼πT [ r(ot, at, z) + αH(π(·|ot, z) ]] (10)
Optimising the objective defined in (10) is possible by using the SAC soft policy iteration approach. The difference to standard SAC is that in our case, the observation is augmented by z resulting in task-conditioned critic (11) and actor (12) losses for the the Q-function Qθ(·) and the policy πθ(·) respectively. V corresponds to the frozen target value function (no gradients stored in the forward pass).
Notice we use the term BT to represent the distribution of transitions in task T , which is modelled by a replay buffer. Moreover, the over-line operator in z denotes that gradients are detached. You may seek further details on the derivation of actor and critic losses in Appendix B of (Haarnoja et al.).
LTcritic = Ec∼T ,z∼qφ(z|c),(o,a,r,o′)∼BT [ Qθ(o, a, z)− (r + V (o′, z)) ]2 (11)
LTactor = Ec∼T ,z∼qφ(z|c),o∼BT ,a∼πθ(a|o,z) [ DKL ( πθ(a|o, z) ∣∣∣∣∣ ∣∣∣∣∣exp(Qθ(o, a, z))Zθ(o) )] (12)
Where Zθ(o) is the normalising partition function, which is intractable yet has no effect on the computation of the gradients. In the computation of LTactor, the gradients for z are detached, allowing the policy loss to exploit the representation of the MDP passed on by the critic without conditioning the inference network qφ(z|c).
Observe in 11 we allow gradients from the inference process q : C φ−→ Z to pass onto the computation of Qθ(o, a, z) such that meta-optimisation of the latent variable z is conditioned on reconstructing the Q-function.

heading: Conditioning on Demonstrations

text:
We minimise the conditioned mutual information term by enduring the following deconstruction process.
min φ −I(z; τ) + Ep(z)[DKL(pπE (τ |z)||p(τ |z))] + Ep(τ)[DKL(p(z|τ)||qφ(z|τ))]
= min φ Ep(z)[DKL(pπE (τ |z)||p(τ |z))] + Ep(z,τ)
[ log p(z)
p(z|τ) + log p(z|τ) qφ(z|τ)
]
= min φ
Ep(z)[DKL(pπE (τ |z)||p(τ |z))] + Ez∼p(z),τ∼p(τ |z) [ log qφ(z|τ) ]
(13)
In order to optimise the two terms in (13), we define Lbc and Linfo as the leftmost and rightmost distribution expectancies respectively. In this definition, the expectancies are not differentiable and requires parametric manipulation. The first step is to adapted this objective into a meta-RL form, by approximating the conditional p(τ |z) with the policy pθ(τ |z). We define the first term Lbc(θ) as the conditional behavioural cloning loss.
Lbc = Ep(z)[DKL(pπE (τ |z)||pθ(τ |z))] (14)
By manipulation of the Kullback-Liebler divergence we obtain the following expansion.
min θ Ep(z)[DKL(pπE (τ |z)||pθ(τ |z)] = min θ Ez∼pθ(z|τ) [ log pπE (τ |z) log pθ(τ |z) ]
min θ Ez∼pθ(z|τ) [ log pπE (τ |z) log pθ(τ |z) ] = min θ Ez∼pθ(z|τ) [ log pπE (τ |z) ] −min θ Ez∼pθ(z|τ) [ log pθ(τ |z) ]
We then substitute policy πθ as the conditional distribution pθ(τ |z) and our variatonal inference approximation for the posterior qφ(z|τ):
min θ
Ez∼pθ(z|τ) [ log pπE (τ |z) ] −min
θ Ez∼pθ(z|τ)
[ log pθ(τ |z) ]
= min θ H(pπE (τ |z))− Ez∼qφ(z|τ)
[ log πθ(τ |z) ] Because the entropy term is not dependent on the parameters θ, we derive a lower bound objective which is identical to the maximum likelihood estimator for trajectory distributional matching in BC:
LTbc = −Eτ∼pTπE (τ),z∼qφ(z|τ)[log πθ(τ |z)] (15)
Note that in the final representation (15), expert trajectory distributions pTπE (τ) are conditioned by task T as would be the case during meta-training. To obtain a tractable form of Linfo we leverage generative processes to switch sampling distributions as we do not have direct access to the prior p(z) to sample z. To that end we sample from both expert distributions τ ∼ pTπE (τ) and the posterior approximation (encoder) z ∼ qφ(z|τ) to arrive to (16).
LTinfo = −Eτ∼pTπE ,z∼qφ(z|τ),τθ∼πθ(τ |z) [ log qφ(z|τθ) ]
(16)
Notice how in LTinfo we generate a set of trajectories τθ from a task belief sampled from expert demonstration trajectories. From this we aim to maximise the likelihood of matching the posteriors on z given τθ. In essence, this can be viewed as a two-player game: we want to match the z generated by demonstrations to the z generated by a policy which is also conditioned on the demonstrations.
In this form, LTinfo is incompatible with off-policy RL as τθ is sampled from the current policy πθ. We propose truncated importance sampling by sampling from the distribution which collected τθ (17). We use a behavioural policy πb(τ |z) which is updated every episode as a frozen copy of πθ.
LTinfo = −Eτ∼pTπE ,z∼qφ(z|τ),τb∼πb(τ |z)
[ min ( log πθ(τb|z) log πb(τb|z) , 1 ) log qφ(z|τb) ] (17)

heading: A.2 ALGORITHM DETAILS

heading: Computational Graph

text: Figure 8 illustrates the computational graph and training procedure of PERIL.

heading: Meta Training

text:
Algorithm 2 is a detailed extension of the meta-training algorithm presented in the main text.
Algorithm 2 PERIL: Meta-training Require: Batch of T training tasks {Ti}i=1,..,T , Ti ∈ p(T )
1: Initialise πθ, Qθ, qφ, bλ and απ , αQ, αφ, αb. 2: Initialise replay BTi , encoder X Ti and demo buffers DTi for T ∈ {Ti}i=1,..,T 3: while not done do 4: for each Ti, i ∈ {1, ..., T} do 5: Initialise context by sampling demonstrations ci = {τE}, τE ∼ Di 6: Xi ← ∅ . Clear task encoder to reduce on-policy mismatch 7: for k = 1,...,Kprior do . Demonstration-conditioned trajectories 8: Sample z ∼ qψ(z|ci) 9: Collect trajectories τ ∼ πθ(τ |z) and add to Bi and Xi
10: for k = 1,...,Kpost do . Exploration-conditioned trajectories via posterior sampling 11: Sample z ∼ qφ(z|ci) 12: τ ∼ πθ(τ |z), Bi ←− Bi ∪ τ 13: if ER[τ ] > ER[τE ] then 14: Di ←− {Di ∪ τ} . Demo augmentation 15: ci ←− {ci ∪ xi}, xi ∈ Xi 16: for i = 1, ..., Nt do 17: Sample R random training tasks from p(T ) 18: Clone behavioural policy πb ←− πθ . To apply TRIS 19: for each Ti, i ∈ R do 20: Sample recent trajectories xi ∼ Xi, τE ∼ Di, bi ∼ Bi 21: Form hybrid context ci = {τE ∪ xi} 22: Sample latent task descriptor z ∼ qφ(z|ci) 23: Compute losses Limi, Libc, Liactor Licritic, LiDKL , L i aux
24: φ←− φ− αφ∇φ ∑ i∈R(Licritic + LiDKL + L i mi + Liaux)
25: θQ ←− θπ − απ∇π ∑ i∈R Licritic
26: θπ ←− θQ − αQ∇Q ∑ i∈R(Liactor + Libc)
27: λ←− λ− αλ∇λ ∑ i∈R Liaux

heading: Meta Testing

text:
Algorithm 3 presentes the process behind meta-testing.
Algorithm 3 Meta-Test adapation
Require: Networks πθ, qφ, test task T ∼ p(T ), demonstration buffer DT , 1: Initialise context by sampling k demonstrations c = {τE}k1 ∼ DT 2: for r = 1, ..., R do 3: Sample z ∼ qφ(z|c) 4: Roll out policy πθ(a|o, z) for N steps and collect trajectory τ = {(ot, at, rt, ot+1)Nt=1} 5: Update context c← {c ∩ τ}

heading: A.3 ENVIRONMENTS

text: 2D task families are set up with Pymunk and the 3D reach task families are set up from MuJoCo’s ReachEnv. Each task T ∈ p(T ) also contains an initial observational distribution p0(o)T . This distribution defines how the robot is initialised in the environment before collecting data. Since in this project we want to focus on meta-learning the dynamics of contact-rich tasks, we provide an initial distribution p0(o)T which allows the agent to start relatively close to the interaction. We summarise p0(o)T in Table 2, where P0 and θ0 represent the initial effector position in Pymunk units (PyU) (or meters if Reach3D) and angle (radians). Note that via inverse kinematics, the other links of the robot are automatically set.

heading: A.3.1 SPACES

text:
Each task has an accessible ground truth task descriptor b̂ ∈ R2 which describes the exact position in space of its goal G. In all 2D tasks, the base (origin) of the robotic manipulator is kept constant. The first component of the observational spaceO is defined by the normalised relative position from the tip of the end effector (P ∈ R2) to the origin Ω of the robot (18), where ω is the vector which defines the width and height in PyU of the visualisation domain (1280, 720). The second component of the observational space corresponds to the sine and cosines of the angle of the end effector θ. The action space A gives complete freedom to the effector and it includes its cartesian velocity (vx, vy) (PyU/s) and angular velocity ωθ (rad/s). Through inverse kinematics, the latter are translated back to the other links. Observation and action spaces are summarised in Table 3.
orel = O(P ) = P − Ω ω
(18)
By definition of the observational space, the agent has no information about the position or condition of the goal. Hence, during adaptation the robot must seek to find the goal and identify the task
dynamics using its own interactions as well as the information given by the demonstration. Note that in real world conditions, observation and action spaces could be augemented with kinematic components such as forces and torques at the end effector. However, in this simulated environment, we will show that deciphering the dynamics can be performed by using the aforementioned observational and action spaces.

heading: A.3.2 REWARD FUNCTIONS

text:
Naive dense reward functions are provided during meta-training of the critic. These reward functions are unshaped and give no information about the dynamics. In essence the reward function r(·) is defined as the absolute distance from the effector tip P to the goal G. The only exception is the Key2D task family, where reward functions are augmented with additional incentives to twist the handle to a desired angle α. Table 4 summarises the reward functions used in each task family.
On the other hand, we only provide binary sparse rewards r(·) ∈ [0, 1] to the task encoder. These are 0 in all cases except when the effector has reached its goal, where we provide a reward of 1.

heading: A.4 HYPER-PARAMETERS

text: Unless otherwise noted, the default hyperparameters for PERIL are as described below.

heading: Training Loop

text:
• Action space |a|: 3 • Observational space |o|: 4 • Latent space |z|: 8 • Maximum path length H: 60 • Prior collection steps Kprior: 240 • Posterior collection steps Kposterior: 240 • RL batch size: 256 • Encoder batch size: 64 • Number of expert demonstrations K: 3 • RL sparse rewards: False • Task belief sparse rewards: True • Maximum context size: 256 • Number of pre-training steps: 1000 • Number of training steps: 1000 • Number of tasks to average gradients in meta-training R: 16 • Number of tasks to collect data from each epoch: 32 • Number of Epochs: 100 • Discount factor γ: 0.98 • Entropy temperature: 0.1 • Soft target value = 0.005 • Kullback-Liebler divergence weighting wKL: 0.1 • Behavioural cloning weighting wbc: 0.5
• Mutual information weighting wmi: 0.2 • Auxilliary loss weighting waux: 4.0

heading: Actor

text:
• Hidden layer size h: 256 • Architecture: [|o, a, z|, h, h, 1] • Learning rate απ: 3× 10−4
• Output: Bounded action [-1,1] via tanh

heading: Critic

text: • Hidden layer size h: 256 • Architecture: [|a, z|, h, h, 1] • Learning rate: αQ: 3× 10−4

heading: Encoder

text: • Latent size: d • Input affinity architecture: [|o, a, r, o|, 256, 256, d] • Output affinity architecture: [d, 64, 64, |z|] • Learning rate: αq: 3× 10−4

heading: Belief Module

text: • Task descriptor dimension |b|: 2 • Architecture: [|z|, 32, 16, |b|] • Learning rate: αb: 3× 10−4