Better Not to Use Vulnerability’s Reference for Exploitability Prediction

: About half of all exploit codes will become available within about two weeks of the release date of its vulnerability. However, 80% of the released vulnerabilities are never exploited. Since putting the same effort to eliminate all vulnerabilities can be somewhat wasteful, software companies usually use different methods to assess which vulnerability is more serious and needs an immediate patch. Recently, there have been some attempts to use machine learning techniques to predict a vulnerability’s exploitability. In doing so, a vulnerability’s related URL, called its reference, is commonly used as a machine learning algorithm’s feature. However, we found that some references contained proof-of-concept codes. In this paper, we analyzed all references in the National Vulnerability Database and found that 46,202 of them contained such codes. We compared prediction performances between feature matrix with and without reference information. Experimental results showed that test sets that used references containing proof-of-concept codes had better prediction performance than ones that used references without such codes. Even though the difference is not huge, it is clear that references having answer information contributed to the prediction performance, which is not desirable. Thus, it is better not to use reference information to predict vulnerability exploitation.

Introduction
According to the Skybox Research Lab 2019 vulnerability and threat trends, the number of vulnerabilities discovered in 2018 was about 12% higher than that in 2017 [1]. In addition, half of exploit codes became available within about two weeks of vulnerability's release date. Security administrators must fix vulnerabilities as quickly as possible [2]. However, a period of two weeks is generally too short to fix all vulnerabilities. Thus, it will be beneficial to patch vulnerabilities that are likely to be exploited early.
The Common Vulnerability Scoring System (CVSS) numerically indicates which vulnerabilities are more dangerous [3,4]. The CVSS was proposed in 2005 by the National Infrastructure Advisory Council (NIAC). It is an open and quantified framework. However, the rating criteria used by the CVSS are ambiguous. Thus, analysts' subjective opinions are inevitable. Some researchers have insisted that the CVSS score is not suitable for predicting the exploitability of a vulnerability [5][6][7][8][9][10][11]. In addition, due to the use of a fixed expression, the scoring method of the CVSS is hard to reflect the ever-changing security environment [11].
Recent studies have used machine learning or deep learning to predict vulnerability exploitation [11][12][13][14][15]. Machine learning is divided into supervised learning to train with answer data and non-supervised learning to train without answer data. Most vulnerability exploitation studies use supervised learning. They mainly use the National Vulnerability Database (NVD) and Exploit Database (E-DB) information as learning data [16,17]. The NVD is a US government-standard vulnerability management repository using the Security Content Automation Protocol (SCAP) [18]. It has various information that can be used as features of learning data such as Common Vulnerabilities and Exposures (CVE) description, reference, vendor, CVSS, Common Weakness Enumeration (CWE), and published date [16]. In previous studies, the data used to learn correct answer information were Open Source Vulnerability Database (OSVDB) and Symantec Worldwide Intelligence Network Environment (WINE) that had actual exploitation data [19,20]. However, their services have been terminated [21]. Thus, recent studies have used the E-DB in which Proof of Concept (PoC) codes showing how exploitation works are available [4][5][6]20].
This paper focuses on the problem of using reference information among many features for determining the exploitability of vulnerabilities. The reference is a list of URLs related to the CVE. The URL lists provide additional information that cannot be directly obtained from the NVD Database, such as vendor information, other vulnerability databases, analysis reports, related mailing lists, and patch information. Thus, the studies described above have treated the NVD reference as a normal feature like other features [3][4][5][6].
However, we found that some references had PoC codes. Thus, we performed an analysis to see if this had a negative impact on the prediction of vulnerability. To do this, we examined a total of 9226 reference hosts having 308,427 references and found that 1712 of these hosts contained PoC codes. We also analyzed how these hosts containing PoC codes might affect the prediction of vulnerability exploitation. As a result, hosts with PoC codes had a dominant influence on the vulnerability exploitation prediction. In other words, the use of reference host as a feature can be problematic in predicting the exploitability. Therefore, we conclude that it is better not to use reference as a feature for more accurate prediction of vulnerability exploitation.
The paper is organized as follows. Section 2 introduces related research that uses references as a feature in predicting the exploitation of a vulnerability. Section 3 explores the existence of PoC code in the NVD Reference that is mainly used as a feature in the CVE Exploit prediction. It then examines how many PoC codes actually exist. Section 4 explores the effect of the reference host on the CVE exploit prediction using machine learning. Concluding remarks are made in Section 5.

Related Works
This section introduces research that uses references as a feature in predicting the exploitation of a vulnerability using machine learning. Bozorgi et al. [3] have studied the possibility of exploitation of a vulnerability and how quickly the vulnerability will be exploited. They used vulnerability information of OSVDB and MITRE for feature extraction. Their study used the reference feature from the MITRE vulnerability information to convert it into binary information using the bag-of-words technique.
Edkrantz has also tried to predict if the vulnerability would be exploited by using five different machine learning algorithms: Naive Bayes, Liblinear, LibSVM linear, LibSVM RBF, and Random Forest [4]. They utilized E-DB and NVD because of OSVDB service's termination. They extracted host information from the NVD's reference URLs to use it as a machine learning feature. Sabottke et al. [5] have used vulnerability information publicly available on twitter to predict vulnerabilities that are likely to be exploited. They used Symantec WINE, Microsoft Security Advisories [22], E-DB for getting answer labels [17], and Twitter Text, Twitter Statistics [23], NVD, and OSVDB (dumped data since 2012) for vulnerability information.
Bullough et al. [6] have compared methodologies used by previous studies predicting the exploitation of vulnerabilities. They identified three drawbacks of previous studies and found that there was a problem in the predictive power. An interesting part of Bullough's work was that they used NVD data as a feature, but excluded E-DB (www.exploit-db.com), Milw0rm (www.milw0rm.com), and OSVDB (www.osvdb.com). However, their study did not identify how many reference sites had the answer information. They did not address the predictive power problem resulting from those references either.

CVE Exploit Prediction and PoC
As mentioned earlier, it is very useful to predict whether a vulnerability will be exploited. The most important information for prediction is the answer data indicating which CVE is exploited in the end. Since OSVDB containing such information has been shut down since 2016, only a few companies with their own exploit data can conduct CVE exploit prediction studies.
However, a number of researchers have undertaken studies of the risk of CVE assuming that CVE in E-DB has been exploited, which is considered as a reasonable assumption in academia [4][5][6]. Recent risk prediction models that can obtain vulnerability information from NVD and answer labels from E-DB are shown in Figure 1. They then applied various machine algorithms to look for appropriate algorithms and parameters to obtain the best prediction result. The core function of Exploit-DB is to provide PoC codes that can exploit CVE. PoC is a simple code snippet that can explain how to exploit a vulnerability. For instance, a PoC code shown in Figure 2 is to exploit CVE-2016-10033, a vulnerability that PHPMailer can run an arbitrary code [24,25]. Hackers can easily perform similar attacks if a PoC is available in the Exploit-DB.

Problem of Reference Inclusion for CVE Exploit Prediction
In order to understand the problem, it is necessary to know what kinds of information that NVD data provide. Figure 3a shows the information of CVE-2016-10033 on the NVD homepage [26]. For example, Current Description gives general information about the CVE and why and how the CVE is dangerous. In particular, the Reference section in a yellow box in Figure 3a provides several URLs, so-called references, related to CVE. These references provide additional information not directly available from the NVD Database, such as CVE's vendor information, other databases, analysis reports, related e-mails, and patch information.
Please note that some references contain PoC codes. Figure 3b shows PoC codes that can be found in the websites of the underlined references shown in Figure 3a [25,27]. Although omitted in Figure 3a, seclists.org, securityfocus.com, securitytracker.com, and legalhackers.com in the References also have PoC codes. Many CVE references, including this CVE, have references of Exploit-DB's website. Most of them provide PoC codes. Of course, when PoC codes become accessible through the Reference, they can be helpful to devise or defend against attacks utilizing these PoC codes.
However, a problem arises when using these references as training data to predict the likelihood of a vulnerability being exploited. The reason for this is that, as explained above, existing studies including the present research assume that a vulnerability is considered to be exploited if its PoC code is available in Exploit-DB. Therefore, if references containing PoC codes are used to predict a CVE's exploitability, it can be a serious error because training data would include the answer information. There exist some studies that excluded osvdb.com and milw0rm.com as well as exploit-db.com from training data since these web sites are believed to have actual exploit information [5,6].  The heuristic decision on whether to include or exclude references can be problematic because numerous references might have PoC codes. Therefore, this study conducted a full survey of NVD references to analyze how many references contained PoC codes and how these references affected CVE exploitation prediction.

Keyword Based Classification
In order to decide whether a web page contains PoC or not, a classification method is necessary. It is challenging to devise a deterministic algorithm because PoC codes are written in different languages. In some cases, codes and texts are mixed. It is also hard to apply deep learning technologies because there is no training or test set. Instead of devising an algorithm, we took a different approach. We defined PoC related keywords and used them for the decision. We looked at many PoC references and figured out that most references containing PoC codes also had the following keywords: Proof of Concept, P0C, Exploit/POC, # PoC, PoC download, Attack Script, or vuln code. This implies that when a security expert or hacker uploads his PoC codes, they put some explanations about the code and are most likely to use PoC related keywords.
We developed a Python script to visit all references and classified them according to PoC related keywords. We collected 123,454 CVEs from cvedetails.com and extracted 308,427 references. The number did not include any reference from exploit-db.com, osvdb.com, or milw0rm.com known to have PoC codes.
One important concern about references is that some of them may have PoC related keywords without actual PoC codes. This can be false-positive cases in our classification method. To remove these cases, we manually checked several references of all domains containing PoC keywords. We realized that all references from exchange.xforce.ibmcloud.com were false-positive cases because, while they provide information on whether related PoC is available or not, their web page format does not have any section for displaying PoC codes. Thus, we excluded this site from the references. For more correct classification, we checked manually all the remaining references and concluded that the number of references containing both PoC related keywords and PoC codes was 46,202. Thus, a total of 40,622 references were not accessible and remaining 221,603 references did not have PoC codes.
Please note that this is the minimum number of references having PoC codes. In other words, there can be some references that have PoC codes without having the related keywords. These are false-negative cases. Different from false-positive cases, we do not have to remove these false-negative ones. The reason for this is that the existence of such references supports our claim more strongly. That is, if we could find more references having PoC codes, it would help us diagnose the adverse impact of these references on CVE exploitability prediction. Therefore, as long as our claim is valid with the current classification method, there is no problem with the fact that there may be more undiscovered references containing PoC codes.

Feature Matrix
In order to verify our hypothesis that there is a reliability problem in predicting vulnerability exploitability by using references that contain PoC codes, we will first analyze previous studies' prediction models and their features. Among several studies that utilized reference information for exploitability prediction [3][4][5][6][7], we chose to follow Edkrantz's research for preparing feature set and answer labels [4]. This is because his data are still publicly available from nvd.nist.gov and exploit-db.com. Table 1 shows final features used in our study.
Note that we will use a new term, reference host, rather than the reference itself. It means domain names hosting related references. For example, https://www.securityfocus.com/bid/95108 is a reference of CVE-2016-10033 and its reference host is www.securityfocus.com. Previous studies also used reference hosts as input for prediction models. Based on whether a reference host has URLs containing PoC codes, there will be three kinds of reference hosts. The first is No-PoC host where reference hosts do not have any PoC codes. The second is PoC host where reference hosts have at least one PoC reference. The third is No-Access host where none of its references is accessible at present.

Data Set Processing
The total number of samples (CVE) of the data set was 123,454. A total of 22,179 CVEs were exploited samples and the remaining 101,275 CVEs were unexploited ones. Around 18% of CVEs were exploited. Unfortunately, datasets with such extreme class ratios are not preferable for training data because machine learning models are likely to be biased. A solution to this problem is the use of the under-sampling technique. This technique can randomly select a smaller number of samples from the bigger group to make the class ratio of the training dataset to be 1:1. As shown in Figure 4, we made 1:1 ratio training sets by under-sampling unexploited samples while maintaining a 2:8 ratio to form test sets to see how our model would work in real situations. In addition, to reduce the possible adverse effect by sampling bias, we divided the entire data set into a total of k training and test sets and tested them independently and used the average of these results. This method is called k-fold cross validation. We set k to 5 in our experiment.

Validation of Reference Host as Machine Learning Feature
To verify our hypothesis, we created four models with the same dataset and features except for reference hosts and used four machine learning algorithms; Random Forest, Linear SVC, Logistic Regression, and Decision Tree. The All host model constructs the feature matrix by including all reference hosts regardless of the existence of PoC codes, which is the same feature matrix that previous studies have used. The PoC host model includes PoC hosts only instead of all reference hosts. The reason for using this model is to see how the prediction performance changes when data containing correct answer are used. The No-PoC host model uses only No-PoC hosts in the feature matrix. Last, No host model does not use any reference host information for the feature matrix. Table 2 shows the average of the 5-fold cross validation experiment results of CVE exploitability prediction using the four machine learning algorithms (the prediction results of each test set of the 5-fold cross validation are included in the Appendix A).
There is no distinct difference among machine learning algorithms, but each algorithm showed the similar pattern as follows. The All host and the PoC host models showed better prediction results than the other two models. The averaged differences between the two groups are 1.11% in accuracy, 1.68% in precision, 1.90% in recall, and 1.84% in F1-score. Let us look carefully at how the inclusion of PoC hosts affected the results. Although the number of reference hosts of the All host model was approximately four times that of the PoC model, its prediction rate is marginally better than that of the PoC model. The averaged difference between the two model are 0.24% in accuracy, 0.34% in precision, 0.46% in recall, and 0.41% in F1-score. In contrast, the No-PoC host model having three times reference hosts showed lower prediction rates than the PoC host model. The averaged difference between the two model are 0.92% in accuracy, 1.40% in precision, 1.45% in recall, and 1.47% in F1-score. This result implies that the inclusion of PoC hosts plays a critical role in the vulnerability exploitability prediction. Therefore, we insist that PoC hosts should be excluded since they have the answer information. If the remaining reference hosts, No-PoC hosts, gave a positive effect on prediction rate, they could be included in the feature matrix. Unfortunately, the prediction result of the No-PoC host model was not much different from that of the No-host model. In the Random Forest, the No-PoC host model even showed lower performances in accuracy and precision. Therefore, it is more preferable not to use the reference host as a feature because reference hosts having answer information will ruin the validity of CVE exploitability prediction.

Conclusions
It is very important to predict whether a disclosed vulnerability will be exploited because it would be helpful to judge which vulnerabilities should be removed first. To this end, existing studies have used various information about CVEs provided by NVD as features of machine learning algorithms. Among them, Reference, which provides URLs related to the CVE, has been used as the main feature. However, many references might have PoC codes. This would mean that the input data include answer information.
We discovered that at least 46,202 references among a total of 308,427 references included PoC codes. We also figured out which hosts contained PoC codes, called PoC hosts. After applying a machine learning algorithm, we found that PoC hosts had a rather crucial influence on the vulnerability exploitation prediction. Moreover, the No PoC host model did not show better prediction performance than the No host model where any hosts were not used at all. Therefore, we conclude that it is better not to use Reference host as a feature.
As a future study, we will continue to investigate into the reason why the prediction performance difference between PoC host and No-PoC host models are not huge, and try to find other features that might falsely affect prediction results like PoC hosts. In addition, we will perform more specific prediction research on how exploitation prediction can change according to specific vendors or application fields like embedded systems since their exploitability prediction can be completely different from the prediction of the entire CVEs.


Meta Automatic Curriculum Learning

A major challenge in the Deep RL (DRL) community is to train agents able to generalize their control policy over situations never seen in training. Training on diverse tasks has been identified as a key ingredient for good generalization, which pushed researchers towards using rich procedural task generation systems controlled through complex continuous parameter spaces. In such complex task spaces, it is essential to rely on some form of Automatic Curriculum Learning (ACL) to adapt the task sampling distribution to a given learning agent, instead of randomly sampling tasks, as many could end up being either trivial or unfeasible. Since it is hard to get prior knowledge on such task spaces, many ACL algorithms explore the task space to detect progress niches over time, a costly tabula-rasa process that needs to be performed for each new learning agents, although they might have similarities in their capabilities profiles. To address this limitation, we introduce the concept of Meta-ACL, and formalize it in the context of black-box RL learners, i.e. algorithms seeking to generalize curriculum generation to an (unknown) distribution of learners. In this work, we present AGAIN, a first instantiation of Meta-ACL, and showcase its benefits for curriculum generation over classical ACL in multiple simulated environments including procedurally generated parkour environments with learners of varying morphologies. Videos and code are available at https://sites.google.com/view/meta-acl .

Introduction
The idea of organizing the learning sequence of a machine is an old concept that stems from multiple works in reinforcement learning [Selfridge et al., 1985, Schmidhuber, 1991, developmental robotics [Oudeyer et al., 2007] and supervised learning [Elman, 1993, Bengio et al., 2009, from which the Deep RL community borrowed the term Curriculum Learning. Automatic CL [Portelas et al., 2020] refers to teacher algorithms able to autonomously adapt their task sampling distribution to their evolving student. In DRL, ACL has been leveraged to scaffold learners in a variety of multi-task control problems, including video-games [Pathak et al., 2017, Burda et al., 2019, Salimans and Chen, 2018, multi-goal robotic arm manipulation [Andrychowicz et al., 2017, Colas et al., 2019, Cideron et al., 2019, Fournier et al., 2018 and navigation in sets of environments [Matiisen et al., 2017, Portelas et al., 2019, Mehta et al., 2019, Florensa et al., 2018, Klink et al., 2020. Concurrently, multiple authors demonstrated the benefits of Procedural Content Generation (PCG) as a tool to create rich task spaces to train generalist agents [Risi and Togelius, 2019, Justesen et al., 2018, Cobbe et al., 2019. The current limit of ACL is that, when applied to such large continuous task spaces, that often have few learnable subspaces, it either relies on 1) human expert knowledge that is hard/costly to provide (and which undermines how automatic the ACL approach is), or 2) it loses a lot of time finding tasks of appropriate difficulty through task exploration.
Given the aforementioned impressive results on training DRL learners with ACL to generalize over tasks (which extended the classical single-task scenarios [Mnih et al., 2015, Schulman et al., 2015, Lillicrap et al., 2016 to multi-tasks), we propose to go further and work on training (unknown) distributions of students on continuous task spaces, thereafter referred to as Classroom Teaching (CT). CT defines a family of problems in which a teacher algorithm is tasked to sequentially generate multiple curricula tailored for each of its students, all having potentially varying abilities. CT differs from the problems studied in population-based developmental robotics [Forestier et al., 2017] and evolutionary algorithms [Wang et al., 2019] as in CT there is no direct control over the characteristics of learners, and the objective is to foster maximal learning progress over all learners rather than iteratively populating a pool of high-performing task-expert policies. Studying CT scenarios brings DRL closer to assisted education research problems and might stimulate the design of methods that alleviate the expensive use of expert knowledge in current state of the art methods [Clément et al., 2015, Koedinger et al., 2013. CT can also be transposed to (multi-task) robotic training scenarios, e.g. when performing iterative design improvements on a robot, which requires to train a sequence of morphologically related (yet different) robots.
Given multiple students to train, no expert knowledge, and assuming at least partial similarities between each students' optimal curriculum, current tabula-rasa exploratory-ACL approaches that do not reuse knowledge between different students do not seem like the optimal choice. This motivates the research of what we propose to call Meta Automatic Curriculum Learning mechanisms, that is algorithms learning to generalize ACL over multiple students. In this work we formalize this novel setup and propose a first Meta-ACL baseline algorithm (based on an existing ACL method [Portelas et al., 2019]). Given a new student to train, our approach is centered on the extraction of adapted curriculum priors from a history of previously trained students. The prior selection is performed by matching competence vectors that are built for each student through pre-testing. We show that this simple method can bring significant performance improvements over classical ACL in both a toy environment without DRL students and on Box2D parkour environments with DRL learners.
Related Work. To approach the problem of curriculum generation for DRL agents, recent works proposed multiple ACL algorithms based on the optimization of surrogate objectives such as learning progress [Portelas et al., 2019, Matiisen et al., 2017, Mysore et al., 2019, Colas et al., 2019, diversity [Eysenbach et al., 2018, Jabri et al., 2019, Bellemare et al., 2016 or intermediate difficulty [Florensa et al., 2018, Racanière et al., 2020, OpenAI et al., 2019, Florensa et al., 2017, Mehta et al., 2019. All these works tackled student training through independent ACL runs, while we propose to investigate how one can share information accross multiple trainings. Within DRL, Policy Distillation [Teh et al., 2017, Czarnecki et al., 2019 consists in leveraging one or several previously trained policies to perform behavior cloning on a new policy (e.g. to speed up training and/or to leverage task-experts to train a multi-task policy). Our work can be seen as proposing a complementary toolbox aiming to perform Curriculum Distillation on a continuous space of tasks.
Similar ideas were developed for supervised learning by [Hacohen and Weinshall, 2019, Furlanello et al., 2018, Yim et al., 2017. In [Hacohen and Weinshall, 2019], authors propose an approach to infer a curriculum from past training for an image classification task: they train their network once without curriculum and use its predictive confidence for each image as a difficulty measure exploited to derive an appropriate curriculum to re-train the network. Although we are mainly interested in training a classroom of diverse students, section 4.3 presents similar experiments in a DRL scenario, showing that our Meta-ACL procedure can be beneficial for a single learner that we train once and re-train using curriculum priors inferred from the first run.
Parallel to this work, Turcheta et. al. [Turchetta et al., 2020] studied how to infer safety constraints (i.e. curriculum) over multiple DRL students in a data-driven way to better perform on a given single task. In their work, students are only varying by their network's initialization and their teacher assumes the existence of a pre-defined discrete set of safety constraints to choose from. By contrast, we consider the problem of training generalist students with varying morphologies (and networks initializations), with a teacher algorithm choosing tasks from a continuous task space.

Main Contributions.
• Introduction to the concept of Meta-ACL, i.e. algorithms that generalize curriculum generation in Classroom Teaching scenarios, and an approach to study these algorithms. Formalization of the interaction flows between Meta-ACL algorithms and (unknown) Deep RL student distributions.
• Introduction of AGAIN, a first Meta-ACL baseline algorithm which learns curriculum priors to fasten the identification of learning progress niches for new Deep RL students. • Design of a toy-environment and of a parametric Box2D Parkour environment featuring a multi-modal distribution of possible agent embodiments well suited to study Meta-ACL. • Analysis of AGAIN on these environments, demonstrating the performance advantages of this approach over classical ACL, including (and surprisingly) when applied to a single student.
2 Meta Automatic Curriculum Learning Framework Figure 1: In Meta Automatic Curriculum Learning (Meta-ACL), the objective is to leverage previous teaching experience to improve the curriculum generation of a new agent, whose embodiment and learning mechanisms have potentially never been seen before: the teacher has to generalize over students.
Black-box students The Meta-ACL framework assumes the existence of policy learners, a.k.a students, capable of interacting in episodic control tasks. These students are assumed non-resettable, as in classical ACL scenarios. Their optimization objective is the maximization of some performance measure P w.r.t the task (e.g. episodic reward, exploration). To make the framework problemindependant, we do not assume expert knowledge over the task space w.r.t the student distribution, e.g. task subspaces could be trivial for some students and unfeasible for others. The objective of Meta-ACL is precisely to autonomously infer such prior knowledge from experience in scenarios where human expert knowledge is either hard or impossible to use. Similarly, we consider a black-box teaching scenario, i.e. we do not assume the knowledge of which learning mechanisms are used by students (e.g. DRL agents, evolutionary algorithms, ...).
Automatic Curriculum Learning Given such a black-box student s to train on a continuous task space A, the purpose of an ACL algorithm is to sequentially sample (parameterized) tasks for s, such that the following evaluation metric, used by the experimenter, is maximized: with E the episode budget, and P E s,a the end performance of student s on task a (e.g. exploration score, cumulative reward). Since direct optimization of such a post-training performance is difficult, ACL is often approached using proxy objectives (e.g. intermediate difficulty). Given one such proxy objective, an ACL algorithm usually relies on observing episodic behavioral features of its student w.r.t to proposed tasks (e.g. episodic rewards), allowing to infer a competence status o ∈ O of its learner (e.g. progress regions in A) which conditions task sampling. This sequential task selection unrolled by ACL methods along a student's training can be transposed into a policy search problem on a high-level non-episodic POMDP. While interacting within this POMDP, an ACL policy Π(o, s) → ρ(A) proposes task distributions ρ(A) ⊂ A to its student, observes the resulting behavioral features to update its competence status o, and collects objective-dependant rewards (e.g. learning progress).
In practice, approaching this task-level control problem with classical DRL algorithms is challenging because of sample efficiency: an ACL policy has to be learned and exploited along interaction windows typically around a few tens of thousands of steps. This has to be compared to the tens of millions or sometimes billions of interaction steps necessary to train a DRL policy for robotic control tasks. For this reason, most recent ACL research has focused on reducing the teaching problem into a Multi Armed Bandit setup, which ignores the sequential dependency over student states implied in POMDP settings [Matiisen et al., 2017, Mysore et al., 2019, Colas et al., 2019, Portelas et al., 2019. Although out of the scope of this paper, the use of classical DRL approaches as Meta-ACL algorithms is worth investigating in future work.
Meta-ACL for Classroom Teaching. We now present the concept of Meta-ACL applied to a Classroom Teaching scenario, i.e. there is no longer a single student s to be trained, but a set of students with varying abilities (e.g. due to morphology and/or learning mechanisms) sequentially drawn from an unknown distribution S. The notion of meta-learning refers to any type of learning guided by prior experience with other tasks [Vanschoren, 2018]. In meta-RL, agents are learning to learn to act [Wang et al., 2016], i.e. their objective is to maximize performance on previously unseen test tasks after 0 or a few learning updates. In other words, it is about leveraging knowledge from previously encountered tasks to generalize on new tasks. We propose to extend this concept into Meta-ACL, that is, algorithms that are learning to learn to teach, i.e. they leverage knowledge from curricula built for previous students to improve the curriculum generation for new ones. More precisely, a Meta-ACL algorithm can be formulated as a function: with H the history of past K training trajectories τ s resulting from the scaffolding of K previous students with an ACL (or Meta-ACL) policy Π, and s K the current student. Given our formalization of ACL (see eq. 1), the experimenter's evaluation objective for Meta-ACL can be expressed as follows: max f s∼S a∼A P E s,a da ds.
As in the case of the ACL evaluation objective expressed in eq. 1, direct optimization of eq. 3 is difficult as it implies the joint maximization of multiple students' performance. In our experiments, we reduce the Meta-ACL problem to the sequential independent training of a set of new students by leveraging priors from previous student trainings (with the hope to maximize performance over the entire set). Figure 1 provides a visual transcription of the workflow of a Meta-ACL algorithm. While in our experiments we use a fixed-size history H of ACL-trained students (to make our experiments computationally tractable), H could be grown incrementally by collecting training trajectories online from Meta-ACL trainings.

A first Meta-ACL baseline: Alp-Gmm And Inferred Progress Niches
In this section, we present AGAIN (Alp-Gmm And Inferred Progress Niches), our proposed Meta-ACL algorithm, and connect it to the formalism described in section 2. We first give a broad overview of the approach and then provide detailed explanations of key components.
Overview Figure 2 provides a schematic pipeline of our Meta-ACL approach. Given a history H of previously trained students and a new student s K ∼ S to train, AGAIN starts by (1)   which first leverages a preliminary run with a high-exploration ALP-GMM curriculum generator, and uses student pre-testing to infer an expert curriculum combined with a low-exploration ALP-GMM.
using ALP-GMM, an existing ACL algorithm from Portelas et al. [2019] (chosen for its simplicity and its non-reliance on expert knowledge). After pre-training, it (2) challenges the student with a set of test tasks to construct a meaningful competence profile of the student, which we thereafter refer to as a Knowledge Component (KC) vector. This KC vector is then (3) used to select a previously trained student s i similar to s K from which the training trajectory τ s i is recovered. Based on τ s i , (4) AGAIN infers a set of curriculum priors C (i.e. promising task sub-spaces). Finally, (5) the training of s K can resume using a composite curriculum generator using both an expert curriculum derived from C (for exploitation), and ALP-GMM (for exploration).
1 -ALP-GMM ALP-GMM [Portelas et al., 2019] is a Learning Progress (LP) based ACL technique for continuous task spaces that does not assume prior knowledge. ALP-GMM frames the task sampling problem into a non-stationary Multi-Armed bandit setup [Auer et al., 2002] in which arms are Gaussians spanning over the task space. The utility of each Gaussian is defined with a local LP measure derived from episodic reward comparisons. The essence of ALP-GMM is to periodically fit a Gaussian Mixture Model (GMM) on recently sampled tasks' parameters concatenated with their respective LP. This periodically updated GMM used for task sampling can be seen as the evolving competence status o described in section 2. The Gaussian from which to sample a new task is chosen proportionally to its mean LP dimension. Task exploration happens initially through a bootstrapping period of random task sampling and during training through residual random task sampling.
2,3 -KC-based curriculum priors selection. For a new student s K , given its capabilities on the considered task space, how to selected the most relevant previously trained student from which to extract curriculum priors? This problem is closely related to knowledge assessment in Intelligent Tutoring Systems setups studied in the educational data mining literature [Vie et al., 2018, Vie, 2016. Inspired by these works, we use pre-tests to derive a Knowledge Component vectors KC pre ∈ R m for all trained students. Each dimensions of KC pre contains the episodic return of the student on the corresponding pre-test task. Given that we do not assume access to expert knowledge, we build this pre-test task set by selecting m tasks uniformly over the task space. We use the same task set to build a post-training KC vector KC post ∈ R m whose dimensions are summed up to get a score j s ∈ R, used to evaluate the end performance of students in H. After the initial pre-training of s K with ALP-GMM, curriculum priors can be obtained in 3 steps: 1) pre-test s K to get its KC vector KC pre s K , 2) infer the k most similar previously trained students in KC space (using a k-nearest neighbor algorithm), and 3) use the training trajectory τ s of the student with maximal post-training score j s among those k. In essence, this method is about re-using curriculum data from a similarly-skilled and successfully-trained student.

-Inferred progress Niches (IN).
Given that the KC-based student selection identified the training trajectory τ s i as the most promising for s K , and assuming ALP-GMM as the underlying ACL teacher used for s i , we can derive an expert curriculum from τ s i by first considering the ordered sequence of GMMs C raw that were periodically fitted along training: with T the total number of GMMs in the list and LP ti the Learning Progress of the i th Gaussian from the t th GMM. By keeping only Gaussians with LP ti above a predefined threshold δ LP , we can get a curated list C containing only Gaussians located on task subspaces on which s i experienced learning progress (i.e. curriculum priors). Given a GMM of C, a task is selected by 1) sampling a Gaussian proportionally to its LP ti value, and 2) sampling the Gaussian to obtain parameters mapping to a task. But how to decide which GMMs to use along the training of the new student s K ?
While the simplest way to obtain such a curriculum would be to start sampling tasks from the first GMM and step to the next GMM at the same rate than the initial ALP-GMM run, we propose a more flexible reward-based method. This method requires to record the mean episodic reward obtained by the previously trained student s i for each GMM of C (which can be done without additional assumptions or computational overhead). Given this, to select which GMM from C is used to sample tasks over time along the training of s K , we start with the first GMM and only iterate over C once the mean episodic reward over tasks recently sampled from the current GMM matches or surpasses the mean episodic reward recorded during the initial ALP-GMM run. In app. D, we show that this reward-based variant outperforms other potential methods. See app. B for algorithmic details. We name the resulting meta-learned expert curriculum approach Infered progress Niches (IN) 5 -Our proposed approach: AGAIN. Simply using IN directly for s K lacks adaptive mechanisms towards the characteristics of the new student (e.g. new embodiment, different initial parameters, ...), which could lead to failure cases where the expert curriculum misses important aspects of training (e.g. detecting task subspaces that are being forgotten). Additionally, the meta-learned ACL algorithm must have the capacity to emancipate from the expert curriculum once the trajectory is completed (i.e. go beyond C). This motivates why our approach combines IN with an ALP-GMM teacher after the initial pre-training. The resulting Alp-Gmm And Inferred progress Niches approach (AGAIN) samples tasks from a GMM that is composed of the current mixture of both ALP-GMM and IN. See appendix A & B for implementation details and pseudo-code algorithms.

Experiments and Results
We organize the analysis of our proposed Meta-ACL algorithm around 3 experimental questions: • What are the properties and important components of AGAIN? In this section we will leverage a toy environment without DRL students to conduct systematic experiments.
• Does AGAIN scale well to Meta-ACL scenarios with DRL students? Here we will present a new Parkour environment that will be used to conduct our experiments.
• Can AGAIN be used for single learners? Here we will show that it can be useful to derive curriculum priors even for a single student (i.e. without any student History H).

Considered baselines and AGAIN variants.
In the following experiments we compare AGAIN to variants where 1) we directly use the expert curriculum instead of combining it with ALP-GMM (IN condition), and 2) where we select a training trajectory at random (AGAIN_RND) or using the ground truth student distribution (AGAIN_GT). We compare these Meta-ACL variants to ACL approaches such as random curriculum generation (Random), ALP-GMM and either Adaptive Domain Randomization (ADR) [OpenAI et al., 2019] or an expert-made Oracle curriculum. See appendix C for details.

Analysing Meta-ACL in a toy environment
To provide in-depth experiments on AGAIN, we first emancipate from DRL students through the use of a modified version of the toy testbed presented in [Portelas et al., 2019]. The objective of this environment is to simulate the learning of a student within a 2D parameter space P = [0, 1] 2 . The parameter space is uniformly divided in 400 square cells C ⊂ P, and each parameter p ∈ P sampled by the teacher is directly mapped to an episodic reward r p based on sampling history and whether C is considered "locked" or "unlocked". Three rules enforce reward collection in P: 1) Every cell C starts "locked", except a randomly chosen one that is "unlocked". 2) If C is "unlocked" and p ∈ C, then r p = min(|C|, 100), with |C| the cumulative number of parameters sampled within C while being "unlocked" (if C is "locked", then r p = 0). Finally, 3) If |C| >= 75, adjacent cells become "unlocked". Given these rules, one can model students with different curriculum needs by assigning them different initially unlocked cells, which itself models what is "easy to learn" initially for a given student, and from where it can expand.
Results Instead of performing a pre-test to construct the KC vector of a student, we directly compute it by concatenating |C| for all cells, giving a 400-dimensional KC vector. This vector is computed after 20k training episodes out of 200k. To study AGAIN, we first populate our training trajectory history H by training with ALP-GMM an initial classroom of 128 students drawn randomly from 4 fixed possible student types (i.e. 4 possible initially unlocked cell positions), and then test it on a new fixed set of 48 random students. Because we assume no expert knowledge over the set of students to train, i.e. their respective initial learning subspace is unknown, ADR -which relies on being given an initial easy task -fails to train most students when given randomly selected starting subspace (among the 4 possible ones). By contrast, this showcases the ability of AGAIN to autonomously and efficiently infer such expert knowledge.
Varying classroom size experiment -An important property that must be met by a meta-learning procedure is to have a monotonic increase of performance as the database of information being leveraged increases. Another important expected aspect of Meta-ACL is whether the approach is able to generalize to students that were never seen before. To assess whether these properties hold on AGAIN, we consider the full student distribution of the toy environment, i.e. 400 possible student types. We populate a new history H by training (with ALP-GMM) a 400-students classroom (one per student type). We then analyse the end performance of AGAIN and IN on a fixed test set of 96 random students when given increasingly smaller subsets of H. The smaller the subset, the harder it becomes to generalize over new students. Results, shown in fig. 3 (right), demonstrate that both AGAIN and IN do have monotonic performance increasements as the classroom grows. With as little as 10% of possible students in the classroom, AGAIN statistically significantly (p < .001) outperforms ALP-GMM on the new student set, i.e. it generalizes to never seen before students. to study Meta-ACL with DRL students.

Meta-ACL for DRL students in the Parkour environment
To study Meta-ACL with DRL students, we present a Box2D Parkour environment with a 2D parametric PCG that encodes a large space of tasks (see fig.  4). The first parameter controls the spacing between walls that are positioned along the track, while the second parameter sets the y-position of a gate that is added to each wall. Positive rewards are collected by going forward. To simulate a multi-modal distribution of students well suited to study Meta-ACL, we randomize the student's morphology for each new training (i.e. each seed): It can be embodied in either a bipedal walker, which will be prone to learn tasks with near-ground gate positions, or a two-armed climber, for which tasks with near-roof gate positions are easiest. We also randomize the student's limb sizes which can vary from the length visible in fig.  4 to 50% shorter.

Results
In the following experiments our Meta-ACL variants leverage a history H built from a classroom of 128 randomly drawn Soft-Actor-Critic [Haarnoja et al., 2018] students (i.e. varying embodiments and initial policy weights) trained with ALP-GMM. We then compare ACL and Meta-ACL variants on a fixed set of 64 new students and report the mean percentage of mastered environments (i.e. r > 230) from 2 fixed expert test sets (one per embodiment type) across training. The KC vector is built using a uniform pre-test set of m = 225 tasks, performed after 2 millions agent steps out of 10. See appendix E for additional experimental details.
Qualitative view - Figure 5 (left) showcases the evolution of task sampling when using AGAIN to train a new student. Three distinct phases emerge: 1) A pre-training exploratory phase used to gather information about the student's capabilities, 2) After building the KC vector and inferring the most appropriate curriculum priors from H, AGAIN paces through the resulting IN curriculum while mixing it to ALP-GMM, and 3) AGAIN emancipates from IN after completing it.
Comparative analysis -As shown in figure 5 ( . This showcases the advantage of emancipating from the expert curriculum once completed. As in the toy environment experiments, when given randomly selected starting subspaces (since we assume no expert knowledge), ADR fails to train most students.

Applying Meta-ACL to a single student: Trying AGAIN instead of trying longer
Given a single DRL student to train (i.e. no history H), and if we do not assume access to expert knowledge, current ACL approaches leverage task-exploration (as in ALP-GMM). We hypothesize that these additional tasks presented to the DRL learner have a cluttering effect on the gathered training data, which adds noise in its already brittle gradient-based optimization and leads to suboptimal performances. We propose to address this problem by modifying AGAIN to fit this no-history setup and by allowing to restart the student along training. More precisely, instead of pre-testing the student to find appropriate curriculum priors in H, we split the training of the target student into a two stage approach where 1) the DRL student is first trained with ALP-GMM (with high-exploration), and then 2) we extract curriculum priors from the training trajectory of the first run and use them to re-train the same agent from scratch. Results. We test our modified AGAIN along with variants and baselines on a parametric version of BipedalWalker proposed in [Portelas et al., 2019], which generates walking tracks paved with stumps whose height and spacing are defined by a PCGencoding 2-D parameter vector. As in their work, we test our approaches with both the default walker and a modified short-legged walker, which constitutes an even more challenging scenario (as the task space is unchanged). Performance is measured by tracking the percentage of mastered tasks from a fixed test set. See App. F for a complete analysis. Figure 6 showcases our proposed approach on the short walker setup (with a SAC student [Haarnoja et al., 2018]). On this short walker scenario, mixing ALP-GMM with IN is essential: while IN end performances are not statistically significantly superior to ALP-GMM, AGAIN clearly outperforms ALP-GMM (p < 0.01), reaching a mean end performance of 19.0. The difference in end-performance between AGAIN and Oracle, our hand-made curriculum using privileged information who obtained 20.1, is not significant (p = 0.6).

Conclusion and Discussion
In this work we attempted to motivate and formalize the study of Classroom Teaching problems, in which a set of diverse students have to be trained optimally, and we proposed to attain this goal through the use of Meta-ACL algorithms. We then presented AGAIN, a first Meta-ACL baseline, and demonstrated its advantages over classical ACL and variants for CT problems in both a toy environment and in a new parametric Parkour environment with DRL learners. We also showed how AGAIN can bring performance gains over ACL in classical single student ACL scenarios.
Future work In future work, AGAIN could be improved by using adaptive approaches to build compact pre-test sets, e.g. using decision tree based test pruning methods, or by combining curriculum priors from multiple previously trained learners. While AGAIN is built on top of an existing ACL algorithm, developing an end-to-end Meta-ACL algorithm that generates curricula using a DRL teacher policy trained across multiple students is also a promising line of work to follow. Additionally, this work opens-up exciting new perspectives in transferring Meta-ACL methods to educational data-mining, e.g. in MOOC scenarios, given a previously trained pilot classroom, one could use Meta-ACL to infer adaptive curricula for new students.
Potential negative societal impact Meta Automatic Curriculum Learning exploits previous curriculum data of DRL students to better train new ones. As such, if previously trained students acquired unwanted biases, they could potentially be transferred and further amplified from the Meta-ACL training of new DRL students. This can have serious negative societal impacts if considering applications in socially impactful domains (e.g. deciding whether to give an insurance to someone). Further work will need to study how one can safely learn curriculum priors that are not harmful in sensitive application areas.

ALP-GMM Portelas et al. [2019] (MIT license) relies on an empirical per-task computation of
Absolute Learning Progress (ALP), allowing to fit a GMM on a concatenated space composed of tasks' parameters and respective ALP. Given a task a new ∈ A whose parameter is p new ∈ P and on which the student's policy collected the episodic reward r new ∈ R, Its ALP is computed using the closest previous tasks a old (Euclidean distance) with associated episodic reward r old : All previously encountered task's parameters and their associated ALP, parameter-ALP for short, recorded in a history database H, are used for this computation. Contrastingly, the fitting of the GMM is performed every N episodes on a window W containing the N most recent parameter-ALP. The resulting mean ALP dimension of each Gaussian of the GMM is used for proportional sampling.
To adapt the number of components of the GMM online, a batch of GMMs having from 2 to k max components is fitted on W, and the best one, according to Akaike's Information Criterion Bozdogan [1987], is kept as the new GMM. In all of our experiments we use the same hyperparameters as in Portelas et al. [2019] (N = 250, k max = 10), except for the percentage of random task sampling ρ rnd which we set to 10% (we found it to perform better than 20%) when running ALP-GMM. See algorithm 1 for pseudo-code and figure 7 for a schematic pipeline. Note that in the main body of this paper we refer to ALP as LP for simplicity (ie. LP ti in C from eq. 4 is equivalent to the mean ALP of Gaussians in ALP-GMM).

Algorithm 1 Absolute Learning Progress Gaussian Mixture Model (ALP-GMM)
Require: Student policy s θ , parametric procedural environment generator E, bounded parameter space P, probability of random sampling ρ rnd , fitting rate N , max number of Gaussians k max Initialize s θ Initialize parameter-ALP First-in-First-Out window W, set max size to N Initialize parameter-reward history database H loop N times # Bootstrap phase Sample random p ∈ P, send E(a ∼ A(p)) to s θ , observe episodic reward r p Compute ALP of p based on r p and H (see equation 5) Store (p, r p ) pair in H, store (p, ALP p ) pair in W end loop loop # Stop after K inner loops Fit a set of GMM having 2 to k max kernels on W Select the GMM with best Akaike Information Criterion loop N times ρ rnd % of the time, sample a random parameter p ∈ P Else, sample p from a Gaussian chosen proportionally to its mean ALP value Send E(a ∼ A(p)) to student s θ and observe episodic reward r p Compute ALP of p based on r p and H Store (p, r p ) pair in H, store (p, ALP p ) pair in W end loop end loop Return s θ B AGAIN IN variants. In order to filter the list C raw (see eq. 4) of GMMs extracted from a training trajectory τ s selected in training trajectory history H into C and use it as an expert curriculum, we remove any Gaussian with a LP ti below δ LP = 0.2 (the LP dimension is normalized between 0 and 1, which requires to choose an approximate potential reward range, set to [−150, 350] for all experiments on Box2D locomotion environments (sec. 4.2 and sec. 4.3). When all Gaussians of a GMM are discarded, the GMM is removed from C. In practice, it allows to 1) remove non-informative GMMs corresponding to the initial exploration phase of ALP-GMM, when the learner has not made any progress (hence no LP detected by the teacher), and 2) remove an entire training trajectory τ s if with T the total number of GMMs in the first run (same as in C raw ), and µ t r the mean episodic reward obtained by the first DRL agent during the last 50 tasks sampled from the t th GMM. R is simply obtained by removing any µ t r that corresponds to a GMM discarded while extracting C from C raw . The remaining rewards are then used as thresholds in IN-R to decide when to switch to the next GMM in C.
AGAIN In AGAIN (see algo. 6), the idea is to use both IN (R,T or P) and ALP-GMM (without the random bootstrapping period) for curriculum generation. Our main experiments use IN-R as it is the highest performing variant (see app. D). This means that in the main sections of this paper, AGAIN = AGAIN-R and IN = IN-R. We combine the changing GMM of IN and ALP-GMM over time, simply by building a GMM G containing Gaussians from the current GMM of IN and ALP-GMM. By selecting the Gaussian in G from which to sample a new task using their respective LP, this approach allows to adaptively modulate the task sampling between both, shifting the sampling towards IN when ALP-GMM does not detect high-LP subspaces and towards ALP-GMM when the current GMM of IN have lower-LP Gaussians. While combining ALP-GMM to IN, we reduce the residual random sampling of ALP-GMM from ρ high = 10%, used for the pretrain phase, to either ρ low = 2% for experiments presented in sec. 4.1 and sec. 4.3, or ρ low = 0% for experiments done in the Parkour environment in sec. 4.2 (here we found ρ low = 0% to be beneficial in terms of performances w.r.t. ρ low = 2%, which means that the task-exploration induced by the periodic GMM fit of ALP-GMM was sufficient for exploration). In AGAIN-R and AGAIN-T, when the last GMM p(T ) of the IN curriculum is reached, we switch the fixed LP T i values of all IN Gaussians to periodically updated LP estimates, i.e. we allow AGAIN to modulate the importance of p(T ) for task sampling depending on its current student's performance.

C Considered ACL and Meta-ACL teachers
Meta-ACL variants Our proposed approach, AGAIN, is based on the combination of an inferred expert curriculum with ALP-GMM, an exploratory ACL approach. In section 3 and appendix B,

Algorithm 2 Pretrain phase (helper function)
Require: Student policy s θ , teacher training history H, task-encoding parameter space P, LP threshold δ LP , experimental pre-train budget K pre , pre-test set size m, number of neighbors for student selection k, random sampling ratio ρ high , parametric procedural environment generator E 1: Init s θ , train it for K pre env. steps with ALP-GMM(ρ high , P) 2: Pre-test s θ with m tasks selected uniformly over P and get KC pre s # Pre-test phase 3: Apply knn algorithm in KC space of H, get k students closest to KC pre s 4: Among those k, keep the one with highest summed post training KC post , extract its C raw 5: Get C from C raw by removing any Gaussian with LP ti < δ LP . 6: Return C

Algorithm 3 Inferred progress Niches -Time-based (IN-T)
Require: Student policy s θ , teacher training history H, task-encoding parameter space P, LP threshold δ LP , update rate N , experimental budget K, experimental pre-train budget K pre , pre-test set size m, number of neighbors for student selection k, random sampling ratio ρ high , parametric procedural environment generator E Set i c to min(i c + 1, len(C)) 5: loop N times 7: Sample p from a Gaussian in G IN chosen proportionally to its LP ti 8: Send E(a ∼ A(p)) to student s θ 9: end loop 10: end loop 11: Add student's training trajectory to H 12: Return s θ

Algorithm 4 Inferred progress Niches -Pool-based (IN-P)
Require: Student policy s θ , teacher training history H, task-encoding parameter space P, LP threshold δ LP , update rate N , experimental budget K, experimental pre-train budget K pre , pre-test set size m, number of neighbors for student selection k, random sampling ratio ρ high , parametric procedural environment generator E Send E(a ∼ A(p)) to student s θ 6: end loop 7: Add student's training trajectory to H 8: Return s θ we present 3 approaches to use such an expert curriculum, giving the AGAIN-R, AGAIN-P and AGAIN-T algorithms. In our experiments, we also consider ablations were we only use the expert curriculum, giving the IN-R, IN-P and IN-T variants. We also consider two additional AGAIN variants that do not use our proposed KC-based student selection method: • AGAIN with Random selection (AGAIN_RND), a lower-baseline ablation were we select the training trajectory τ from which to extract the expert curriculum randomly in history H.
• AGAIN with Ground Truth selection (AGAIN_GT), an upper-baseline using privileged information. Instead of performing the knn algorithm in the KC space, this approach directly uses the true student distribution. For instance, in the Parkour environment, given

Algorithm 5 Inferred progress Niches -Reward-based (IN-R)
Require: Student policy s θ , teacher training history H, task-encoding parameter space P, LP threshold δ LP , update rate N , experimental budget K, experimental pre-train budget K pre , pre-test set size m, number of neighbors for student selection k, random sampling ratio ρ high , parametric procedural environment generator E 1: Launch Pretrain phase and get expert GMM list C # See algo. 2 2: Initialize reward First-in-First-Out window W, set max size to N 3: Initialize expert curriculum index i c to 0 4: loop Stop after K − K pre environment steps 5: If W is full, compute mean reward µ w from W 6: If µ w superior to i th c reward threshold in R, set i c to min(i c + 1, len(C)) 7: Set current GMM G IN to i th c GMM in C 8: Sample p from a Gaussian in G IN chosen proportionally to its LP ti 9: Send E(a ∼ A(p)) to student s θ and add episodic reward r p to W 10: end loop 11: Add student's training trajectory to H 12: Return s θ

Algorithm 6 Alp-Gmm And Inferred progress Niches (AGAIN)
Require: Student policy s θ , teacher training history H, task-encoding parameter space P, LP threshold δ LP , update rate N , experimental budget K, experimental pre-train budget K pre , pre-test set size m, number of neighbors for student selection k, random sampling ratio ρ low and ρ high , parametric procedural environment generator E Get composite GMM G from the current GMM of both ALP-GMM and IN 6: ρ low % of the time, sample a random parameter p ∈ P 7: Else, sample p from a Gaussian chosen proportionally to its LP 8: Send E(a ∼ A(p)) to student s θ and observe episodic reward r p 9: Send (p, r p ) pair to both ALP-GMM and IN 10: end loop 11: Add student's training trajectory to H 12: Return s θ a new student s, AGAIN_GT selects the k previously trained students from H that are morphologically closest to s (i.e. same embodiment type and closest limb sizes), and uses the training trajectory of the student with highest score j s (see sec. 3).
Note that both for AGAIN_RND and AGAIN_GT, there is no need to pre-test the student, which means we can use the IN expert curriculum directly at the beginning of training rather than after a pre-training phase.
ACL conditions A first natural ACL approach to compare our AGAIN variants to is ALP-GMM, the underlying ACL algorithm in AGAIN. We also add as a lower-baseline a random curriculum teacher (Random), which samples tasks' parameters randomly over the task space.
In both the toy environment (sec. 4.1, toy env. for short) and the Parkour environment (sec. 4.2), we additionally compare to Adaptive Domain Randomization (ADR), an ACL algorithm proposed in OpenAI et al. [2019], which is based on inflating a task distribution sampling from a predefined initially feasible task p easy (w.r.t a given student). Each lower and upper boundaries of each dimension of the sampling distribution are modified independently with step size ∆ step whenever a predefined mean reward threshold r thr is surpassed over a window (of size q) of tasks occasionally sampled (with probability ρ b ) at the sampling dimension boundary. More details can be found in OpenAI et al. [2019]. In our experiments, as we do not assume access to expert knowledge over students sampled within the student distribution, we randomize the setting of p easy uniformly over the task space in Parkour experiments and uniformly over the 4 possible student starting subspaces in toy env. experiments. Based on the hyperparameters proposed in OpenAI et al. [2019] and on informal hyperparameter search, we use [ρ b = 0.5, r thr = 1, ∆ step = 0.05, q = 10] in toy env. experiments and [ρ b = 0.5, r thr = 230, ∆ step = 0.1, q = 20] in Parkour experiments.
In experiments described in sec 4.3, we compare our approaches to an oracle condition (Oracle), which is a hand-made curriculum that is very similar to IN-R, except that the list C is built using expert knowledge before training starts (i.e. no pre-train and pre-test phases), and all reward thresholds µ i r in R (see eq. 6) are set to 230, which is an episodic reward value often used in the literature as characterizing a default walker having a "reasonably efficient" walking gate in environments derived from the Box2D gym environment BipedalWalker Wang et al. [2019], Portelas et al. [2019].In practice, Oracle starts proposing tasks from a Gaussian (with std of 0.05) located at the simplest subspace of the task space (ie. low stump height and high stump spacing) and then gradually moves the Gaussian towards the hardest subspaces (high stump height and low stump spacing) by small increments (50 steps overall) happening whenever the mean episodic reward of the DRL agent over the last 50 proposed tasks is superior to 230.

D Analysing Meta-ACL in a toy environment
In this section we report the full comparative experiments done in the toy environment, which includes comparisons with AGAIN-T and AGAIN-P to AGAIN-R, shown in table 1. We also provide visualizations of the KC-based curriculum priors selection process (see fig. 9) happening after the pretraining phase in AGAIN along with a visualization of the fixed set of 96 randomly drawn students used to perform the varying classroom experiments reported in sec. 4.1 (see fig. 8). Table 1 summarizes the post-training performances obtained by our considered Meta-ACL conditions and ACL baselines on the toy environment with only 4 possible students on a fixed set of 48 randomly drawn students. Meta-ACL conditions are given a training trajectory H created by training an initial classroom of 128 students. Using a Reward-based iterating scheme over the inferred expert curriculum (AGAIN-R and IN-R) outperforms the Time-based and Pool-based variants (p < .001). This result was expected as both these last two variants do not have flexible mechanisms to adapt to the student being trained. The pool based variants (AGAIN-P and IN-P), which discard the temporal ordering of the expert curriculum are the worst performing variants, statistically significantly inferior to both Reward-based and Time-based conditions (p < .001). episodes is reported (48 seeds per conditions). For Meta-ACL variants we report results with column 1) the regular KC-based curriculum prior selection performed after 20k pre-training episodes, column 2) An ablation that performs the selection at random before training, and column 3) An oracle condition selecting before training the curriculum prior using student ground truth type. * Denotes stat. significant advantage w.r.t. ALP-GMM (Welch's t-test at 200k ep. with p < 0.05).  tion of the starting cells of students from a 10% sample of a classroom of 400 students (one per student type) trained with ALP-GMM and used to populate the training trajectory H. Each blue circle marks the starting cell of each student (i.e. its type) within the 2D parameter space P, which is an initial learning subspace that needs to be detected by the teacher for successful training. Right:Visualization of the fixed set of 96 randomly drawn students that have to be trained by Meta-ACL variants given H. As not all student types are represented in H, Meta-ACL approaches have to generalize their curriculum generation to these new students.

Condition
(a) with type 0 new student (b) with type 1 new student (c) with type 2 new student (d) with type 3 new student Figure 9: Examples of student selection process in 4-student type toy environment. In all figures, we plot the 2D PCA visualization of the KC pre vectors (after pre-training) of the initial classroom (128 students) trained with ALP-GMM and used to populate the training trajectory H used by AGAIN variants in our 4-student type toy env experiments (see sec. 4.1). We then use these 4 figures to showcase the selection process happening in 4 different AGAIN-R runs (one per student type). Each triangle represents a student, whose ground truth type (i.e. its initial learning cell) is denoted by the orientation of the triangle. Given a new student to train, AGAIN pretrains the student, constructs its KC vector (purple border triangle), infers the k closest previously trained students from H (red and golden border triangles), and use the one with highest end of training performance (i.e. highest score s, see sec. 3), denoted by a golden border triangle, to infer curriculum priors for the new student.

E Meta-ACL for DRL students in the Parkour environment
In this section we give additional details on the Parkour environment presented in section 4.2, and we provide additional details and visualizations on the experiments that were performed on it.
Details on the Parkour environment. In our experiments, we bound the wall spacing dimension of the task space to ∆ w = [0, 6], and the gate y position to µ gate = [2.5, 7.5]. In practice, given a single parameter tuple (µ gate , ∆ w ), we actually encode a distribution of tasks, since for each new wall along the track we add an independent Gaussian noise to each wall's gate y position µ gate . Examples of parkour tasks randomly sampled within these bounds are available in figure 10 (right). At the beginning of training a given DRL policy, the agent is embodied in either a bipedal walker morphology with two joints per legs or a two-armed climber morphology with 3-joints per arms ended by a grasping "hand". Both morphologies are controlled by torque. Climbers have an additional action dimension g ∈ [−1, 1] used to grasp: if g ∈ [−1, 0[, the climber closes its gripper, and if g ∈]0, 1] it keeps it open. To avoid falling (which aborts the episode with a −100 penalty) while moving forward to collect rewards, climber agents must learn to swing themselves forward by successive grasp-and-release action sequences. To increase the diversity of the student distribution, we also randomize limb sizes. See figure 10 (left) for examples of randomly sampled embodiments.
Soft Actor-Critic students In our experiments, we use an implementation of Soft Actor-Critic provided by OpenAI 1 (MIT license). We use a 2 layered (400,300) network for V, Q1, Q2 and the policy. Gradient steps are performed each 10 environment steps, with a learning rate of 0.001 and a batch size of 1000. The entropy coefficient is set to 0.005.
Evaluation procedure To report the performance of our students on the Parkour environment, we use two separate test sets, one per embodiment type. For walkers we use a 100-tasks test set, uniformly sampled over a subspace of the task space with ∆ w ∈ [0, 6] and µ gate ∈ [2.5, 3.6], which we chose based on 1) what we initially believed to be morphologically feasible for walkers, and 2) based on previously designed test sets built in recent work Portelas et al. [2019] on comparable bipedal walker experiments). For climbers, because there is no similar experiments in the literature and since it is hard to infer beforehand what will be achievable by such a morphology, we simply use a uniform test set of 225 tasks sampled over the full task space. Importantly, the customized test set used for walkers is solely used for visualization purposes. In our AGAIN approaches, we pre-test all students with the expert-knowledge-free set of 225 tasks uniformly sampled over the task space.
Compute resources Each of the 576 seeds required to reproduce our experiments (128 seeds for the classroom and 7*64 seeds for our 7 conditions) takes 36 hours on a single cpu. This amounts to around 21 000 cpu hours. Each run requires less than 1GB of RAM.
Visualizing student diversity. To assess whether our proposed multi-modal distribution of possible students in the Parkour environment do have diverse competence profiles (which is desirable as it creates a challenging Meta-ACL scenario), we plot the 2D PCA of the post training KC vector for each students of the initial classroom trained with ALP-GMM (used to populate H). The result, visible in figure 11 (top), shows that climber-students and walker-students are located in two independant clusters, i.e. they do have clearly different competence profiles. The spread of each clusters also demonstrates that variations in initial policy parameters and limb sizes also creates students with diverse learning potentials. The competence differences between walkers and climbers can also be seen in Figure 11 (left and right), which shows the episodic reward obtained for each of the 225 tasks of the KC vector after training by a representative walker student (left) and climber student (right). with ALP-GMM. left and right: Episodic reward obtained for each task that compose the KC vector by a walker-student (left) and a climber-student (right) of this classroom. Stars are added for all tasks for which the agent obtained more than r = 230 (which corresponds to an efficient locomotion policy). Walkers only manage to learn tasks with very low gate positions while climbers learn only tasks with medium to high gate positions.
F Applying Meta-ACL to a single student: Trying AGAIN instead of trying longer In the following section we report all experiments on applying AGAIN variants to train a single DRL student (i.e. no history H), which is briefly presented in sec. 4.3.
Parametric BipedalWalker env. We test our modified AGAIN variants along with baselines on an existing parametric BipedalWalker environment proposed in Portelas et al. [2019], which generates walking tracks paved with stumps whose height and spacing are defined by a 2D parameter vector used for the procedural generation of tasks. We keep the original bounds of this task space, i.e. we bound the stump-height dimension to µ h ∈ [0, 3] and the stump-spacing dimension to δ s ∈ [0, 6]. As in their work, we also test our teachers when the learning agent is embodied in a modified short-legged walker, which constitutes an even more challenging scenario (as the task space is unchanged, i.e. more unfeasible tasks). The agent is rewarded for keeping its head straight and going forward and is penalized for torque usage. The episode is terminated after 1) reaching the end of the track, 2) reaching a maximal number of 2000 steps, or 3) head collision (for which the agent receives a strong penalty). See figure 12 for visualizations. Results To perform our experiments, we ran each condition for either 10Millions (IN and AGAIN variants) or 20Millions (others) environment steps (30 repeats). The preliminary ALP-GMM runs used in IN and AGAIN variants correspond to the first 10 Million steps of the ALP-GMM condition (whose end-performance after 20 Million steps is reported in table 2. All teacher variants are tested when paired with a Soft-Actor Critic Haarnoja et al. [2018] student, with same hyperparameters as in the Parkour experiments (see app. E). Performance is measured by tracking the percentage of mastered tasks (i.e. r > 230) from a fixed test set of 100 tasks sampled uniformly over the task space. We thereafter report results for 2 independent experiments done with either default walkers or short walkers.
Is re-training from scratch beneficial? -The end performances of all tested conditions are summarized in table 2. Interestingly, retraining the DRL agent from scratch in the second run gave superior end performances than fine-tuning using the weights of the first run in all tested variants. This showcases the brittleness of gradient-based training and the difficulty of transfer learning. Despite this, even fine-tuned variants reached superior end-performances than classical ALP-GMM, meaning that the change in curriculum strategy in itself is already beneficial.
Is it useful to re-use ALP-GMM in the second run? -In the default walker experiments, AGAIN-R, T and P conditions mixing ALP-GMM and IN in the second run reached lower mean performances than their respective IN variants. However, the exact opposite is observed for IN-R and IN-T variants in the short walker experiments. This can be explained by the difficulty of short walker experiments for ACL approaches, leading to 16/30 preliminary 10M steps long ALP-GMM runs to have a mean end-performance of 0, compared to 0/30 in the default walker experiments. All these run failures led to many GMMs lists C used in IN to be of very low-quality, which illustrates the advantage of AGAIN that is able to emancipate from IN using ALP-GMM.   (others) is reported (30 seeds). For IN and AGAIN we also test variants that do not retrain the weights of the policy used in the second run from scratch but rather fine-tune them from the preliminary run. * /− Indicates whether perf. difference with ALP-GMM is statistically significant ie. p < 0.05 in a post-training Welch's student t-test ( * for performance advantage w.r.t ALP-GMM and − for perf. disadvantage).

Condition
Short walker Default walker


s2fieldsofstudy:
Computer Science
Physics

text:
Data-driven solutions and discoveries in mechanics using physics informed neural network

Deep learning has achieved remarkable success in diverse computer science applications, however, its use in other traditional engineering fields has emerged only recently. In this project, we solved several mechanics problems governed by differential equations, using physics informed neural networks (PINN). The PINN embeds the differential equations into the loss of the neural network using automatic differentiation. We present our developments in the context of solving two main classes of problems: data-driven solutions and data-driven discoveries, and we compare the results with either analytical solutions or numerical solutions using the finite element method. The remarkable achievements of the PINN model shown in this report suggest the bright prospect of the physics-informed surrogate models that are fully differentiable with respect to all input coordinates and free parameters. More broadly, this study shows that PINN provides an attractive alternative to solve traditional engineering problems.

Introduction
It is well-known that many mechanics phenomenons are governed by conservation laws which can be converted to differential equations for most of the applications. Solving these differential equations and obtaining parameter estimations from given observations are always intriguing topics, and significant research has been done to develop many advanced (semi-)analytical or numerical algorithms. While in the last decade, machine learning especially neural networks have yielded revolutionary results across diverse disciplines, including image and pattern recognition, natural language processing, genomics, and material constitutive modeling [15,23,26,34], among which a fair amount of research has also been done related to differential equations [1,9,12,13,17,18,19,22,24,25,27,28,29,30,31,32,33,35,36,38], owing to the dramatic increase in the computing resources. Therefore, it will be interesting to adopt neural networks as an important alternative to traditional mathematical methods to approximate the solutions to differential equations through iterative update of the network weights and biases.

Related work
Early researches that used neural network in scientific computing could date back to [12,22], in which a single hidden layer neural network was adopted where N is the network output which is assumed to be a scalar for illustration purposes, H is the width of the hidden layer, and n is the dimension of the input vector x ∈ R n , w i j and v i are called weights while u i andũ are always denoted as the biases, σ i (·) is called activation function. More recently, the development of automatic differentiation [2] has led to emerging literature on the use of deep neural networks to solve differential equations [17,33,38,39] even in complex geometries, and Berg et al. [3] provided details of this back propagation algorithm for advection and diffusion equations. Unlike traditional machine learning methods, deep neural networks sometimes can overcome the curse of dimensionality [17]. In the work done by Raissi et al [30,31,32], they named such strong form approach for differential equation as the physicsinformed neural network (PINN) for the first time. Today, the PINN becomes more and more popular in many engineering fields such as hydrogeology [7,18,35], geomechanics [19], cardiovascular system [21] and so on. One attractive feature of PINN is that it is fully differentiable with respect to all the input coordinates and free parameters, in other words, the trial solution (via the trained PINN) represents a smooth approximation that can be evaluated and differentiated continuously on the domain [9]. Another important feature of PINN is that it can be used to solve inverse (data discovery) problems [16] with minimum change of the code from the forward problems [24].
It is also worth to mention that there are also other machine learning methods that didn't rely on the neural network to solve differential equations [28,29], but in this report, we will focus on the PINN.

Methods
In this section, we will give a very brief overview of PINN in order to demonstrate how the loss function L is defined in such context. We refer interested readers to look at [32] for more details. Consider a system of differential equations defined on the domain Ω with boundary ∂Ω: where u is the unknown solution vector, D denotes the abstract (non-linear) differential operator (e.g., ∂/∂x, u • ∂/∂x, x • ∂/∂x, higher order derivatives, non-linear functions of u or x, their combinations, etc.), and the operator B expresses arbitrary boundary conditions (e.g., Dirichlet, Neumann, Robin, etc.) associated with the problem. Note here for time-dependent problems, we consider t as a special component of x, i.e., the Ω contains the temporal domain. The initial condition can be simply treated as a special type of Dirichlet boundary condition on the spatio-temporal domain [24]. Now we construct a neural network with L layers, or L − 1 hidden layers to approximate u (x), the input to neural network is x, and we will denote the output of the neural network asû (x; θ) (we wantû (x; θ) ≈ u (x)), where θ represents the collection of weight matrix W [l] ∈ R n l ×n l−1 and bias vector b [l] ∈ R n l for each layer l with n l neurons. In the inverse or data discovery problems, there are some unknown parameters γ in the original differential equations, thus we will rewrite our approximation asû (x; θ, γ). Now we can define the physics-informed loss L as where w f is the weight for the loss due to mismatch with the governing equations and w b is the weight for the loss due to mismatch with the boundary conditions and w d is the weight for the loss due to mismatch with the given data observations u * (x) In practice, Γ f and Γ b are always known as the set of locations of the "residual" points, and Γ d is known as the set of measurement locations. We then optimize θ and γ together, and our solution is Above process is approximately summarized in Fig. 1. Due to the network architecture and incorporation of differential equations, the loss will be highly non-linear and also non-convex, thus we will need some gradient-based optimizers, such as steepest descent [4], Newton's method [4], Adam [20], and L-BFGS (limited-memory quasi-Newton method) [5]. In this report, we will adopt purely Adam or Adam combined with L-BFGS optimizers (i.e., we first use Adam for a certain number of iterations, and then switch to L-BFGS until convergence).
For estimation of the test error, we will evaluate the loss L f on a given test set Γ test ⊂ Ω. In addition, if the true solution is available, we will also calculate the mean square error (similar to the form of L d ) on the same test set.

One dimensional consolidation problem
In soil mechanics, one of the most important processes is the consolidation process, while under some specific assumptions made by Prof. Terzaghi and Prof. Biot [6,8,37], we can mathematically quantify this process as a one dimensional diffusion problem [14] with specific boundary conditions and initial conditions. Suppose our problem domain is 0 ≤ x ≤ L where L can be understood as the thickness of the aquifer that rests on a rigid impermeable base, at time t = 0, surface load p 0 is applied on the top of the aquifer which will lead to an initial pore pressure field p u = γ l p 0 at t = 0 + where γ l is called loading efficiency [37]. As time goes on, the pressure p will gradually decrease due to the effect of the drainage boundary at x = 0, and it satisfies following partial differential equation with boundary conditions p = 0 at x = 0 and ∂p/∂x = 0 at x = L. The initial condition is just p = p u as mentioned before. In above Eq. (9), the c is called generalized consolidation coefficient [11] which has the unit of length squared divided by time. However, the order of magnitudes of different physical quantities x (order of meters), t (order of months) and p (order of kPa) could cast great difficulty on the training of the neural networks [21], and to overcome this problem, we employ a non-dimensionalization technique with the purpose of scaling the input and the output of the neural network in proper scales (e.g., O (1)). As a result, we will introduce characteristic values for x and p, and for this problem an obvious choice would be characteristic length L and characteristic pressure p u . At this point we define the dimensionless quantities aŝ For the time t, the dimensionless timet is given ast Now all the variables take values with O (1) magnitude and we will rewrite Eq. (9) using dimensionless variables, the result is shown as follows The analytical solution for Eq. (12) is given aŝ For the PINN hyper-parameters, we assumet max = 0.5, and we draw 500, 250, 125 "residual" points randomly from (0, 1) × 0,t max , boundary ofx and boundary oft respectively to form Γ f and Γ b . The test set Γ test will have 10011 points. The loss weights are assigned as w f = w b = 1/4 (w d is not used in this problem). For the architecture of the neural network, it will contain 5 hidden layers with 50 neurons each, and we will use hyperbolic tangent tanh as our activation function in hidden layers, the output activation will be linear activation but we will multiply the network output withx as our final output, because in this way our output will automatically satisfy the second expression in Eq. (12). For the optimization procedure, we will run Adam for 50000 epochs with learning rate 0.001 and then run L-BFGS until convergence. For the network initialization, we will use the Glorot normal initializer.
The convergence history, reference solution, PINN solution, and the absolute error is shown in the Fig. 2, from which we can see a perfect match is achieved, and the locations with highest error is close to the origin due to sharp gradient. Note in Fig. 2, we have enforced all the output values are between 0 and 1, and initial condition is strictly satisfied.

Steady state heat conduction problem
Our second example is a time-independent problem on 2D. In the steady state heat transfer problem with constant heat conduction coefficient, constant heat source and constant boundary temperature (Dirichlet boundary condition), the T will satisfy Poisson's equation [14]. Without loss of generality and to avoid the scaling differences, we will solve following PDE on Ω = [−1, 1] × [−1, 1] Even though the analytical solution for this problem doesn't exist, the finite element solution with high accuracy is available in deal.ii library, thus it will be interesting to compare these two numerical solutions. For the PINN hyper-parameters, we choose 1500 and 500 random points to form Γ f and Γ b , respectively. The test set Γ test will have 10000 points. The loss weights are assigned as w f = w b = 1/2 (w d is not used in this problem). For the architecture of the neural network, it will contain 4 hidden layers with 50 neurons each, and we will not apply any transform for the network output. The learning rate for this problem is 0.0005. For the network initialization, we will use the Glorot uniform initializer. The remaining settings are the same as the first example. For the fourth figure, we need to mention the PDE test loss is only due to mismatch with the governing equation, i.e., L f . The green line is also evaluated from the same test set. For the PDE train loss, we recall pde, [bc1,bc2,ic] from the source code, thus we take an average to get train loss which is consistent with a weight of 1/4.
The training results are shown in the Fig. 3, where the two numerical solutions are consistent with each other. Note in Fig. 3, we have enforced all the output values are larger than 0. The largest absolute error happens near the Dirichlet boundary because Eq. (15) is not regarded as a hard constrain in the construction of the loss L. A feasible way to handle this problem would be to introduce a smooth distance functiond (x) that gives the distance for x ∈ Ω to ∂Ω [3], and then we multiplyd with the original network output to get our new improved output, which will automatically satisfy Eq. (15).

Inverse problem for simple structural dynamic system
The last example comes from the dynamic response of the single degree of freedom system [10] and we want to use this example to illustrate the capability of PINN in the inverse modeling or data discovery process. In structural dynamics, the free vibration equation with viscous damping for single mass point is obtained through D'Alembert principle [10] m with u (0) = u 0 andu (0) = v 0 . In above equation, m > 0 is the mass, c > 0 is the damping coefficient, k > 0 is the stiffness, u is the displacement, u 0 is the initial displacement and v 0 is the initial velocity. The analytical solution to this ODE is given as where ω n = √ k/m is known as the natural frequency, ξ = c/ (2mω n ) is known as the damping ratio, ω D = ω n 1 − ξ 2 . In this problem, we will assume ξ < 1 which is known as the underdamped system. Now suppose m = u 0 = v 0 = 1, c and k are the two parameters to be identified from the observations at certain times. The observations are produced with the underlying true parameters c = 0.4 and k = 4 using Eq. (17). We choose 50 equispaced points to form Γ d , 100 random points to form Γ f , 50 random points to form Γ b and 10000 points to form Γ test . The loss weights are assigned as w f = w b = w d = 1/4. For the architecture of the neural network, it will contain 3 hidden layers with 32 neurons each, and we will not apply any transform for the network output. For the optimization procedure, we will only run Adam for 100000 epochs with learning rate 0.0005. For the network initialization, we will use the Glorot uniform initializer.
The evolution trajectories of c and k are presented in Fig. 4, with final values ofk = 3.9933197 and c = 0.40125215, which agree with their true values.

Conclusions
In this project, we have adopted PINN, a new class of universal function approximators that is capable of encoding any underlying conservation laws which can be described by differential equations. Compared to the traditional numerical methods, PINN employs automatic differentiation to handle differential operators, and thus are mesh-free. We use PINN as a data-driven algorithm for inferring solutions and parameter identifications of differential equations. A series of promising results for a diverse collection of problems in mechanics are presented and discussed, which opens the path for endowing deep learning with the powerful capacity of mathematical physics to model the world around us. We hope this project could benefit future practitioners across a wide range of scientific domains who want to incorporate deep learning methods in modeling physics-related problems.
For the future work, it is worth to try a group of coupled differential equations such as Eqs. (18) (19) in poromechanics [40] to see the PINN's performance, and develop some training points selection strategy similar to the adaptive re-meshing in the mesh-based methods to achieve better accuracy.