summaries: The authors propose a novelty detection module to help unsupervised class-incremental learning. The novelty detection relies on the percentage of accuracy drop during a model update when treating incoming data as a new class. If the model maintains high accuracy, then the module treats the incoming data as familiar, thereby choosing one of the existing classes as the correct label. The paper investigates the effectiveness of the proposed method on MNIST, SVHN, CIFAR-10, and CIFAR-100.

summaries: This article proposes a method for predicting whether a batch of data is of the same class as one of the classes already seen by a classifier or whether it contains data from another class. The idea is to then be able to incorporate this batch to the previous training set, in an unsupervised learning context. It is assumed that each batch contains data from only one class. Experiments are there to show the interest of this method for anomaly detection or incremental learning.

summaries: This paper proposes to tackle the problem of unsupervised class-incremental learning, where the training data is composed of a sequence of "exposures". Each exposure is comprised of a set of images that pertains to a single class, where the class label is unknown while the boundaries between exposures are known. The key difficulty in such unsupervised class-incremental learning is to determine whether an arriving exposure belongs to what the classification model $L$ has learnt previously or is a novel one, thus relating to the problem of novelty detection. The proposed method address the novelty detection by an interesting idea: they always treat the current exposure as a novel class and use it to train the copy of classification model $\hat{L}$ together with the training exemplars of previously-learnt classes, if the current exposure actually belongs to one of the previous-learnt classes, the confusion occurs to make the classification accuracy significantly decrease (over a threshold) on that specific class, where the accuracy is computed based on the validation exemplars. Moreover, a technique of introducing class-imbalance into such confusion-based novelty detection is proposed and helps to boost the robustness of novelty detection.

abstractText: While many works on Continual Learning have shown promising results for mitigating catastrophic forgetting, they have relied on supervised training. To successfully learn in a label-agnostic incremental setting, a model must distinguish between learned and novel classes to properly include samples for training. We introduce a novelty detection method that leverages network confusion caused by training incoming data as a new class. We found that incorporating a class-imbalance during this detection method substantially enhances performance. The effectiveness of our approach is demonstrated across a set of image classification benchmarks: MNIST, SVHN, CIFAR-10, CIFAR-100, and CRIB.

authors:
Alexander A Alemi
Ian Fischer
Joshua V Dillon

title: Uncertainty in the variational information bottleneck

venue: arXiv preprint arXiv:1807.00906,

authors:
Rahaf Aljundi
Francesca Babiloni
Mohamed Elhoseiny
Marcus Rohrbach
Tinne Tuytelaars

title: Memory aware synapses: Learning what (not) to forget

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Francisco M Castro
Manuel J Marín-Jiménez
Nicolás Guil
Cordelia Schmid
Karteek Alahari

title: End-to-end incremental learning

authors:
Hyunsun Choi
Eric Jang

title: Generative ensembles for robust anomaly detection

authors:
Matthias De Lange
Rahaf Alijundi
Marc Masana
Sarah Parisot
Xu Jia
Ales Leonardis
Gregory Slabaugh
Tinne Tuytelaars

title: A continual learning survey: Defying forgetting in classification

authors:
Terrance DeVries
Graham W Taylor

title: Learning confidence for out-of-distribution detection in neural networks

venue: arXiv preprint arXiv:1802.04865,

title: Anomaly detection over noisy data using learned probability distributions

authors:
Ian J Goodfellow
Mehdi Mirza
Da Xiao
Aaron Courville
Yoshua Bengio

title: An empirical investigation of catastrophic forgetting in gradient-based neural networks

venue: arXiv preprint arXiv:1312.6211,

authors:
Ville Hautamaki
Ismo Karkkainen
Pasi Franti

title: Outlier detection using k-nearest neighbour graph

venue: In Proceedings of the 17th International Conference on Pattern Recognition,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: arXiv preprint arXiv:1610.02136,

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

venue: arXiv preprint arXiv:1812.04606,

authors:
Nitin Kamra
Umang Gupta
Yan Liu

title: Deep generative dual memory network for continual learning

venue: arXiv preprint arXiv:1710.10368,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Mark Kliger
Shachar Fleishman

title: Novelty detection with gan

venue: arXiv preprint arXiv:1802.10560,

authors:
Edwin M Knorr
Raymond T Ng
Vladimir Tucakov

title: Distance-based outliers: algorithms and applications

venue: The VLDB Journal,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

title: The mnist database of handwritten digits. http://yann

venue: lecun. com/exdb/mnist/,

authors:
Soochan Lee
Junsoo Ha
Dongsu Zhang
Gunhee Kim

title: A neural dirichlet process mixture model for task-free continual learning

venue: arXiv preprint arXiv:2001.00689,

authors:
Shiyu Liang
Yixuan Li
Rayadurgam Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

venue: arXiv preprint arXiv:1706.02690,

authors:
David Lopez-Paz
Marc’Aurelio Ranzato

title: Gradient episodic memory for continual learning

venue: In Advances in Neural Information Processing Systems,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
German I Parisi
Ronald Kemker
Jose L Part
Christopher Kanan
Stefan Wermter

title: Continual lifelong learning with neural networks: A review

venue: Neural Networks,

title: Continual unsupervised representation learning

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Georg Sperl
Christoph H Lampert

title: icarl: Incremental classifier and representation learning

venue: In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,

authors:
David Rolnick
Arun Ahuja
Jonathan Schwarz
Timothy Lillicrap
Gregory Wayne

title: Experience replay for continual learning

authors:
Mohammad Sabokrou
Mohammad Khalooei
Mahmood Fathy
Ehsan Adeli

title: Adversarially learned one-class classifier for novelty detection

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Chandramouli S Sastry
Sageev Oore

title: Zero-shot out-of-distribution detection with feature correlations

authors:
Walter J Scheirer
Anderson de Rezende Rocha
Archana Sapkota
Terrance E Boult

title: Toward open set recognition

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Walter J Scheirer
Lalit P Jain
Terrance E Boult

title: Probability models for open set recognition

authors:
Joan Serrà
David Álvarez
Vicenç Gómez
Olga Slizovskaia
José F Núñez
Jordi Luque

title: Input complexity and out-of-distribution detection with likelihood-based generative models

authors:
Hanul Shin
Jung Kwon Lee
Jaehong Kim
Jiwon Kim

title: Continual learning with deep generative replay

authors:
James Smith
Constantine Dovrolis

title: Unsupervised progressive learning and the stam architecture

venue: arXiv preprint arXiv:1904.02021,

authors:
Stefan Stojanov
Samarth Mishra
Ngoc Anh Thai
Nikhil Dhanda
Ahmad Humayun
Chen Yu
Linda B Smith
James M Rehg

title: Incremental object learning from contiguous views

authors:
Chenshen Wu
Luis Herranz
Xialei Liu
Joost van de Weijer
Bogdan Raducanu

title: Memory replay gans: Learning to generate new categories without forgetting

venue: In Advances In Neural Information Processing Systems,

authors:
Yue Wu
Yinpeng Chen
Lijuan Wang
Yuancheng Ye
Zicheng Liu
Yandong Guo
Yun Fu

title: Large scale incremental learning

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

authors:
Kenji Yamanishi
Jun-Ichi Takeuchi
Graham Williams
Peter Milne

title: On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms

venue: Data Mining and Knowledge Discovery,

authors:
Jaehong Yoon
Eunho Yang
Jeongtae Lee
Sung Ju Hwang

title: Lifelong learning with dynamically expandable networks

venue: arXiv preprint arXiv:1708.01547,

authors:
Friedemann Zenke
Ben Poole
Surya Ganguli

title: Continual learning through synaptic intelligence

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

heading: 1 INTRODUCTION

text:
The development of continually learning systems remains to be a major obstacle in the field of artificial intelligence. The primary challenge is to mitigate catastrophic forgetting: learning new tasks while maintaining the ability to perform old ones. This domain of research is often referred to as Continual Learning, Lifelong Learning, Sequential Learning, or Incremental Learning: each with subtleties in the learning environment and training process, but most with the use of supervision (De Lange et al. (2020)).
Recently, Stojanov et al. (2019) introduced a novel unsupervised class-incremental learning problem motivated by the desire to simulate how children’s play behaviors support their ability to learn object models (IOLfCV). Here, sequential tasks take the form of exposures. Each exposure is comprised of a set of images that pertains to a single class that is hidden from the learner. Exposure boundaries, the transition from one exposure to the next, are known. The model trained in this setting, is analogous to a young child that has been placed in a playpen with a set of new toys. The child steadily gathers information over time by picking up, examining, and putting down new/old objects continuously. Similar to how the child does not have any guidance to the toys they will examine, the agent does not have access to the exposure’s identity during training.
To learn in the unsupervised class-incremental setting, an agent must conduct two procedures successfully. Given a new learning exposure, the key step is to perform novelty detection: to identify whether an exposure corresponds to a class that has been learned. If the agent determines that an exposure is familiar, the second step is to identify its label such that the exposure can be leveraged to update the model. Both procedures must be performed reliably. Otherwise, the novelty detection mistakes will result in label noise that distorts the learned model, increasing the likelihood of subsequent mistakes.
Deep neural networks are known to make overconfident decisions for anomalous data distributions that were not seen during training (Hendrycks & Gimpel (2016)). To address this problem, research related to out-of-distribution (OOD) detection have utilized supervised methods (Liang et al. (2017); Alemi et al. (2018)) and unsupervised methods (Choi & Jang (2018); Hendrycks et al. (2018); Serrà et al. (2019)). Works related to open set recognition have also addressed the OOD problem by applying distance-based thresholds computed from known class scores (Scheirer et al. (2012; 2014)). The work by Stojanov et al. (2019) applies a similar method to the unsupervised incremental setting by computing class features produced from a set of supervised samples. In contrast, we propose a model, Incremental Learning by Accuracy Performance (iLAP), that determines class novelty and identity by considering performance changes of previously learned tasks when an incoming set of exposure images are trained under a new label.
Instead of using a distance-based metric, our novelty detection threshold relies on the percentage of accuracy that was maintained by performing a model update using the incoming exposure. This poses several practical advantages: First, the threshold value does not rely on supervised samples and is more intuitive (Section 3.3). Second, the performance of our method is independent of the sequence of the incoming exposure classes (Section 5.2). Finally, the model is able to distinguish between similar classes more reliably (Section 5.3).
From our experiments, we demonstrate that the confusion resulting from training with label ambiguity results in a more reliable signal for novelty detection in comparison to previous methods. We demonstrate that our technique is more robust and results in substantial performance gains in comparison to various baselines. Furthermore, despite the absence of labels, our model was able to perform similarly to supervised models under several benchmarks.
In summary, this work provides three contributions:
• We present a novel framework, iLAP, that achieves learning in the unsupervised classincremental environment where the exposure identities are unknown.
• We demonstrate that by including a class-imbalance technique, our unsupervised method is able to closely match supervised performance for several image classification benchmarks trained in the incremental setting.
• We identify failure cases that are overlooked by traditional OOD methods that leverage distance-based thresholds.

heading: 2 RELATED WORKS

text: Introduced by Stojanov et al. (2019), the unsupervised class-incremental setting contains a set of sequential tasks that are single-class exposures; classes pertaining to the exposures may repeat and are unknown. This is not to be mistaken with unsupervised continual learning (UCL) where task boundaries and task identities are unavailable (Lee et al. (2020); Smith & Dovrolis (2019); Rao et al. (2019)). Our work presents an agent that is able to leverage the boundary information from the unsupervised class-incremental environment to achieve performances that are close to models trained under supervision.

heading: 2.1 CONTINUAL LEARNING/INCREMENTAL LEARNING

text: Prior works in this field primarily aim to improve a model’s ability to retain information while incorporating new tasks (Goodfellow et al. (2013); Parisi et al. (2019); Rebuffi et al. (2017); LopezPaz & Ranzato (2017); Aljundi et al. (2018); Castro et al. (2018)). Typically, these models reside in learning settings where both task labels and task boundaries are available. Methods include replay techniques, the usage of artifacts and generated samples to refresh a model’s memory (Kamra et al. (2017); Wu et al. (2018); Rolnick et al. (2019); Shin et al. (2017); Wu et al. (2019)), and regularization-based practices, the identification and preservation of weights that are crucial for the performance of specific tasks (Kirkpatrick et al. (2017); Zenke et al. (2017); Yoon et al. (2017)). In contrast to prior works, our method addresses incremental learning in a setting where exposure labels are unavailable.

heading: 2.2 UNSUPERVISED CONTINUAL LEARNING

text: Recently, a series of works tackle the UCL problem where task boundaries and task identities are unknown. Smith & Dovrolis (2019) performs novelty detection by analyzing an input image through a series of receptive fields to determine if an input patch is an outlier. Meanwhile, CURL proposes a method to learn class-discriminative representations through a set of shared parameters (Rao et al. (2019)). CN-DPM, introduces an expansion-based approach that utilizes a mixture of experts to learn feature representations (Lee et al. (2020)). Although CN-DPM performs in a task-free setting, incoming tasks are multi-class and individual class labels are provided. This supervision is required to train the existing experts and determine when a new one is needed. While boundary information is not required for these works, the performances are far below supervised baselines (77.7% on and MNIST 13.2% Omniglot) (Rao et al. (2019)).

heading: 2.3 OUT-OF-DISTRIBUTION DETECTION

text: This ongoing area of research aims to detect outliers in training and testing data. Current approaches can be largely categorized by statistical, distance-based, and deep learning methods (Eskin (2000); Yamanishi et al. (2004); Knorr et al. (2000); Hautamaki et al. (2004); Sabokrou et al. (2018); Kliger & Fleishman (2018)). Recent techniques involve using a threshold to determine class novelty from network confidence values (Hendrycks & Gimpel (2016)). ODIN uses input perturbations to increase softmax scores for neural networks to distinguish in-distribution images from out-of-distribution images (Liang et al. (2017)). DeVries & Taylor (2018) incorporates a confidence branch to obtain out-of-distribution estimations. Our method (iLAP) is the first to incorporate a threshold value that is dependent on class-accuracy changes caused by data poisoning.

heading: 3 APPROACH

text: In this section, we provide an overview of our method. We begin by identifying the learning setting, followed by details of our training process. Finally, insights for choosing threshold values are provided.

heading: 3.1 SETTING

text: In the unsupervised class-incremental setting, a learner L perceives an input stream of exposures denoted as E1, E2, ..., En. Each exposure contains a set of images, Ei = {ei1, ei2, ..., eini}, e i j ∈ RC×H×W , where C, H , and W are the number of channels, height, and width of the input image respectively. Each exposure belongs to a single class yi ∈ N, which has been sampled from class distribution P (C). For each Ei, L does not have knowledge of the true class label yi. Two exemplars, Ptrain = (P 1train, P 2train, ..., P K̂train) and Pval = (P 1val, P 2val, ..., P K̂val), are maintained at all times, where K̂ denotes the total number of classes L has currently determined. The exemplars are used to store samples from the exposure for replay and accuracy assessment. The sizes of both exemplars,∣∣P itrain∣∣ and ∣∣P ival∣∣, are bounded per class.

heading: 3.2 DETECTION TRAINING

text:
For each incoming exposure, the model is tasked to determine whether the class associated with the exposure was learned previously. Our solution is to perform a model update by treating the incoming exposure as a new class, we coin this technique detection training. Under the circumstances that the exposure class is repeated, the performance for the previously learned class would suffer drastically after training. The reason for this behavior is because the model has associated two different labels to a similar class distribution.
During detection training, L̂, a copy of L is produced. The incoming exposure is assigned with label K̂ + 1. Train-validation split is performed on the incoming exposure to obtain Etrain and Eval, and are aggregated with exemplars Ptrain and Pval respectively. The combined samples are used to train L̂ via validation-based early stopping. We denote the vector {∆ŷ}ŷ∈[K̂] to represent the percentage decrease of the class accuracies (computed using Pval) before and after the update. If max({∆ŷ}) exceeds a threshold, θ, the incoming exposure is likely to have been learned by L. In this case, the correct identity pertaining to the exposure is arg maxŷ∈[K̂] ∆ŷ . Otherwise, if θ is unsatisfied, K̂ + 1 is the appropriate label for the new class.

heading: 3.3 CLASS-IMBALANCE FOR DETECTION TRAINING

text:
Introducing a class-imbalance during detection training creates a more distinct decision boundary by exacerbating the class-accuracy drop for repeated exposures. Consider a theoretical case where an optimal model has learned K̂ classes. The incoming exposure, Ei, contains a distribution that is equal to that of some previously learned class ŷi. If the model were updated with equal samples of Ei labeled K̂ + 1, and P ŷitrain labeled ŷi, the accuracy for class ŷi would become ambiguous during validation (ŷi≈ 50%, K̂ + 1≈ 50%). However, if the model were updated with a greater sample of K̂ + 1 labels, the accuracy drop for class ŷi would be considerably larger because K̂ + 1 would be favored during inference.
L̂ has the option to use an imbalanced dataset during detection training where a fraction of the size for each class from Ptrain are used in comparison to the size of Etrain. Let P isampled ⊂ P itrain, the class-imbalance ratio λ is:
λ = 1− |P isampled| |Etrain|
(1)
In Figure 2, we use Fashion-MNIST (Xiao et al. (2017)) and Imagenette (Howard) to obtain a general value for λ and θ for our experiments with other datasets. Performance drops are tracked for the repeated class and non-repeated classes during detection training with various values of λ. The
goal is to maximize the performance loss for the repeated class while maintaining the performance for the other classes. In Section 4 a θ value of 0.6 and a λ value of 0.5 are used. λ is determined by the point of the maximum distance between the two accuracy curves; θ, the expected accuracy drop for a repeated exposure, is the mean of the two values at λ Figure 2.

heading: 3.4 MODEL AND EXEMPLAR UPDATE

text:
After obtaining the predicted label (K̂ + 1 or arg maxŷ∈[K̂] ∆ŷ) from detection training, L is trained with the aggregated dataset obtained from Ptrain and Etrain. Two sets with the most representative samples from Etrain and Etest are created and added to their respective exemplars for future replay. The selection process is determined by ranking the distance between the image features and the class-mean image features. This methodology is consistent with the procedure introduced by Castro et al. (2018).
The availability of Pval allows L to assess how well old and new classes are considered. If the accuracy for any class interpreted by L falls below a percentage, the class is altogether discarded. This allows the model to remove extraneous classes that were learned insufficiently or have been forgotten. In Section 4, a percentage threshold of 20% is used for all experiments.

heading: 4 EXPERIMENTS

text: The performance of our framework is evaluated using a series of image classification benchmarks: MNIST, SVHN, CIFAR-10, CIFAR-100, and CRIB (LeCun (1998); Netzer et al. (2011); Krizhevsky et al. (2009); Stojanov et al. (2019)). First, we compare our novelty detector to related OOD methods. Next, we evaluate performance of iLAP to that of other incremental learners: BiC (Wu et al. (2019)) (supervised) and IOLfCV (unsupervised).

heading: 4.1 EXPERIMENTAL DETAILS

text:
For the following experiments, a ResNet-18 model (He et al. (2016)) pre-trained with ImageNet is used for iLAP and all baselines (additional experiments without pretraining are presented in Appendix A.4). λ = 0.5 and θ = 0.6 are used for iLAP with class-imbalance detection training, while a λ = 0 and θ = 0.4 are used for iLAP without class-imbalance detection training. The parameters are maintained across all benchmarks.
For each exposure, the model is trained for 15 epochs with 16 batch size, using an Adam (Kingma & Ba (2014)) optimizer with validation-based early stopping; a learning rate of 2e−4 is used. The feature extraction layers use a ten times lower learning rate at 2e−5. For all models, the exemplar size per class is equal to the exposure size. The exposure validation split ratio is 0.8 (e.g. for exposure size = 200, iLAP: [Etrain] = 160 and [Eval] = 40). The thresholds used for IOLfCV in 4.1 were determined by maximizing the F-score for the classification of in-distribution exposures versus out-of-distribution exposures. To obtain the best performance possible, the entirety of the dataset was used. The values are 0.46, 0.63, 0.57, and 0.62 for MNIST, SVHN, CIFAR-10, and CIFAR-100 respectively.

heading: 4.2 OUT-OF-DISTRIBUTION DETECTION RESULTS

text: The OOD detectors are assessed in an incremental setting with size 200 exposures. The detectors are evaluated on their ability to determine whether an exposure is novel by using the common established metrics: FPR95, AUROC, and AUPR (Hendrycks & Gimpel (2016)). Details of the compared works and evaluation methods are described in Appendix A.1. The results are illustrated in Table 1.

heading: 4.3 INCREMENTAL LEARNING RESULTS

text:
The accuracy of learner L is computed using the ground-truth mapping m : [K̂] → [K] with the equations:
S(x, y) =
{ 1
|m−1(y)| , if L(x) ∈ m −1(y)
0, otherwise
Accuracy = Ex,y∼test [S(x, y)]
The learner accuracy is the mean of the sample accuracy scores evaluated on the test set, where (x, y) represents a single sample. For each sample with label y, let m−1(y) represent the corresponding labels from the learner. In the case that a learner output does not belong to set m−1(y), an accuracy score of 0 is assigned (class is not detected). Otherwise, an accuracy score of 1|m−1(y)| is designated to penalize the learner if m is non-injective and have attributed multiple labels to a single ground truth class. Performance results are illustrated in Table 2 & 3. Additional visualizations are provided in Appendix A.2, A.3 and A.4.

heading: 5 ANALYSIS

text: Traditional OOD methods that rely on distance-based thresholds are restricted by the supervised samples that are available. These values are non-intuitive and vary drastically across datasets (whereas our percentage threshold are ≈ 50% for all datasets). In the incremental learning setting, early mistakes are amplified as more exposures are introduced, a proper threshold initialization dictates a model’s feasibility. However, we argue that even with a good threshold these methods will consistently fail in particular conditions. The purpose of this section is to discuss the results obtained from our experiments. Subsequently, we highlight a few cases that are overlooked by traditional distance-based methods.

heading: 5.1 OUT-OF-DISTRIBUTION DETECTION ANALYSIS

text: iLAP with class-imbalance detection training (CI) was able to outperform related OOD methods for the MNIST, SVHN, CIFAR-10, and CIFAR-100 benchmarks under all metrics Table 1. However, the results for iLAP without CI were not as definitive. CE performed the worst in the incremental setting, possibly because the performance of the confidence branch is reliant on larger training samples. IOLfCV’s method performed on par with related methods.

heading: 5.2 UNSUPERVISED INCREMENTATL LEARNING ANALYSIS

text:
iLAP with CI was able to beat IOLfCV by 10.0, 41.3, 22.1, 7.0, and 1.0 percentage points for the MNIST, SVHN, CIFAR-10, CIFAR-100, and CRIB benchmarks respectively Table 2. iLAP was also able to maintain its performance when exposure sizes are decreased Appendix A.3. We found that the reason for the lower performance beat for CIFAR-100 and CRIB is not directly attributed to the larger number of classes in the dataset. Rather, the problem lies with how the exposure sequence for the incremental learning setting is created and how the distance-based threshold is calculated.
The threshold for IOLfCV is computed by maximizing the F-score for the binary classification of novel versus non-novel classes. In our experiments, the entirety of the dataset was used to compute the baseline’s threshold. Although this is impractical, we wanted to illustrate that iLAP is able to beat the baseline even under the most optimal conditions.
In Figure 3 we illustrate the behavior of the class feature distances as more classes are incorporated by a network. The most optimal threshold that maximizes the F-score lies in the mid-point between the two graphs when all 100 classes are learned. However, because the threshold is fixed, the novelty detector fails to correctly identify repeated classes early on during training and is more inclined to label repeated classes as unseen, (shaded red area in Figure 3). Consistent with the described setting in Stojanov et al. (2019), each class within a benchmark is repeated an equal number of times in a randomized sequence. For datasets with a large number of classes, there is a higher chance that the repeated exposures are further apart. Therefore, IOLfCV seemingly performs comparatively better on CIFAR-100 than CIFAR-10, but would fail if early repeated exposures were frequent.

heading: 5.3 CLASS SIMILARITY

text:
iLAP was able to detect all classes for MNIST, SVHN, and CIFAR-10 and on average 96.5 out of 100 classes for CIFAR-100. Meanwhile, IOLfCV struggles to identify unique classes for all evaluated benchmarks. Through closer inspection, we found that the distance-based method is unable to distinguish classes when they are too similar.
Consider two classes, k1 and k2, that can be separated by a classifier C in learned feature space F . An incoming exposure, class k3, shares a similar distribution to class k1 in feature space F , but separable in some feature space F ′. The distance-based method is highly probable to fail because it is likely to categorize k1 and k3 as identical classes. However, because our method always trains the incoming exposure as a new class, C is forced to learn the feature space F ′ in which these two classes are separable. Figure 4 illustrates two prior classes, boy and lamp, in some feature space. The incoming exposure, class girl, is unable to be distinguished from class boy by the distance-based method Figure 4 (left). However, because detection training always attempts to classify the incoming exposure as a new class, our method is able to identify F ′ Figure 4 (right).

heading: 6 CONCLUSION

text: To achieve learning in an unsupervised class-incremental setting, a reliable novelty detector is needed. Current methods utilize a detection threshold that is calibrated using class feature distances. In our work, we illustrate that the use of a static distance-based threshold is not only impractical but also unreliable. Instead, we introduce a technique that leverages confusion error to perform novelty detection by always training the incoming exposure as a new class. Using a series of image classification benchmarks, we illustrate that our method is able to closely rival supervised performance despite the lack of labels.

heading: A APPENDIX

text:
A.1 OOD EXPERIMENTAL DETAILS
A brief description of the evaluation metrics used for Table 1.
• False Positive Rate at 95% (FPR95): This measure determines the False Positive Rate (FPR) when True Positive Rate (TPR) is equal to 95%. FPR is calculated as FPFP+TN where FP is the number of False Positives and TN is the number of True Negatives. TPR is calculated as TPTP+FN where TP is the number of True Positives and FN is the number of False Negatives.
• Area Under the Receiver Operating Characteristic (AUROC): This metric illustrates the relationship between the TPR and the FPR. This measure determines the probability that a novel class will have a higher detection score compared to a non-novel class.
• Area Under the Precision-Recall (AUPR): This metric is constructed by plotting precision versus recall. The AUPR curve treats the novel examples as the positive class. A high area value represents high recall and high precision.
The training detection method used in iLAP is compared to a set of related works in OOD. The following works reflect the acronyms used in Table 1
• MSP (Hendrycks & Gimpel (2016)): MSP is computed from a trained classifier to perform out-of-distribution detection. The mean MSP for all images belonging to an exposure is used to determine novelty.
• CE (DeVries & Taylor (2018)): CE requires extending a model with a confidence branch to obtain a set of values. These values reflect the model’s ability to produce a correct prediction. When the mean estimation value for a set of input images is low, the sample is likely to be novel.
• ODIN (Liang et al. (2017)): ODIN uses temperature scaling and input perturbations to widen the MSP difference between in-distribution and out-of-distribution samples. The optimal value for temperature and perturbation magnitude are found by minimizing FPR using grid search. 2 and 0.0012 are used for temperature and perturbation magnitude values respectively, for both the MNIST and CIFAR-10 dataset.
• Zero-Shot OOD (Sastry & Oore (2019)): Zero-Shot OOD uses pairwise correlations of network features to detect novel samples. The class-conditional feature correlations, on
the training data, are computed across all layers of the network. The mean correlations are then compared with the pairwise mean feature correlations from a test sample to obtain a deviation value.
• IOLfCV (Stojanov et al. (2019)): IOLfCV determines a distance-based threshold computed using average-feature means from a network trained from supervised samples. Two initialized classes are used to compute the threshold by finding the optimal point using precision-recall analysis.
A.2 MAIN EXPERIMENT RESULTS
The following are produced with the use of a GTX TITAN X gpu. For each exposure, the model is trained for 15 epochs with 16 batch size, using an Adam optimizer. The learning rate used is 2e−4. The feature extraction layers use a ten times lower learning rate at 2e−5. The input size is 224.
A.3 EXPERIMENTS WITH LOWER EXEMPLAR SIZES
In this section, we showcase iLAP’s results at lower exposure sizes.
A.4 EXPERIMENTS WITHOUT PRE-TRAINING


summaries: The paper addresses how to learn policies for tasks in which constraints are specified in natural language. Towards this, the paper proposes a model that encodes the different types of natural language constraints into intermediate representations that model both spatial and temporal information between states. Then, they use this as input along with the observation to produce an action at each time step for a safe trajectory.

summaries: The paper proposed an algorithm to learn a policy when provided with natural language constraints. The paper defined a navigation task called Hazard World, in which an agent navigation on the map to collect items. The authors defined three types of constraints to restrict agents to visit certain states: 1. budgetary constraints, 2. relational constraints and 3. sequential constraints. The three constraints are described in natural language. The authors proposed a two-step solution. In step one, the algorithm learns a mapping between a natural language constraint to an intermediate representation. In step two the algorithm takes the intermediate representation to learn a policy that satisfy the constraints.

summaries: This paper presents a new test environment, Hazard World, for learning the safe reinforcement learning agents with given natural language constraints. In this problem, the goal of the agent is to find an optimal policy that maximizes the cumulative rewards while satisfying the constraints given in natural language. The authors introduce the model that contains the following two separate components; constraint interpreter for encoding the language constraints and policy network for learning the RL agent. Finally, they report the results of their proposed algorithm and compare it with the baselines.

abstractText: While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HAZARDWORLD, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HAZARDWORLD, we show that our method achieves higher rewards (up to 11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work.1

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis

title: Human-level control through deep reinforcement learning

authors:
David Silver
Richard S. Sutton
Martin Müller

title: Reinforcement learning of local shape in the game of go

venue: In Proceedings of International Joint Conferences on Artificial Intelligence,

authors:
S.R.K. Branavan
David Silver
Regina Barzilay

title: Learning to win by reading manuals in a Monte-Carlo framework

venue: In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,

authors:
Peter Anderson
Qi Wu
Damien Teney
Jake Bruce
Mark Johnson
Niko Sünderhauf
Ian D. Reid
Stephen Gould
Anton van den Hengel

title: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Dipendra Misra
Andrew Bennett
Valts Blukis
Eyvind Niklasson
Max Shatkhin
Yoav Artzi

title: Mapping instructions to actions in 3D environments with visual goal prediction

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Gabriel Dulac-Arnold
Daniel Mankowitz
Todd Hester

title: Challenges of real-world reinforcement learning

venue: arXiv preprint arXiv:1904.12901,

authors:
Joshua Achiam
David Held
Aviv Tamar
Pieter Abbeel

title: Constrained policy optimization

venue: In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning,

authors:
Yinlam Chow
Ofir Nachum
Aleksandra Faust
Mohammad Ghavamzadeh
Edgar Duenez- Guzman

title: Lyapunov-based safe policy optimization for continuous control

authors:
Tsung-Yen Yang
Justinian Rosca
Karthik Narasimhan
Peter J. Ramadge

title: Projection-based constrained policy optimization

venue: In 8th International Conference on Learning Representations,

title: Accelerating safe reinforcement learning with constraint-mismatched policies

venue: arXiv preprint arXiv:2006.11645,

authors:
Joshua Achiam
Dario Amodei

title: Benchmarking safe exploration in deep reinforcement learning

authors:
Yinlam Chow
Ofir Nachum
Edgar Duenez-Guzman
Mohammad Ghavamzadeh

title: A lyapunov-based approach to safe reinforcement learning

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

authors:
Felix Berkenkamp
Matteo Turchetta
Angela Schoellig
Andreas Krause

title: Safe model-based reinforcement learning with stability guarantees

venue: Advances in Neural Information Processing Systems,

authors:
Mahmoud El Chamie
Yue Yu
Behçet Açıkmeşe

title: Convex synthesis of randomized policies for controlled markov chains with density safety upper bound constraints

venue: In 2016 American Control Conference (ACC),

authors:
Matteo Turchetta
Andrey Kolobov
Shital Shah
Andreas Krause
Alekh Agarwal

title: Safe reinforcement learning via curriculum induction

venue: arXiv preprint arXiv:2006.12136,

authors:
Brijen Thananjeyan
Ashwin Balakrishna
Suraj Nair
Michael Luo
Krishnan Srinivasan
Minho Hwang
Joseph E. Gonzalez
Julian Ibarz
Chelsea Finn
Ken Goldberg

title: Recovery rl: Safe reinforcement learning with learned recovery zones

venue: IEEE Robotics and Automation Letters,

authors:
Maxime Chevalier-Boisvert
Dzmitry Bahdanau
Salem Lahlou
Lucas Willems
Chitwan Saharia
Thien Huu Nguyen
Yoshua Bengio

title: Babyai: A platform to study the sample efficiency of grounded language learning

venue: In 7th International Conference on Learning Representations,

authors:
Alex Ray
Joshua Achiam
Dario Amodei

title: Benchmarking Safe Exploration in Deep Reinforcement Learning

venue: arXiv preprint arXiv:1910.01708,

authors:
Matt MacMahon
Brian Stankiewicz
Benjamin Kuipers

title: Walk the talk: Connecting language, knowledge, and action in route instructions

venue: In Proceedings of the Conference on Artificial Intelligence,

authors:
David L. Chen
Raymond J. Mooney

title: Learning to interpret natural language navigation instructions from observations

venue: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence,

authors:
Yoav Artzi
Luke Zettlemoyer

title: Weakly supervised learning of semantic parsers for mapping instructions to actions

venue: Transactions of the Association for Computational Linguistics,

authors:
Dipendra Misra
John Langford
Yoav Artzi

title: Mapping instructions and visual observations to actions with reinforcement learning

venue: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,

authors:
Karl Moritz Hermann
Mateusz Malinowski
Piotr Mirowski
Andras Banki-Horvath
Keith Anderson
Raia Hadsell

title: Learning to follow directions in street view

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Weituo Hao
Chunyuan Li
Xiujun Li
Lawrence Carin
Jianfeng Gao

title: Towards learning a generic agent for vision-and-language navigation via pre-training

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Javier Garcia
Fernando Fernandez

title: A comprehensive survey on safe reinforcement learning

venue: Journal of Machine Learning Research,

authors:
Thiago D Simão
Nils Jansen
Matthijs TJ Spaan

title: Alwayssafe: Reinforcement learning without safety constraint violations during training

venue: In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems,

authors:
Chen Tessler
Daniel J. Mankowitz
Shie Mannor

title: Reward constrained policy optimization

authors:
Adam Stooke
Joshua Achiam
Pieter Abbeel

title: Responsive safety in reinforcement learning by PID lagrangian methods

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
Stéphane Ross
Geoffrey Gordon
Drew Bagnell

title: A reduction of imitation learning and structured prediction to no-regret online learning

venue: In Proceedings of International Conference on Artificial Intelligence and Statistics,

authors:
Aravind Rajeswaran
Vikash Kumar
Abhishek Gupta
Giulia Vezzani
John Schulman
Emanuel Todorov
Sergey Levine

title: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

venue: In Proceedings of Robotics: Science and Systems,

authors:
Yang Gao
Ji Lin
Fisher Yu
Sergey Levine
Trevor Darrell

title: Reinforcement learning from imperfect demonstrations

venue: In Proceedings of the 35th International Conference on Machine Learning,

authors:
Adam Vogel
Daniel Jurafsky

title: Learning to follow navigational directions

venue: In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics,

authors:
Stefanie Tellex
Thomas Kollar
Steven Dickerson
Matthew Walter
Ashis Banerjee
Seth Teller
Nicholas Roy

title: Understanding natural language commands for robotic navigation and mobile manipulation

authors:
Joohyun Kim
Raymond Mooney

title: Adapting discriminative reranking to grounded language learning. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 218–227

venue: Sofia, Bulgaria,

authors:
Jacob Andreas
Dan Klein

title: Alignment-based compositional semantics for instruction following

venue: In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,

authors:
Jesse Thomason
Michael Murray
Maya Cakmak
Luke Zettlemoyer

title: Vision-and-dialog navigation

venue: In Proceedings of Conference on Robot Learning,

authors:
Jelena Luketina
Nantas Nardelli
Gregory Farquhar
Jakob N. Foerster
Jacob Andreas
Edward Grefenstette
Shimon Whiteson
Tim Rocktäschel

title: A survey of reinforcement learning informed by natural language

venue: In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence,

authors:
Stefanie Tellex
Nakul Gopalan
Hadas Kress-Gazit
Cynthia Matuszek

title: Robots that use language

venue: Annual Review of Control, Robotics, and Autonomous Systems,

authors:
Christopher Wang
Candace Ross
Yen-Ling Kuo
Boris Katz
Andrei Barbu

title: Learning a natural-language to ltl executable semantic parser for grounded robotics

venue: arXiv preprint arXiv:2008.03277,

authors:
Yonatan Bisk
Kevin J. Shih
Yejin Choi
Daniel Marcu

title: Learning interpretable spatial operations in a rich 3d blocks world

venue: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,

authors:
Howard Chen
Alane Suhr
Dipendra Misra
Noah Snavely
Yoav Artzi

title: TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Harm de Vries
Kurt Shuster
Dhruv Batra
Devi Parikh
Jason Weston
Douwe Kiela

title: Talk the walk: Navigating new york city through grounded dialogue

venue: arXiv preprint arXiv:1807.03367,

authors:
Michael Janner
Karthik Narasimhan
Regina Barzilay

title: Representation learning for grounded spatial reasoning

authors:
Valts Blukis
Dipendra Misra
Ross A. Knepper
Yoav Artzi

title: Mapping navigation instructions to continuous control actions with position-visitation prediction

venue: In Proceedings of the Conference on Robot Learning,

authors:
Daniel Fried
Ronghang Hu
Volkan Cirik
Anna Rohrbach
Jacob Andreas
Louis-Philippe Morency
Taylor Berg-Kirkpatrick
Kate Saenko
Dan Klein
Trevor Darrell

title: Speakerfollower models for vision-and-language navigation

venue: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems

authors:
Ninghao Liu
Mengnan Du
Xia Hu

title: Representation interpretation with spatial encoding and multimodal analytics

venue: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining,

authors:
Vihan Jain
Gabriel Magalhaes
Alexander Ku
Ashish Vaswani
Eugene Ie
Jason Baldridge

title: Stay on the path: Instruction fidelity in vision-and-language navigation

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
David Gaddy
Dan Klein

title: Pre-learning environment representations for data-efficient neural instruction following

authors:
Yordan Hristov
Daniel Angelov
Michael Burke
Alex Lascarides
Subramanian Ramamoorthy

title: Disentangled relational representations for explaining and learning from demonstration

authors:
Justin Fu
Anoop Korattikara
Sergey Levine
Sergio Guadarrama

title: From language to goals: Inverse reinforcement learning for vision-based instruction following

authors:
Sagar Gubbi Venkatesh
Anirban Biswas
Raviteja Upadrashta
Vikram Srinivasan
Partha Talukdar
Bharadwaj Amrutur

title: Spatial reasoning from natural language instructions for robot manipulation

venue: arXiv preprint arXiv:2012.13693,

authors:
Philip S. Thomas
Bruno Castro da Silva
Andrew G. Barto
Stephen Giguere
Yuriy Brun
Emma Brunskill

title: Preventing undesirable behavior of intelligent

venue: machines. Science,

authors:
Thomas M Howard
Stefanie Tellex
Nicholas Roy

title: A natural language planner interface for mobile manipulators

venue: In 2014 IEEE International Conference on Robotics and Automation (ICRA),

authors:
Bharat Prakash
Nicholas Waytowich
Ashwinkumar Ganesan
Tim Oates
Tinoosh Mohsenin

title: Guiding safe reinforcement learning policies using structured language constraints

venue: UMBC Student Collection,

title: Constrained Markov decision processes, volume 7

venue: CRC Press,

authors:
Maxime Chevalier-Boisvert
Lucas Willems
Suman Pal

title: Minimalistic gridworld environment for openai gym

authors:
Michael Buhrmester
Tracy Kwang
Samuel D. Gosling

title: Amazon’s mechanical turk: A new source of inexpensive, yet high-quality data? 2016

authors:
Aaron Walsman
Yonatan Bisk
Saadia Gabriel
Dipendra Misra
Yoav Artzi
Yejin Choi
Dieter Fox

title: Early fusion for goal directed robotic vision

venue: In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
John Schulman
Sergey Levine
Pieter Abbeel
Michael I. Jordan
Philipp Moritz

title: Trust region policy optimization

venue: Proceedings of the 32nd International Conference on Machine Learning, ICML 2015,

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparameterization with gumbel-softmax. volume

authors:
John Schulman
Philipp Moritz
Sergey Levine
Michael I. Jordan
Pieter Abbeel

title: Highdimensional continuous control using generalized advantage estimation

venue: In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations,

authors:
Yan Duan
Xi Chen
Rein Houthooft
John Schulman
Pieter Abbeel

title: Benchmarking deep reinforcement learning for continuous control

venue: Proceedings of the 33nd International Conference on Machine Learning,

heading: 1 Introduction

text:
Although reinforcement learning (RL) has shown promise in several simulated domains such as games [1, 2, 3] and autonomous navigation [4, 5], deploying RL in real-world scenarios remains challenging [6]. In particular, real-world RL requires ensuring the safety of the agent and its surroundings, which means accounting for constraints during training that are orthogonal to maximizing rewards. For example, a cleaning robot must be careful to not knock the television over, even if the television lies on the optimal path to cleaning the house.
Safe RL tackles these challenges with algorithms that maximize rewards while simultaneously minimizing constraint violations during exploration [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. However, these algorithms have two key limitations that prevent their widespread use. First, they require us
∗ Equal contribution. 1Code and data are available at https://github.com/princeton-nlp/SRL-NLC
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
to provide constraints in mathematical or logical forms, which calls for specific domain expertise. Second, a policy trained with a specific set of constraints cannot be transferred easily to learn new tasks with the same set of constraints, since current approaches do not maintain an explicit notion of constraints separate from reward-maximizing policies. This means one would have to retrain the policy (with constraints) from scratch.
We consider the use of natural language to specify constraints (which are orthogonal to rewards) on learning. Human languages provide an intuitive and easily-accessible medium for describing constraints–not just for machine learning experts or system developers, but also for potential end users interacting with agents such as household robots. Consider the environment in Fig. 1 for example. Instead of expressing a constraint as ∑T t=0 1st∈lava · 16∃st′∈water, t′∈[0,1,...,t−1] = 0, one could simply say “Do not visit the lava before visiting the water”. The challenge of course, lies in training the RL agent to accurately interpret and adhere to the textual constraints as it learns a policy for the task.
To study this problem, we first create HAZARDWORLD, a collection of grid-world and robotics environments for safe RL with textual constraints (Fig. 1). HAZARDWORLD consists of separate ‘safety training’ and ‘safety evaluation’ sets, with disjoint sets of reward functions and textual constraints between training and evaluation. To do well on HAZARDWORLD, an agent has to learn to interpret textual con-
straints during safety training and safely adhere to any provided constraints while picking up new tasks during the safety evaluation phase. Built on existing RL software frameworks [17, 18], HAZARDWORLD consists of navigation and object collection tasks with diverse, crowdsourced, free-form text specifying three kinds of constraints: (1) budgetary constraints that limit the frequency of being in unsafe states, (2) relational constraints that specify unsafe states in relation to surrounding entities, and (3) sequential constraints that activate certain states to be unsafe based on past events (e.g., “Make sure you don’t walk on water after walking on grass”). Our setup differs from instruction following [19, 20, 21, 22, 23, 24] in two ways. First, instructions specify what to do, while textual constraints only inform the agent on what not to do, independent of maximizing rewards. Second, learning textual constraints is a means for ensuring safe exploration while adapting to a new reward function.
In order to demonstrate learning under this setting, we develop Policy Optimization with Language COnstraints (POLCO), where we disentangle the representation learning for textual constraints from policy learning. Our model first uses a constraint interpreter to encode language constraints into representations of forbidden states. Next, a policy network operates on these representations and state observations to produce actions. Factorizing the model in this manner allows the agent to retain its constraint comprehension capabilities while modifying its policy network to learn new tasks.
Experiments demonstrate that our approach achieves higher rewards (up to 11x) while maintaining lower constraint violations (up to 1.8x) compared to several baselines on two different domains within HAZARDWORLD. Nevertheless, HAZARDWORLD remains far from being solved, especially in tasks with high-dimensional observations, complex textual constraints and those requiring high-level planning or memory-based systems.

heading: 2 Related Work

text:
Safe RL. Safe RL deals with learning constraint-satisfying policies [25], or learning to maximize rewards while minimizing constraint violations.2 This is a constrained optimization problem, and thus different from simply assigning negative reward values to unsafe states. Furthermore, large negative reward values for constraint violations can destabilize training and lead to degenerate behavior, such as the agent refusing to move. In prior work, the agent typically learns policies either by (1) exploring the environment to identify forbidden behaviors [7, 27, 8, 9, 28], or (2) using expert demonstration data to recognize safe trajectories [29, 30, 31, 10]. All these works require a human to specify the cost constraints in mathematical or logical form, and the learned constraints cannot be easily reused for new learning tasks. In this work, we design a modular architecture to learn to interpret textual constraints, and demonstrate transfer to new learning tasks.
Instruction following. Our work closely relates to the paradigm of instruction following in RL, which has previously been explored in several environments [19, 32, 20, 33, 21, 34, 35, 36, 37, 38, 39]. Prior work has also focused on creating realistic vision-language navigation datasets [40, 41, 4, 42] and proposed computational models to learn multi-modal representations that fuse images with goal instructions [43, 44, 45, 46, 47, 48, 49, 50, 51]. Our work differs from the traditional instruction following setup in two ways: (1) Instruction following seeks to (roughly) ‘translate’ an instruction directly into an action policy. This does not apply to our setting since the textual constraints only tell an agent what not to do. To actually obtain rewards, the agent has to explore and figure out optimal policies on its own. (2) Since constraints are decoupled from rewards and policies, agents trained to understand certain constraints can transfer their understanding to respect these constraints in new tasks, even when the new optimal policy is drastically different. Therefore, we view this work as orthogonal to traditional instruction following–one could of course combine both instructions and textual constraints to simultaneously advise an agent on what to do and what not to do.
Connection to Seldonian algorithms. POLCO can also be interpreted as a Seldonian algorithm [52]. Seldonian algorithms ensure ML safety through three steps: (1) defining a goal, (2) defining an interface for users to provide constraints, and (3) creating an algorithm that satisfies the goal and constraints. Here, we use natural language as the interface for end users and map natural language into optimization constants and vector representations. Thus, POLCO is also a potential step towards widely developing and deploying Seldonian algorithms.
Constraints in natural language. Our notion of ‘constraints’ in this paper differs from prior work that uses instructions to induce planning constraints [33, 53, 39]–these works again provide instructions for the agent on how to perform the task. Perhaps closest to this paper is the work of Misra et al. [5], which proposes datasets to study spatial and temporal reasoning, containing a subset focusing on trajectory constraints (e.g., “go past the house by the right side of the apple”). However, they do not disentangle the rewards from the constraints, which may be orthogonal to each other. Prakash et al. [54] train a constraint checker to identify whether a constraint (specified in text) has been violated in a trajectory. While their motivation is similar, they ultimately convert constraints to negative rewards, whereas we use a modular approach that allows disentangling reward maximization from minimizing constraint violations and is compatible with modern algorithms for safe RL.

heading: 3 Preliminaries

text:
Problem formulation. Our learning problem can be viewed as a partially observable constrained Markov decision process [55], which is defined by the tuple < S,O,A, T, Z,X , R, C >. Here S is the set of states, O is the set of observations, A is the set of actions, T is the conditional probability T (s′|s, a) of the next state s′ given the current state s and the action a, and Z is the conditional probability Z(o|s) of the observation o given the state s. In addition, X is the set of textual constraint specifications, R : S ×A → R is the reward function, which encodes the immediate reward provided when the agent takes an action a in state s, and C : S ×A×X → R is the true underlying constraint function described by x ∈ X , which specifies positive penalties for constraint violations due to an action a in a state s. Finally, we assume each x ∈ X corresponds to a specific cost function C.
2In this paper, we consider minimizing constraint violations in expectation [9] and leave stricter notions such as enforcing zero violations [26] to future work.
RL with constraints. The goal of the learning agent is to acquire a good control policy that maximizes rewards, while adhering to the specified constraints as much as possible during the learning process. Thus, the agent learns a policy π : O ×X → P(A), which is a mapping from the observation space O and constraint specification X to the distributions over actions A. Let γ ∈ (0, 1) denote a discount factor, µ(S) denote the initial state distribution, and τ denote a trajectory sequence of observations and actions induced by a policy π, i.e., τ = (o0, a0, o1, · · · ). For any given x, we seek a policy π that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost JC below a specified cost constraint threshold hC(x):
max π JR(π) . = E τ∼π [ ∞∑ t=0 γtR(st, at) ] s.t. JC(π) . = E τ∼π [ ∞∑ t=0 γtC(st, at, x) ] ≤ hC(x),
where τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : s0 ∼ µ, ot ∼ Z(·|st), at ∼ π(·|ot, x), st+1 ∼ T (·|st, at). We use C(st, at, x) and hC(x) here to emphasize that both functions depend on the particular constraint specification x.
Task setup. Our goal is to show that constraints specified in natural language allow for generalization to new tasks that require similar constraints during learning. With this in mind, we consider the following safety training and safety evaluation setup:
(1) Safety training: During training, we generate random environment layouts and starting states s0 while keeping the reward function R fixed. For each episode, we randomly generate a constraint function C and limit hC . We then sample a constraint text x that describes C and hC from the training set of texts. The constraint text x is an input to the agent’s policy. Whenever the agent violates a constraint (at any step), it is provided with a scalar cost penalty learned by the model from C(s, a, x). The agent, therefore, sees a variety of different task layouts and constraints, and learns a policy with respect to the constraints for this task as well as how to interpret textual constraints.
(2) Safety evaluation: During evaluation, we place the agent in new environments with randomly generated layouts, with a different reward function R′. The set of possible constraints C is the same as seen in training, but the corresponding constraint texts are from an unseen test set. During this phase, the agent is not provided any cost penalties from the task. This setup allows us to measure two things: (1) how well an agent can learn new tasks while following previously learned textual constraints, and (2) the applicability of our method when using textual constraints unseen in training.

heading: 4 HAZARDWORLD

text:
To our knowledge, there do not currently exist datasets for evaluating RL agents that obey textual constraints.3 Thus, we design a new benchmark called HAZARDWORLD in which the agent starts
3Even though there are several instruction following tasks, our task setup is different, as mentioned previously.
each episode at a random location within a procedurally generated environment and receives a textual constraint x, sampled from a pool of available constraints. The agent’s goal is to collect all the reward-providing entities while adhering to the specified constraint. Other than the constraint specified, the agent has complete freedom and is not told about how to reach reward-providing states.
HAZARDWORLD contains three types of constraints–(1) budgetary constraints, which impose a limit on the number of times a set of states can be visited, (2) relational constraints, which define a minimal distance that must be maintained between the agent and a set of entities, and (3) sequential constraints, which are constraints that activate unsafe states when a specific condition has been met. In total, we collect 984 textual constraints for HAZARDWORLD-grid (GridWorld environment) and 2,381 textual constraints for HAZARDWORLD-robot (robotic tasks). Table 1 provides examples.
HAZARDWORLD-grid. We implement HAZARDWORLD-grid (Table 1(a)) atop the 2D GridWorld layout of BabyAI [17, 56]. We randomly place three reward entities on the map: ‘ball,’ ‘box,’ and ‘key,’ with rewards of 1, 2, and 3, respectively. We also randomly place several cost entities on the map: ‘lava,’ ‘water,’ and ‘grass’. We give a cost penalty of 1 when agents step onto any cost entities, which are specified using a textual constraint x. The entire state st is a grid of size 13×13, including the walls, and the agent’s observation ot is a 7×7 grid of its local surroundings. There are 4 actions–for moving up, down, left and right. We use the deterministic transition here.
Train-test split. We generate two disjoint training and evaluation datasets Dtrain and Deval. Dtrain consists of 10,000 randomly generated maps paired with 80% of the textual constraints (787 constraints overall), i.e., on average each constraint is paired with 12.70 different maps. Deval consists of 5,000 randomly generated maps paired with the remaining 20% of the textual constraints (197 constraints), i.e., on average one constraint is paired with 25.38 maps. In Deval we change the rewards for ball, box, and key to 1, 2, and -3, respectively. Therefore, in Deval, the agent has to avoid collecting the key to maximize reward.
HAZARDWORLD-robot. We build HAZARDWORLD-robot (Table. 1(b)) atop the SAFETY GYM environment [18] to show the applicability of our model to tasks involving high-dimensional continuous observations. In this environment, there are five constraint entities paired with textual constraints: hazards (dark blue puddles), vases (stationary but movable teal cubes), pillars (immovable cylinders), buttons (touchable orange spheres), and gremlins (moving purple cubes). This task is more challenging than the 2D case since some obstacles are constantly moving. The agent receives a reward of 4 for reaching a goal position and a cost penalty of 1 for bumping into any constraint entities. The observation ot is a vector of size 109, including coordinate location, velocity of the agent, and observations from lidar rays that detect the distance to entities. The agent has two actions–control signals applied to the actuators to make it move forward or rotate. The transitions are all deterministic.
Train-test split. We follow the same process for obtaining a train-test split as in HAZARDWORLD-grid. Dtrain consists of 10,000 randomly generated maps paired with 80% of textual constrains (1,905 constraints), i.e., on average one constraint is paired with 5.25 maps. Deval consists of 1,000 randomly generated maps paired with the remaining 20% of textual constrains (476 constraints), i.e., on average one constraint is paired with 2.10 maps. In Deval we add four additional goal locations to each map (i.e., the maximum reward is 20). The agent has to learn to navigate to these new locations.
Data collection. For the textual constraints in both environments, we collected free-form text in English using Amazon Mechanical Turk (AMT) [57]. To generate a constraint for HAZARDWORLD, we provided workers with a description and picture of the environment, the cost entity to be avoided, and one of the following: (a) the cost budget (budgetary), (b) the minimum safe distance (relational), or (c) the other cost entity impacted by past events (sequential). We then cleaned the collected text by writing a keyword matching script followed by manual verification to ensure the constraints are valid.

heading: 5 Learning to Interpret Textual Constraints

text: We seek to train agents that can adhere to textual constraints even when learning policies for new tasks with different reward structures. We now describe our model and training and evaluation procedures.

heading: 5.1 Model

text:
We design the RL agent as a deep neural network that consists of two parts (Fig. 2)–(1) a constraint interpreter which processes the text into structured safety criteria (a constraint mask and threshold)
and (2) a policy network which uses the output of the interpreter along with observations to produce an action. For simplicity, in the following descriptions, we assume state s and observation o to be 2D matrices, although the model can easily be extended to other input representations.
(1) Constraint interpreter (Fig. 3). We concatenate an observation embedding of size n× n×m from observations o of size n× n with the embedding of the textual constraints x of size l from a long-short-term-memory (LSTM), followed by using a convolutional neural network (CNN) to get an embedding vector. We use this vector to produce a constraint mask M̂C , a binary matrix with the same dimension as o–each cell of the matrix is 0/1 depending on whether the model believes the absence or presence of a constraint-related entity (e.g., ‘lava’) in the corresponding cell of the observation o. In addition, we feed the textual constraints into an LSTM to produce ĥC , a real-valued scalar which predicts the constraint threshold, i.e., the number of times an unsafe state is allowed.
For the case of sequential constraints with long-term dependency of the past events, M̂C will depend on the past states visited by the agent. For example, in Fig. 3(b), after the agent visits ‘water’, M̂C starts to locate the cost entity (‘grass’). Thus, for sequential constraints, we modify the interpreter by adding an LSTM layer before computing M̂C to take the state history into account. Using M̂C and ĥC allows us to embed textual constraints in the policy network.
(2) Policy network. The policy network produces an action using the state observation ot and the safety criteria produced by the constraint interpreter. The environment embedding is concatenated with the constraint mask M̂C (predicted by the constraint interpreter) and a cost budget mask, denoted by M̂B . The cost budget mask is derived from ĥC (also predicted by the constraint interpreter) and keeps track of the number of constraint violations that the agent has made in the past over the threshold. M̂B is an n × n matrix where each element takes the value of ∑t′ t=0 Ĉ(st, at;x) − ĥC (i.e., the value of constraint violations past the budget until t′th step) if there is a cost entity in ot(i, j), or zero otherwise. During the safety evaluation phase, we estimate the cumulative cost ∑t′ t=0 Ĉ(st, at;x) using the predicted M̂C and the agent’s current location at time t. After concatenating both the constraint mask M̂C and cost budget mask M̂B to the observation embedding, we then feed the resulting tensor into CNN to obtain a vector (grey in Fig. 2). This vector is concatenated with a vectorized int(ĥC) (i.e., ĥC rounded down) and fed into an MLP to produce an action.
POLCO in HAZARDWORLD-robot. To apply POLCO in this environment, the constraint interpreter predicts the cost entity given the textual constraints. We then map the cost entity to the pre-defined embedding vector (i.e., one-hot encoding). We then concatenate the embedding vector, the embeddings of the predicted ĥC , and the value of cost budget (rounded down) to the observation vector. Finally, the policy network takes in this concatenated observation and produces a safe action.
Advantages of the design. The design of POLCO tightly incorporates textual constraints into the policy network. Our model factorization–into (1) a constraint interpreter and (2) a policy network–
allows us to design specific constraint interpreters for different types of constraints.4 Furthermore, our approach scales gracefully to handling multiple constraints. While existing safe RL algorithms require retraining the policy for each unique combination of constraints, we can simply add together the M̂C of each constraint to handle multiple constraints imposed simultaneously.

heading: 5.2 Safety training

text:
We first train the constraint interpreter using a random policy to collect trajectories, and then we use the trained interpreter to predict constraints while training the policy network.
Stage 1: Interpreter learning. We use a random policy to explore the environment, and obtain trajectories consisting of observations ot, along with the corresponding textual constraint x. Using the constraint violations encountered in the trajectory and the cost specification C, we obtain a target MC for training the constraint interpreter. In addition, we also obtain the ground-truth value of hC for learning the constraint threshold module.
We train the constraint mask module of the constraint interpreter by minimizing the following binary cross-entropy loss over these trajectories: L(Θ1) = −E(ot,x)∼Dtrain [ 1 |MC | ∑n i,j=1 y log ŷ + (1 −
y) log(1 − ŷ) ] , where y is the target MC(i, j; ot, x), which denotes the target (binary) mask label
in ith row and jth column of the n × n observation ot, ŷ is the predicted M̂C(i, j; ot, x), i.e., the probability prediction of constraint mask, and Θ1 are the parameters of the constraint mask module.
For the constraint threshold module, we minimize the following loss: L(Θ2) = E(ot,x)∼Dtrain [ (hC(x)− ĥC(x))2 ] , where Θ2 are the parameters of the constraint threshold module.
This approach ensures cost satisfaction during both policy learning and safety evaluation, an important feature of safe RL. If we train both the policy and the interpreter simultaneously, then we risk optimizing according to inaccurate M̂C and ĥC values, as observed in our experiments.
Stage 2: Policy learning. We use a safe RL algorithm called projection-based constrained policy optimization (PCPO) [9] to train the policy network. During training, the agent interacts with the environment to obtain rewards and penalty costs (M̂C) are provided from the trained constraint interpreter for computing JR(π) and JC(π) (ground-truth C is not used). PCPO is an iterative method that performs two key steps in each iteration5–optimize the policy according to reward and project the policy to a set of policies that satisfy the constraint. During safety evaluation, we evaluate our model in the new task with the new reward function and the textual constraints from Deval.
4M̂B equates to a scaled up version of M̂C since we assume only one constraint specification per episode, but this is not necessary in general since we may have multiple constraints over different cost entities. In that case, M̂B may have different cost budgets for different cells (entities).
5One can use other safe RL algorithms such as Constrained Policy Optimization (CPO) [7]

heading: 5.3 Safety evaluation

text:
(1) Transfer to new tasks: We take the policy trained in Dtrain and fine-tune it on tasks having new reward functions with textual constraints from Deval. We do not retrain the constraint interpreter on Deval. The policy is fine-tuned to complete the new tasks without the penalty signals from the cost function C. In HAZARDWORLD-robot, we optimize the policy using CPO [7].
(2) Handling multiple textual constraints: We also test the ability of our model to handle multiple constraints imposed simultaneously (from Deval), by adding the cost constraint masks M̂C of each constraint together when given multiple constraints. During safety training, the policy is still trained with a single constraint. No fine-tuning is performed and the reward function is maintained the same across training and evaluation in this case.

heading: 6 Experiments

text: Our experiments aim to study the following questions: (1) Does the policy network, using representations from the constraint interpreter, achieve fewer constraint violations in new tasks with different reward functions? (2) How does each component in POLCO affect its performance?

heading: 6.1 Setup

text:
Baselines. We consider the following baselines: (1) Constraint-Fusion (CF) with PCPO: This model [58] takes a concatenation of the observations and text representations as inputs (without MC , MB and hC ) and produces an action, trained with an end-to-end approach using PCPO. This model jointly processes the observations and the constraints. (2) CF with TRPO: We train CF using trust region policy optimization (TRPO) [59], which ignores all constraints and only optimizes the reward. This is to demonstrate that the agent will have substantial
constraint violations when ignoring constraints. (3) Random Walk (RW): We also include a random walk (RW) baseline, where the agent samples actions uniformly at random.
Evaluation metrics. To evaluate models, we use (1) the average value of the reward JR(π), and (2) the average constraint violations ∆C := max(0, JC(π)−hC). Good models should have a small ∆C (i.e., close to zero) while maximizing JR(π). More details on the implementation, hyper-parameters, and computational resources are included in the Appendix A,B, and C.

heading: 6.2 Results

text:
HAZARDWORLD-grid. Fig. 4(1) shows results for all models in the first evaluation setting of transfer to new tasks. POLCO has lower constraint violations in excess of hC while still achieving better reward performance in all cases. In comparison, the high cost values (∆C ) obtained by RW and CF with TRPO indicate the challenges of task. This supports our idea of using the learned constraint interpreter to learn a new task with similar textual constraints while ensuring constraint satisfaction. CF with PCPO has higher constraint violations, and in the most cases, does not optimize the reward, which suggests that it cannot transfer the constraint understanding learned in Dtrain to Deval.
Fig. 4(2) shows our evaluation with multiple textual constraints. We see that POLCO achieves superior reward and cost performance compared to the baselines, while CF with PCPO has worse reward and cost performance. This shows that our approach is flexible enough to impose multiple constraints than that of existing safe RL methods which requires retraining the policy for each unique combination of constraints.
HAZARDWORLD-robot. Fig. 5 shows transfer to new tasks in HAZARDWORLD-robot. The JR(π) and ∆C of RW is relatively small since the agent does not move much because of random force applied to each actuator. For the budgetary constraints, although CF with TRPO achieves the best reward when hC = 0, it has very large constraint violations. POLCO performs better than the baselines–it induces policies with higher reward under fewer constraint violations in most cases. In contrast, CF with CPO has lower reward performance.
Having demonstrated the overall effectiveness of POLCO, our remaining experiments analyze (1) the learned models’ performance evaluated on the same reward function as in Dtrain, and (2) the importance of each component–MB ,MC and hC embedding in POLCO. For compactness, we restrict our consideration in HAZARDWORLD-grid.
Evaluation with reward function from Dtrain. To provide another point of comparison in addition to our main results, we evaluate all models using the same reward function as in Dtrain, but with unseen textual constraints from Deval. ( Fig. 6) We observe POLCO achieves the lowest violations across different choices of hC compared to the baselines. This implies that merely combining the observations and the text is not sufficient to learn an effective representation for parsing the constraints. In addition, POLCO achieves the best reward performance under cost satisfaction for the more complex relational and sequential constraints. For the relational case, although the CF agent trained with PCPO satisfies the constraints, it has a relatively low reward.
Ablation studies. We also examine the importance of each part in POLCO (Fig. 7). To eliminate prediction errors from the constraint interpreter, we use the true MC and hC here. Our full model achieves the best performance in all cases, averaging 5.12% more reward and 2.22% fewer constraint violations. Without MC , the agent cannot recognize cost entities effectively, which causes the agent to incur 66.67% higher ∆C compared with the full model (which has a ∆C close to zero). This shows that hC embedding and the MB mask are useful in enabling constraint satisfaction given textual constraints.

heading: 7 Conclusion

text:
Our work provides a view towards machines that can interoperate with humans. As autonomous agents proliferate into our world, they should be able to understand safety constraints set by human agents around them. Accordingly, we proposed the problem of safe RL with natural language constraints, created a new benchmark called HAZARDWORLD to test agents and developed a new algorithm for the task (POLCO) that learns to interpret constraints. Our paper defines and trains machine agents that understand what not to do in natural language, much like instruction following tasks enable agents in understanding what to do.
The thesis of our POLCO approach is that modularity enables reuse. By bootstrapping a modular constraint interpreter through exploration, our model scales easily to multiple constraints and to shifts in the environment’s reward structure, all while exploring new environments safely. We applied POLCO within HAZARDWORLD to train an agent that navigates safely by obeying natural language constraints. This agent is a step towards creating applications like cleaning robots that can obey free form constraints, such as “don’t get too close to the TV” – a relational constraint in our formulation.
No model is without limitations. The absolute scores of POLCO on HAZARDWORLD still leave a lot of room for improvement using better models or training techniques. The current version of HAZARDWORLD is also not all-encompassing – we envision it as a benchmark that evolves over time, with the addition of new types of constraints and new environments. Future work can investigate training without explicit labels for the constraint interpreter, potentially using techniques like Gumbel softmax [60], or extending POLCO to tasks with more realistic visuals.

heading: Acknowledgments

text:
The authors would like to thank members of the Princeton NLP Group, the anonymous reviewers, and the area chair for their comments and feedback. Tsung-Yen Yang thanks Siemens Corporation, Corporate Technology for their support.
Funding Transparency Statement
Funding in direct support of this work: scholarship by Siemens Corporation, Corporate Technology, and the Princeton CSML Summer Research Award.

title: Safe Reinforcement Learning with Natural Language Constraints