SEMI-SUPERVISED DETECTION OF EXTREME WEATHER EVENTS IN LARGE CLIMATE DATASETS
The detection and identification of extreme weather events in large scale climate simulations is an important problem for risk management, informing governmental policy decisions and advancing our basic understanding of the climate system. Recent work has shown that fully supervised convolutional neural networks (CNNs) can yield acceptable accuracy for classifying well-known types of extreme weather events when large amounts of labeled data are available. However, there are many different types of spatially localized climate patterns of interest (including hurricanes, extra-tropical cyclones, weather fronts, blocking events, etc.) found in simulation data for which labeled data is not available at large scale for all simulations of interest. We present a multichannel spatiotemporal encoder-decoder CNN architecture for semi-supervised bounding box prediction and exploratory data analysis. This architecture is designed to fully model multichannel simulation data, temporal dynamics and unlabelled data within a reconstruction and prediction framework so as to improve the detection of a wide range of extreme weather events. Our architecture can be viewed as a 3D convolutional autoencoder with an additional modified one-pass bounding box regression loss. We demonstrate that our approach is able to leverage temporal information and unlabelled data to improve localization of extreme weather events. Further, we explore the representations learned by our model in order to better understand this important data, and facilitate further work in understanding and mitigating the effects of climate change.
1 INTRODUCTION
Climate change is one of the most important challenges facing humanity in the 21st century and climate simulations are one of the only viable mechanisms for understanding the future impact of various carbon emission scenarios and intervention strategies. Large climate simulations produce massive datasets: a single 30-year run from a 25-km resolution model produces on the order of 10TB of multi-variate data. This scale of data makes post-processing such datasets to make quantitative assessments challenging and as a result, climate analysts and policy makers typically take global and annual averages of temperature or sea-level rise. These quantities are very coarse measurements and while they are perfect for public and media consumption, they ignore spatially (and temporally) resolved extreme weather events such as extra-tropical cyclones and tropical cyclones (hurricanes). Because the general public and policy makers are concerned about the local impacts of climate change, it is critical that we be able to analyze trends in extreme weather events which can have dramatic and tragic impacts on local and national populations and economies.
The task of finding extreme weather events in climate data has some similarities to the task of detecting objects and activities in video streams - a popular application area for deep learning techniques. An important difference is that in the case of climate data the ’video’ has 16 or more ’channels’ of information (such as water vapour, pressure and temperature), while conventional video only has 3 (RGB). In addition, these climate simulations do not share the same statistics as natural images. As a result, we cannot build off of some of the computer vision communities’ efforts, such as pretrained weights from VGG or AlexNet with ImageNet.
Deep neural networks, especially deep convolutional neural networks, have enjoyed breakthrough success in recent recent years, achieving state-of-the-art results on many benchmark datasets (Krizhevsky et al. (2012); He et al. (2015); Szegedy et al. (2015)) and also compelling results on many practical tasks such as disease diagnosis (Hosseini-Asl et al. (2016)), facial recognition (Parkhi et al. (2015)), autonomous driving (Chen et al. (2015)), and many others. Furthermore, deep neural networks have also been very effective in the context of unsupervised and semi-supervised learning; some recent examples include variational autoencoders (Kingma & Welling (2013)), adversarial networks (Goodfellow et al. (2014); Makhzani et al. (2015)), and what-where autoencoders (Zhao et al. (2015)).
In this paper we present a 3D convolutional architecture for the task of semi-supervised climate event detection. Our contribution is the creation of a 3D spatiotemporal autoencoding CNN architecture for bounding box prediction and the exploration of its use for this important problem. Our architecture is capable of semi-supervised event detection, as well as exploratory data analysis through clustering representations learned using this approach. Our framework allows climate researchers to identify important climate events which have not yet been precisely defined, and could provide a basis for mining large climate simulation data archives and potentially near-real-time prediction of extreme weather events.
2 RELATED WORK
2.1 DEEP LEARNING FOR CLIMATE AND WEATHER DATA
The climate science community primarily relies on expert engineered systems and ad-hoc rules for characterizing climate and weather patterns. Prabhat et al. (2012; 2015) have implemented an assortment of heuristics in the TECA MapReduce framework to process large scale climate datasets on high-performance computing (HPC) platforms. Using the output of TECA analysis as ground truth, Liu et al. (2016) demonstrated for the first time that convolutional architectures could be successfully applied to predict the class label for two extreme weather event types. Their work considered the binary classification task on centered, cropped patches from 2D (single-timestep) multi-channel images. Like Liu et al. (2016) we use TECA’s planetary-scale output as ground truth, but we build on this work by: 1) using uncropped images, 2) considering the temporal axis of the data 3) doing multi-class bounding box detection and 4) exploring a semi-supervised approach based on a hybrid predictive and reconstructive model.
Some recent work has applied deep learning methods to weather forecasting. Xingjian et al. (2015) have explored a convolutional LSTM architecture (described in 2.2 for predicting future precipitation on a local scale (i.e. the size of a city) using radar echo data. In contrast, we focus on extreme event detection on planetary-scale data. Our aim is to capture patterns which are very local in time (e.g. a hurricane may be present in half a dozen sequential frames) compared to the scale of our underlying climate data, consisting of global simulations over many years. As such we use full blown 3D CNNs as they make more sense for our detection application compared to LSTMs, whose strength is in capturing long-term dependencies.
2.2 RELATED METHODS AND MODELS
Following the dramatic success of CNNs in static 2D images, a wide variety of CNN architectures have been explored for video, ex. (Karpathy et al., 2014; Yao et al., 2015; Tran et al., 2014). The details of how CNNs are extended to capture the temporal dimension are important and Karpathy et al. (2014) explored different strategies for fusing information from 2D CNN subcomponents. In contrast, Yao et al. (2015) created 3D volumes of statistics from low level image features.
CNNs have been combined with RNNs for modeling video and other sequence data and we briefly review some relevant video models here. The most common approach to modeling sequential images is to feed single-frame representations from a CNN at each timestep to an RNN. This approach has been examined for a number of different types of video (Donahue et al., 2015). Srivastava et al. (2015) have explored an LSTM architecture for the unsupervised learning of video representations using a pretrained CNN representation as input. Another popular model, also used on 1D data, is a convolutional RNN, wherein the hidden-to-hidden transition layer is 1D convolutional (i.e. the state
is convolved over time). Ballas et al. (2016) combine these ideas, applying a convolutional LSTM to frames processed by a (2D) CNN.
The 3D CNNs we use here are based on 3-dimensional convolutional filters, taking the height, width, and time axes into account for each feature map, as opposed to aggregated 2D CNNs. This approach was studied in detail in Tran et al. (2014). 3D convolutional neural networks have been used for various tasks ranging from human activity recognition (Ji et al., 2013), to large-scale YouTube video classification (Karpathy et al., 2014), and video description (Yao et al., 2015). Hosseini-Asl et al. (2016) use a 3D convolutional autoencoder for diagnosing Alzheimer’s disease through MRI - in this case, the 3 dimensions are height, width, and depth. Whitney et al. (2016) use 3D (height, width, depth) filters to predict consecutive frames of a video game for continuation learning. Some recent work has also examined ways to use CNNs to generate animated textures and sounds Xie et al. (2016). This work is similar to our approach in that it uses a 3D convolutional encoder, but they use a stochastic approach whereas we use a deterministic autoencoder with a 3D convolutional decoder for the unsupervised component of our overall objective function.
Stepping back, our approach and model is closely related to Tran et al. (2014) in that we use a 3D deconvolutional CNN approach to construct the autoencoding component of our model. Our approach is related to recent work from Zhang et al. (2016) (and others) in that we also use a hybrid prediction and autoencoding architecture, but our model is for multidimensional 3D data and yields multiple bounding box predictions. Our formulation for the bounding box prediction loss is inspired by the approach in Redmon et al. (2015) and extended in Ren et al. (2015), the single shot multiBox detector formulation used in Liu et al. (2015) and the seminal bounding box work in OverFeat Sermanet et al. (2013).
3 THE MODEL
We use a 3D convolutional encoder-decoder architecture (the code can be found here: https://github.com/eracah/hur-detect/); feature maps are 3D: time, height, and width. The encoder part of this architecture is described in Table 2. The decoder is the equivalent structure in reverse, using tied weights and deconvolutional layers. As we take a semi-supervised approach, the code (bottleneck) layer of the autoencoder is used as the input to the loss layers, which make predictions for bounding box location and size, the class associated with the bounding box, and the confidence (sometimes called ’objectness’) of the bounding box. In training, we input one day’s simulation at a time (8 time steps, 16 variables) and reconstruct all 8 time steps, while only predicting the bounding box for 4 time steps, since only 4 out of 8 time steps are labelled, so a 1:1 ratio of unlabelled data to labelled data. The entire architecture is depicted in Figure 1.
For comparison, to evaluate how useful the time axis is to recognizing extreme weather events, we also run experiments with a 2D (width, height) version of this architecture.
In order to regress bounding boxes, we use some techniques from Redmon et al. (2015), Ren et al. (2015), and Liu et al. (2015). We split the original 768 by 1152 image into a 12x18 grid of 64x64 “anchor” boxes We then make a guess for a box at each grid point by transforming the representation to 216 “scores”. Each score encodes three pieces of information: how much the predicted box differs in size and location from the anchor box, how confident we are that an object of interest is in the predicted box, and the class probability distribution for that object. We compute each component of the score by several 3x3 convolutions applied to the 640 12x18 feature maps from output of the last layer of the encoder (”code layer”). Because each set of pixels in each feature map at a given x, y coordinate can be thought of as a learned representation of the climate data in a 64x64 patch of the input image, we can thinks of the 3x3 convolutions as using the representation of a 192 x 192 neighborhood from the input image as context to determine the box and object centered in the given 64x64 patch. Our approach is similar to Liu et al. (2015) and Sermanet et al. (2013), which use convolutions from small local receptive field filters to regress boxes. This choice is motivated by the fact that extreme weather events occur in relatively small spatiotemporal volumes, with the ’background’ context being highly consistent across event types and between events and non-events. This is in contrast to that of Redmon et al. (2015), which uses a fully connected layer to consider the whole image as context. Their approach is appropriate to the objective of object identification in natural images, where there is often a strong relationship between background and object.
The bounding box regression loss is determined as follows:
Lsup = 1
N (Lbox + Lconf + Lcls), (1)
where N is the number of time steps in the minibatch, and Lbox is defined as: Lbox = α ∑ i 1 obj i R(ui − u ∗ i ) + β ∑ i 1 obj i R(vi − v ∗ i ), (2)
where i ∈ [0, 216) is the index of the anchor box for the ith grid point (12x18 grid, so 216 grid points). where 1obji is 1 if an object is present at the ith grid point and 0 if not, R(z) is the smooth L1 loss (Ren et al. (2015)), ui = (tx, ty)i and u∗i = (t ∗ x, t ∗ y)i, vi = (tw, th)i and v ∗ i = (t ∗ w, t ∗ h)i
where t is the parametrization defined in (Ren et al. (2015)) such that:
tx = (x− xa)/wa, ty = (y − ya)/ha, tw = log(w/wa), th = log(h/ha)
t∗x = (x ∗ − xa)/wa, t∗y = (y∗ − ya)/ha, t∗w = log(w∗/wa), t∗h = log(h∗/ha),
where (xa, ya, wa, ha) is the center coordinates and height and width of the closest anchor box, (x, y, w, h) are the predicted coordinates and (x∗, y∗, w∗, h∗) are the ground truth coordinates.
Lconf is the weighted cross entropy of the object confidence (probability distribution for whether object is present in a grid cell or not):
Lconf = ∑ i 1 obj i [(− log(p(obj)i)] + γ ∗ ∑ i 1 noobj i [− log(p(obj)i] (3)
Lcls is just the cross-entropy between the one-hot encoded class distribution and the softmax predicted class distribution, evaluated only for predicted boxes at the grid points containing a ground truth box:
Lcls = ∑ i 1 obj i ∑ c∈classes −p∗(c) log(p(c)) (4)
This loss is similar in spirit to YOLO (Redmon et al. (2015)), but has a few differences. One, the object confidence and class probability terms in YOLO are squared-differences between ground truth and prediction, so we make the logical change to cross-entropy, which is more appropriate for probabilities and is used in the region proposal network from Faster R-CNN Ren et al. (2015) and the network from Liu et al. (2015) for the object probability term and the class probability term respectively.
In addition, there is a difference in the parametrization for the coordinates and the size of the bounding box.
In YOLO, the parametrizations for x and y are equivalent to Faster R-CNN’s tx and ty for an anchor box the same size as the patch it represents (64x64). For the w and h parametrizations in YOLO, they would only be equivalent to Faster-RCNN’s th and tw for a 64x64 anchor box if the anchor box had a height and width equal to the size of the whole image and if there were no log transform in the faster-RCNN’s parametrization. We found in practice that both these differences in YOLO did not work well in practice. For one, without the log term and using a RELU transform as the nonlinearity with a standard weight initialization scheme sampled from a distribution centered around 0, most outputs (more than half) for the parametrization were zero or close to zero as expected (at least at the beginning of training); when the parametrization was converted to raw width and height, this resulted in 0 height and width boxes, so technically the default boxes were of 0 width and height. In practice, it took many epochs for the network to learn to resize 0 area boxes to the correct size due to all the other terms of the loss that were being optimized. Adding the log term, would in effect make the ”default” box (an output of 0 ) equal to a box of height and width ha and wa, which in YOLO’s case was the height and width of the entire image. This once again proved to be a problem as it took a long time to resize the boxes to the size of the much smaller ground truth boxes. Making ha and wa equal to 64x64 made training a lot more efficient, as the optimization was focused more on picking which box contained an object and not as much on what size the box should be. The last difference was YOLO used squared difference between predicted and ground truth for the coordinate parametrizations, as opposed to smooth L1. We used smooth L1 due its lower sensitivity to outlier predictions Ren et al. (2015).
The loss is a weighted combination of reconstruction error and bounding box regression loss:
L = Lsup + λLrec, (5)
where Lunsup is the classic squared difference between input and reconstruction:
Lrec = ||X −X∗||22, (6)
4 EXPERIMENTS AND DISCUSSION
4.1 CLIMATE DATA
The climate science community utilizes three flavors of global datasets: observational products (satellite, gridded weather station); reanalysis products (obtained by assimilating disparate observational products into a climate model) and simulation products. In this study, we analyze output from the third category: free-running climate simulations with prescribed initial and boundary conditions. In particular, we consider a 27-year run of the CAM5 (Community Atmospheric Model v5); configured at 25-km spatial resolution (Wehner et al. (2015)), with ground-truth labelling for four extreme weather events: Tropical Depressions (TD) Tropical Cyclones (TC), Extra-Tropical Cyclones (ETC) and Atmospheric Rivers (AR).
At 25-km resolution, each snapshot of the global atmospheric state in the CAM5 model output corresponds to a 768x1152 image. Each image is comprised of 16 variables (including surface temperature, surface pressure, precipitation, zonal wind, meridional wind, humidity, cloud fraction, water vapor, etc.) stored in double-precision floating point representation. While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study. Model output is stored every 3 hours.
The training data (which can be downloaded here: http://portal.nersc.gov/project/dasrepo/climate/) consists of 365 days of simulations from 1979 with 8 frames a day (every three hours), but labels are only every 6 hours, so one day for supervised is 4 frames and 8 for semi-supervised. The test set consisted of 365 days from 1984. Table 3 shows the breakdown of the dataset splits for each class.
TECA, the ground truth labelling framework, tries to implement the state of the art heuristics for determining ’ground truth’ data for the four types extreme weather events. However, it is entirely possible there are errors in the labeling: for instance, there is little agreement in the climate community on a standard heuristic for capturing Extra-Tropical Cyclones (Neu et al. (2013)); Atmospheric Rivers have been extensively studied in the northern hemisphere (Lavers et al. (2012); Dettinger et al. (2011)), but not in the southern hemisphere; and spatial extents of such events not universally agreed upon. As such, there is potential for many false negatives, resulting in partially annotated images. This, in addition to lower representation for classes, like AR’s and Tropical Depressions, is part of our motivation in exploring semi-supervised methods to better understand the features underlying extreme weather events. Moreover, although there is more than enough labelled data for our training set sizes, not all flavors of climate datasets can be labelled by TECA due it’s ”hard-coded” set up for a specific resolution simulation with specific parameters, so demonstrating the success of semi-supervised learning for climate pattern detection is important. Lastly, TECA only labels four types of events, so detection of other climate events would require hand-labelling, which would likely result in only 10’s of labels, necessitating semi-supervised approaches.
4.2 FRAMEWISE RECONSTRUCTION
As a simple experiment, we first train a 2D convolutional autoencoder on the data, treating each timestep as an individual training example (everything else about the model is as described in Section 3), in order to visually assess reconstructions and ensure reasonable accuracy of detection. Figure 2 shows the original and reconstructed feature maps for the 16 climate variables of one image in the training set. Though we are showing reconstructions on the training set, the reconstruction loss on the validation set was also similar.
As the reconstruction visualizations suggest, the 2D convolutional autoencoder architecture does a good job of encoding spatial information from climate images. In the upcoming subsections, we utilize the full 3D architecture for the detection of climate events.
4.3 COMPARING FULLY SUPERVISED AND SEMI-SUPERVISED DETECTION
We run two experiments of 3D supervised and semi-supervised bounding box prediction, where values for α, β, γ were selected with experimentation and some inspiration from (Redmon et al. (2015)) to be 5, 7 and 0.5 respectively.
A lower value for γ also can be interpreted as way to deal with ground truth false negatives, as we want more emphasis on pushing up the confidence of true positive examples in order to learn the features of the four events, as opposed to pushing down the confidence of false negative examples, which may interfere with generalization. We used the Adam optimizer (Kingma & Ba (2014)) with a base learning rate of 0.0001. We used a weight decay coefficient of 0.0005.
In Table 4 and Table 5, we present 2D and 3D supervised and semi-supervised results for various settings of λ. Because the 3D model has inherently higher capacity in terms of number of parameters than the 2D model, we also show some experiments with higher capacity 2D models by doubling the number of filters that are in each layer in Table 2. Average Precision (AP) was calculated in the manner of ImageNet (Russakovsky et al. (2015)), where we integrated the precision-recall curve for each class. Results are shown for two modes of evaluation: denoting a true postive as a bounding box prediction with IOU with the ground truth box of 0.1 and 0.5. Mean average precision was just the average of the AP over the four classes. Furthermore, in Figure 3 we provide bounding box predictions shown on 2 consecutive (6 hours in between) simulation frames comparing the 3D supervised vs 3D semi-supervised model predictions.
As we can see from Table 4, Table 5 and Figure 3, 3D and semi-supervised approaches help to some degree for rough localization of the weather events (IOU=0.1). Namely, semi-supervised approaches help the performance of the 3D model for IOU=0.1. In contrast, semi-supervised approach 2D does not have the same effect for IOU=0.1. It is important to note that a more thorough hyperparameter search for λ and other parameters may yield better results for semi-supervised. Also of note is that the 3D models perform significantly better than their 2D counterparts for ETC and TC (hurricane) classes. This potentially means that the time evolution of these weather events is an important criteria for discriminating them. In addition, the semi-supervised model significantly improves the ETC and TC performance, which suggests unsupervised shaping of the spatio-temporal representation is important for these events. It would be interesting to further explore and confirm the spatio-temporal representation learned for these weather events in the 3D models.
In contrast, Table 4 shows that for a more stringent, but standard detection metric (IOU=0.5), the semi-supervised models do much worse than their semi-supervised counterparts. This could potentially be improved by using a dataset from a different year for obtaining unlabelled data, so as not to
have as much correlation with the labelled data. Moreover, the 3D supervised models perform worse than the 2D supervised models and 3D semi-supervised perform comparably to the corresponding 2D semi-supervised models. The potential reason for this can be explained to a certain degree with Figure 3. As shown in the figure, the 3D models do a good job roughly localizing the various events. However, as mentioned in Section 3, the network has a hard time adjusting the size of the boxes. As such, in this figure we see mostly boxes of size 64x64. For example, for TD’s, which are usually much smaller than 64x64 and for AR’s, which are always much bigger than 64x64, a 64x64 box roughly centered on the event is sufficient to count as a true positive with IOU=0.1, but not for the more stringent IOU=0.5. This lead to a large dropoff in performance for AR’s and TD’s and a sizable dropoff in the variable sized TC’s as well as shown in Table 5. While the model has the capacity to adjust the sizes, it can take a while to converge on more different sizes; with longer training time we expect it (and have seen it for 2D) to be able to detect other size boxes, but due to the size of the model we did not have time for longer training. Hence, we believe the 2D model was able to do more size and location adjustments in its training and as a result, it’s AP dropoff for the 4 classes was not as large. However, the 3D semi-supervised model’s ability to better roughly localize events (IOU=0.1) is still useful for the application and shows promise toward future use in an improved system. Future work would involve harnessing anchor boxes of many shapes and sizes, like (Ren et al. (2015), Liu et al. (2015), which we believe would greatly increase the detection accuracy for IOU=0.5.
In Figure 4 we provide t-SNE visualisation of the first 7 days in the training sets for both 3D supervised (left) and semi-supervised (right) experiments. It is possible to make out some grouping of various spiral patterns in the latent space for the TC’s and ETC’s. Und r review as a confer nce paper at ICLR 2017
8
5 CONCLUSIONS AND FUTURE WORK
We have explored semi-supervised methods for object detection and bounding box prediction using 3D CNNs. These architectures and approaches are motivated by finding extreme weather patterns;
a meaningful and important problem for society. Thus far, the climate science community has used hand-engineered criteria to characterize patterns. Our results indicate that there is much promise in considered deep learning based approaches. In addition, this work represents our first attempt at applying semi-supervised learning and exploratory data analysis (i.e. clustering in a learned representation) to climate data. In the future, we think it would be interesting to delve deeper into interpreting and visualizing what the network has learned. This would not only help open the ”black box” of deep learning for skeptics in the climate community, but also potentially help resolve ambiguities in the definition of extreme weather events in addition to highlighting unknown underlying factors in extreme weather events. On a similar note, another goal would be to be able to segment every pixel/voxel in the multi-variate spatio-temporal volume as belonging to a particular weather pattern. Once we are able to decompose the climate system along various patterns, we will be able to conduct sophisticated conditional analysis: how much of global precipitation can be attributed to various patterns (cyclones vs. atmospheric rivers vs. fronts). Moreover, we think it would be interesting to apply the models learned from this CAM5 model output to satellite, reanalysis and other simulation model output. However, these datasets have different spatial resolution, variables, and missing values (in the case of satellite observations). Lastly, applying this extreme weather detection framework to simulations of years in the future would be a very informative exercise for the climate community and the world.
ACKNOWLEDGMENTS
This research used resources of the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Code relies on open-source deep learning frameworks Theano (Bergstra et al.; Team et al., 2016) and Lasagne (Team, 2016), whose developers we gratefully acknowledge. We would also like to thank Yunjie Liu and Michael Wehner for providing access to the climate datasets; Alex Lamb and Thorsten Kurth for helpful discussions; and Samsung for their support of this research.

[EDIT: The thoughtful author responses addressed my major concerns. The github links for data and code will be really helpful for reproducing results (I haven't looked carefully, but this is great). The revision addressed many issues, including the additional results. As such I am upgrading my rating from a 5 to a 6 and recommend acceptance of the paper.]
The paper proposes to apply deep nets to perform detection and localization of extreme weather events in simulated weather data. The problem is related to object detection in computer vision in that the input is a 2D “image” (multichannel spatial weather data) or 3D “video” (temporal version of the data) and the output is a bounding box (spatial-temporal localization of a weather event) and class label (weather event type). It differs from standard object detection in that the input has multiple heterogenous channels and labeled data is scarce.
A simple but quite reasonable deep net is proposed for the task based on similar approaches in computer vision. While proposal based systems are most popular in vision currently (in particular Faster-RCNN) the proposed approach is simple and a fine starting point. There is little innovation on the part of the detection system, but as noted, it is a valid application of ideas from computer vision to the task at hand.
The authors propose both a supervised approach (only ground truth bounding box location/label is used) and a semi-supervised approach that additionally incorporates the reconstruction loss as a regularization. In all cases the losses are fairly standard and again, reasonable. The only confusing bit is that the “semi-supervised” loss actually has all the labels used for the “supervised” loss and additionally incorporates the reconstruction loss. Hence, the “semi-supervised” loss is actually stronger, which makes the terminology a bit confusing.
The paper is easy to follow, but notation is sloppy. For example, above equation 5 it states that “the loss is a weighted combination of reconstruction error and bounding box regression loss”; actually it’s a combination of the supervised and unsupervised loss (Lsup and Lunsup), and Lrec is not defined (although I assume Lrec=Lunsup). The paper is fairly non-technical, but nevertheless these minor issues should be fixed. (E.g., see also reference to “figure 4 and 4”).
The biggest concern w the paper though is experimental results. Only a single figure and table of results are shown (figure 4 and table 4). The metrics are not defined (what is mean average recall?). Only 2D versus 3D version of the model are shown, and supervised and semi-supervised. Moreover, numbers seem a bit all over the place, without consistent patterns (e.g., why is 2D supervised better than the seemingly much strong 3D semi-supervised?). One of the things that is unclear is how many events are actually in the training/testing data, and more importantly, how good are these results in absolute terms? Regardless, the experiments are fairly sparse and ablation studies and more discussion lacking. It is also unclear if future researchers will be able to reproduce the experimental setting (a commitment to open-source the data or a way to reproduce the experiments would be critical for future authors).
Minor nit: the authors use both a classification loss and an “objectness” loss. I’ve never seen both used together like this (normally objectness is used in two-stage object proposal systems where in the first stage class-agnostic proposals are given and in the second stage these are cropped and a class-specific classifier is applied). I strongly suspect removing the objectness loss would not impact results since the classification loss should provide strictly stronger supervisory signal. Regardless, this is a fairly non-standard choice and should be justified (experimentally).
Overall this is a borderline paper. I do believe that it is valuable to apply computer vision techniques to a domain that I’ve see little work on in our community. That being said, I have no expertise on this type of data -- it’s possible this deep learning techniques are now routinely used in the climate science literature (I suspect not, though). Overall, there is little novelty on the algorithmic side in this paper (the equations in section 3 are commonly used in the cv literature). The use of reconstruction loss to improve results in the data-sparse setting is interesting, but the experimental results are inconclusive. The experimental validation is generally insufficient. Reproducibility for future research is difficult unless the data is open-sourced. Overall, I think this paper is a good start, and with improved experiments and more careful writing I think could eventually make for a decent paper.


STOCHASTIC NEURAL NETWORKS FOR HIERARCHICAL REINFORCEMENT LEARNING
Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments1 show2 that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.
Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments1 show2 that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.
1 INTRODUCTION
In recent years, deep reinforcement learning has achieved many impressive results, including playing Atari games from raw pixel inputs (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015), mastering the game of Go (Silver et al., 2016), and acquiring advanced manipulation and locomotion skills from raw sensory inputs (Schulman et al., 2015; Lillicrap et al., 2015; Watter et al., 2015; Schulman et al., 2016; Heess et al., 2015; Levine et al., 2016). Despite these success stories, these deep RL algorithms typically employ naive exploration strategies such as -greedy or uniform Gaussian exploration noise, which have been shown to perform poorly in tasks with sparse rewards (Duan et al., 2016; Houthooft et al., 2016; Bellemare et al., 2016). Tasks with sparse rewards are common in realistic scenarios. For example, in navigation tasks, the agent is not rewarded until it finds the target. The challenge is further complicated by long horizons, where naive exploration strategies can lead to exponentially large sample complexity (Osband et al., 2014).
To tackle these challenges, two main strategies have been pursued: The first strategy is to design a hierarchy over the actions (Parr & Russell, 1998; Sutton et al., 1999; Dietterich, 2000). By composing low-level actions into high-level primitives, the search space can be reduced exponentially. However, these approaches require domain-specific knowledge and careful hand-engineering. The second strategy uses intrinsic rewards to guide exploration (Schmidhuber, 1991; 2010; Houthooft et al., 2016; Bellemare et al., 2016). The computation of these intrinsic rewards does not require domain-specific knowledge. However, when facing a collection of tasks, these methods do not provide a direct answer about how knowledge about solving one task may transfer to other tasks, and by solving each of the tasks from scratch, the overall sample complexity may still be high.
1Code available at: https://github.com/florensacc/snn4hrl 2Videos available at: http://bit.ly/snn4hrl-videos
In this paper, we propose a general framework for training policies for a collection of tasks with sparse rewards. Our framework first learns a span of skills in a pre-training environment, where it is employed with nothing but a proxy reward signal, whose design only requires very minimal domain knowledge about the downstream tasks. This proxy reward can be understood as a form of intrinsic motivation that encourages the agent to explore its own capabilities, without any goal information or sensor readings specific to each downstream task. The set of skills can be used later in a wide collection of different tasks, by training a separate high-level policy for each task on top of the skills, thus reducing sample complexity uniformly. To learn the span of skills, we propose to use Stochastic Neural Networks (SNNs) (Neal, 1990; 1992; Tang & Salakhutdinov, 2013), a class of neural networks with stochastic units in the computation graph. This class of architectures can easily represent multi-modal policies, while achieving weight sharing among the different modes. We parametrize the stochasticity in the network by feeding latent variables with simple distributions as extra input to the policy. In the experiments, we observed that direct application of SNNs does not always guarantee that a wide range of skills will be learned. Hence, we propose an information-theoretic regularizer based on Mutual Information (MI) in the pre-training phase to encourage diversity of behaviors of the SNN policy. Our experiments find that our hierarchical policy-learning framework can learn a wide range of skills, which are clearly interpretable as the latent code is varied. Furthermore, we show that training high-level policies on top of the learned skills can results in strong performance on a set of challenging tasks with long horizons and sparse rewards.
2 RELATED WORK
One of the main appealing aspects of hierarchical reinforcement learning (HRL) is to use skills to reduce the search complexity of the problem (Parr & Russell, 1998; Sutton et al., 1999; Dietterich, 2000). However, specifying a good hierarchy by hand requires domain-specific knowledge and careful engineering, hence motivating the need for learning skills automatically. Prior work on automatic learning of skills has largely focused on learning skills in discrete domains (Chentanez et al., 2004; Vigorito & Barto, 2010). A popular approach there is to use statistics about state transitions to identify bottleneck states (Stolle & Precup, 2002; Mannor et al., 2004; Şimşek et al., 2005). It is however not clear how these techniques can be applied to settings with high-dimensional continuous spaces. More recently, Mnih et al. (2016) propose a DRAW-like (Gregor et al., 2015) recurrent neural network architecture that can learn temporally extended macro actions. However, the learning needs to be guided by external rewards as supervisory signals, and hence the algorithm cannot be straightforwardly applied in sparse reward settings.
There has also been work on learning skills in tasks with continuous actions (Schaal et al., 2005; Konidaris et al., 2011; Daniel et al., 2013; Ranchod et al., 2015). These methods extract useful skills from successful trajectories for the same (or closely related) tasks, and hence require first solving a comparably challenging task or demonstrations. Guided Policy Search (GPS) Levine et al. (2016) leverages access to training in a simpler setting. GPS trains with iLQG Todorov & Li (2005) in state space, and in parallel trains a neural net that only receives raw sensory inputs that has to agree with the iLQG controller. The neural net policy is then able to generalize to new situations.
Another line of work on skill discovery particularly relevant to our approach is the HiREPS algorithm by Daniel et al. (2012) where, instead of having a mutual information bonus like in our proposed approach, they introduce a constraint on an equivalent metric. The solution approach is nevertheless very different as they cannot use policy gradients. Furthermore, although they do achieve multimodality like us, they only tried the episodic case where a single option is active during the rollout. Hence the hierarchical use of the learned skills is less clear. Similarly, the Option-critic architecture (Bacon & Precup, 2016) can learn interpretable skills, but whether they can be reuse across complex tasks is still an open question. Recently, Heess et al. (2016) have independently proposed to learn a range of skills in a pre-training environment that will be useful for the downstream tasks, which is similar to our framework. However, their pre-training setup requires a set of goals to be specified. In comparison, we use a signle proxy reward as the only signal to the agent during the pre-training phase, the construction of which only requires minimal domain knowledge or instrumentation of the environment.
3 PRELIMINARIES
We define a discrete-time finite-horizon discounted Markov decision process (MDP) by a tupleM = (S,A,P, r, ρ0, γ, T ), in which S is a state set, A an action set, P : S × A × S → R+ a transition probability distribution, r : S ×A → [−Rmax, Rmax] a bounded reward function, ρ0 : S → R+ an initial state distribution, γ ∈ [0, 1] a discount factor, and T the horizon. When necessary, we attach a suffix to the symbols to resolve ambiguity, such as SM . In policy search methods, we typically optimize a stochastic policy πθ : S × A → R+ parametrized by θ. The objective is to maximize its expected discounted return, η(πθ) = Eτ [ ∑T t=0 γ
tr(st, at)], where τ = (s0, a0, . . .) denotes the whole trajectory, s0 ∼ ρ0(s0), at ∼ πθ(at|st), and st+1 ∼ P(st+1|st, at).
4 PROBLEM STATEMENT
Our main interest is in solving a collection of downstream tasks, specified via a collection of MDPs M. If these tasks do not share any common structure, we cannot expect to acquire a set of skills that will speed up learning for all of them. On the other hand, we want the structural assumptions to be minimal, to make our problem statement more generally applicable.
For each MDP M ∈ M, we assume that the state space SM can be factored into two components, Sagent, and SMrest, which only weakly interact with each other. The Sagent should be the same for all MDPs inM. We also assume that all the MDPs share the same action space. Intuitively, we consider a robot who faces a collection of tasks, where the dynamics of the robot are shared across tasks, and are covered by Sagent, but there may be other components in a task-specific state space, which will be denoted by Srest. For instance, in a grasping task M , SMrest may include the positions of objects at interest or the new sensory input associated with the task. This specific structural assumption has been studied in the past as sharing the same agent-space (Konidaris & Barto, 2007).
Given a collection of tasks satisfying the structural assumption, our objective for designing the algorithm is to minimize the total sample complexity required to solve these tasks. This has been more commonly studied in the past in sequential settings, where the agent is exposed to a sequence of tasks, and should learn to make use of experience gathered from solving earlier tasks to help solve later tasks (Taylor & Stone, 2009; Wilson et al., 2007; Lazaric & Ghavamzadeh, 2010; Devin et al., 2017). However, this requires that the earlier tasks are relatively easy to solve (e.g., through good reward shaping or simply being easier) and is not directly applicable when all the tasks have sparse rewards, which is a much more challenging setting. In the next section, we describe our formulation, which takes advantage of a pre-training task that can be constructed with minimal domain knowledge, and can be applied to the more challenging scenario.
5 METHODOLOGY
In this section, we describe our formulation to solve a collection of tasks, exploiting the structural assumption articulated above. In Sec. 5.1, we describe the pre-training environment, where we use proxy rewards to learn a useful span of skills. In Sec. 5.2, we motivate the usage of Stochastic Neural Networks (SNNs) and discuss the architectural design choices made to tailor them to skill learning for RL. In Sec. 5.3, we describe an information-theoretic regularizer that further improves the span of skills learned by SNNs. In Sec. 5.4, we describe the architecture of high-level policies over the learned skills, and the training procedure for the downstream tasks with sparse rewards. Finally, in Sec. 5.5, we describe the policy optimization for both phases of training.
5.1 CONSTRUCTING THE PRE-TRAINING ENVIRONMENT
Given a collection of tasks, we would like to construct a pre-training environment, where the agent can learn a span of skills, useful for enhancing exploration in downstream tasks. We achieve this by letting the agent freely interact with the environment in a minimal setup. For a mobile robot, this can be a spacious environment where the robot can first learn the necessary locomotion skills; for a manipulator arm which will be used for object manipulation tasks, this can be an environment with many objects that the robot can interact with.
The skills learned in this environment will depend on the reward given to the agent. Rather than setting different rewards in the pre-training environment corresponding to the desired skills, which requires precise specification about what each skill should entail, we use a generic proxy reward as the only reward signal to guide skill learning. The design of the proxy reward should encourage the existence of locally optimal solutions, which will correspond to different skills the agent should learn. In other words, it encodes the prior knowledge about what high level behaviors might be useful in the downstream tasks, rewarding all of them roughly equally. For a mobile robot, this reward can be as simple as proportional to the magnitude of the speed of the robot, without constraining the direction of movement. For a manipulator arm, this reward can be the successful grasping of any object.
Every time we train a usual uni-modal gaussian policy in such environment, it should converge towards a potentially different skill. As seen in Sec. 7.1, applying this approach to mobile robots gives a different direction (and type) of locomotion every time. But it is an inefficient procedure: training each policy from scratch is not sample efficient (sample complexity grows linearly with the number of skills to be learned) and nothing encourages the individual skills to be distinct. The first issue will be addressed in the next subsection by using Stochastic Neural Networks as policies and the second issue will be addressed in Sec. 5.3 by adding an information-theoretic regularizer.
5.2 STOCHASTIC NEURAL NETWORKS FOR SKILL LEARNING
To learn several skills at the same time, we propose to use Stochastic Neural Networks (SNNs), a general class of neural networks with stochastic units in the computation graph. There has been a large body of prior work on special classes of SNNs, such as Restricted Boltzmann Machines (RBMs) (Smolensky, 1986; Hinton, 2002), Deep Belief Networks (DBNs) (Hinton et al., 2006), and Sigmoid Belief Networks (SBNs) (Neal, 1990; Tang & Salakhutdinov, 2013). They have rich representation power and can in fact approximate any well-behaved probability distributions (Le Roux & Bengio, 2008; Cho et al., 2013). Policies modeled via SNNs can hence represent complex action distributions, especially multi-modal distributions.
For our purpose, we use a simple class of SNNs, where latent variables with fixed distributions are integrated with the inputs to the neural network (here, the observations from the environment) to form a joint embedding, which is then fed to a standard feed-forward neural network (FNN) with deterministic units, that computes distribution parameters for a uni-modal distribution (e.g. the mean and variance parameters of a multivariate Gaussian). We use simple categorical distributions with uniform weights for the latent variables, where the number of classes, K, is an hyperparameter that upper bounds the number of skills that we would like to learn.
The simplest joint embedding, as shown in Figure 1(a), is to concatenate the observations and the latent variables directly.3 However, this limits the expressiveness power of integration between the observation and latent variable. Richer forms of integrations, such as multiplicative integrations and bilinear pooling, have been shown to have greater representation power and improve the optimization landscape, achieving better results when complex interactions are needed (Fukui et al., 2016; Wu et al., 2016). Inspired by this work, we study using a simple bilinear integration, by forming the outer product between the observation and the latent variable (Fig. 1(b)). Note that the concatenation
3In scenarios where the observation is very high-dimensional such as images, we can form a lowdimensional embedding of the observation alone, say using a neural network, before jointly embedding it with the latent variable.
integration effectively corresponds to changing the bias term of the first hidden layer depending on the latent code z sampled, and the bilinear integration to changing all the first hidden layer weights. As shown in the experiments, the choice of integration greatly affects the quality of the span of skills that is learned. Our bilinear integration already yields a large span of skills, hence no other type of SNNs is studied in this work.
To obtain temporally extended and consistent behaviors associated with each latent code, in the pretraining environment we sample a latent code at the beginning of every rollout and keep it constant throughout the entire rollout. After training, each of the latent codes in the categorical distribution will correspond to a different, interpretable skill, which can then be used for the downstream tasks.
Compared to training separate policies, training a single SNN allows for flexible weight-sharing schemes among different policies. It also takes a comparable amount of samples to train an SNN than a regular uni-modal gaussian policy, so the sample efficiency of training K skills is effectively K times better. To further encourage the diversity of skills learned by the SNN we introduce an information theoretic regularizer, as detailed in the next section.
5.3 INFORMATION-THEORETIC REGULARIZATION
Although SNNs have sufficient expressiveness to represent multi-modal policies, there is nothing in the optimization that prevents them from collapsing into a single mode. We have not observed this worst case scenario in our experiments, but sometimes different latent codes correspond to very similar skills. It is desirable to have direct control over the diversity of skills that will be learned. To achieve this, we introduce an information-theoretic regularizer, inspired by recent success of similar objectives in encouraging interpretable representation learning in InfoGAN Chen et al. (2016).
Concretely, we add an additional reward bonus, proportional to the mutual information (MI) between the latent variable and the current state. We only measure the MI with respect to a relevant subset of the state. For a mobile robot, we choose this to be the c = (x, y) coordinates of its center of mass (CoM). Formally, let C be a random variable denoting the current CoM coordinate of the agent, and let Z be the latent variable. Then the MI can be expressed as I(Z;C) = H(Z) −H(Z|C), where H denotes the entropy function. In our case, H(Z) is constant since the probability distribution of the latent variable is fixed to uniform during this stage of training. Hence, maximizing the MI is equivalent to minimizing the conditional entropy H(Z|C). As entropy is a measure of uncertainty, another interpretation of this bonus is that, given where the robot is, it should be easy to infer which skill the robot is currently performing. To penalize H(Z|C) = −Ez,c log p(Z = z|C = c), we modify the reward received at every step as specified in Eq. (1), where p̂(Z = zn|cnt ) is an estimate of the posterior probability of the latent code zn sampled on rollout n, given the coordinates cnt at time t of that rollout.
Rnt ← Rnt + αH log p̂(Z = zn|cnt ) (1)
To estimate the posterior p̂(Z = zn|cnt ), we apply the following discretization: we partition the (x, y) coordinate space into cells and map the continuous-valued CoM coordinates into the cell containing it. Overloading notation, now ct is a discrete variable indicating the cell where the CoM is at time t. This way, calculation of the empirical posterior only requires maintaining visitation counts mc(z) of how many times each cell c is visited when latent code z is sampled. Given that we use a batch policy optimization method, we use all trajectories of the current batch to compute all the mc(z) and estimate p̂(Z = zn|cnt ), as shown in Eq. (2). If c is more high dimensional, the posterior p̂(Z = zn|cnt ) can also be estimated by fitting a MLP regressor by Maximum Likelihood.
p̂ ( Z = z ∣∣(x, y)) ≈ p̂(Z = z|c) = mc(z)∑ z′ mc(z ′) (2)
5.4 LEARNING HIGH-LEVEL POLICIES
Given a span of K skills learned during the pre-training task, we now describe how to use them as basic building blocks for solving tasks where only sparse reward signals are provided. Instead of learning from scratch the low-level controls, we leverage the provided skills by freezing them and training a high-level policy (Manager Neural Network) that operates by selecting a skill and committing to it for a fixed amount of steps T . The imposed temporal consistency and the quality
of the skills (in our case given by optimizing the proxy reward in the pre-train environment) yield an enhanced exploration allowing to solve the downstream tasks with sparse rewards.
For any given task M ∈ M we train a new Manager NN on top of the common skills. Given the factored representation of the state space SM as Sagent and SMrest, the high-level policy receives the full state as input, and outputs the parametrization of a categorical distribution from which we sample a discrete action z out of K possible choices, corresponding to the K available skills. If those skills are independently trained uni-modal policies, z dictates the policy to use during the following T time-steps. If the skills are encapsulated in a SNN, z is used in place of the latent variable. The architecture is depicted in Fig. 2.
The weights of the low level and high level neural networks could also be jointly optimized to adapt the skills to the task at hand. This end-to-end training of a policy with discrete latent variables in the Stochastic Computation Graph could be done using straight-through estimators like the one proposed by Jang et al. (2017) or Maddison et al. (2017). Nevertheless,
we show in our experiments that frozen low-level policies are already sufficient to achieve good performance in the studied downstream tasks, so these directions are left as future research.
5.5 POLICY OPTIMIZATION
For both the pre-training phase and the training of the high-level policies, we use Trust Region Policy Optimization (TRPO) as the policy optimization algorithm (Schulman et al., 2015). We choose TRPO due to its excellent empirical performance and because it does not require excessive hyperparameter tuning. The training on downstream tasks does not require any modifications, except that the action space is now the set of skills that can be used. For the pre-training phase, due to the presence of categorical latent variables, the marginal distribution of π(a|s) is now a mixture of Gaussians instead of a simple Gaussian, and it may even become intractable to compute if we use more complex latent variables. To avoid this issue, we consider the latent code as part of the observation. Given that π(a|s, z) is still a Gaussian, TRPO can be applied without any modification.
Algorithm 1: Skill training for SNNs with MI bonus Initialize: Policy πθ; Latent dimension K; while Not trained do
for n← 1 to N do Sample zn ∼ Cat ( 1 K ) ;
Collect rollout with zn fixed; end Compute p̂(Z = z|c) = mc(z)∑
z′ mc(z ′) ;
Modify Rnt ← Rnt + αH log p̂(Z = zn|cnt ); Apply TRPO considering z part of the observation;
end
6 EXPERIMENTS
We have applied our framework to the two hierarchical tasks described in the benchmark by Duan et al. (2016): Locomotion + Maze and Locomotion + Food Collection (Gather). The observation space of these tasks naturally decompose into Sagent being the robot and SMrest the task-specific attributes like walls, goals, and sensor readings. Here we report the results using the Swimmer robot, also described in the benchmark paper. In fact, the swimmer locomotion task described therein corresponds exactly to our pretrain task, as we also solely reward speed in a plain environment. We report results with more complex robots in Appendix C-D.
To increase the variety of downstream tasks, we have constructed four different mazes. Maze 0 is the same as the one described in the benchmark (Duan et al., 2016) and Maze 1 is its reflection, where the robot has to go backwards-right-right instead of forward-left-left. These correspond to Figs. 3(a)-3(b), where the robot is shown in its starting position and has to reach the other end of the U turn. Mazes 2 and 3 are different instantiations of the environment shown in Fig. 3(c), where the goal has been placed in the North-East or in the South-West corner respectively. A reward of 1 is only granted when the robot reaches the goal position. In the Gather task depicted in Fig. 3(d), the robot gets a reward of 1 for collecting green balls and a reward of -1 for the red ones, all of which are positioned randomly at the beginning of each episode. Apart from the robot joints positions and velocities, in these tasks the agent also receives LIDAR-like sensor readings of the distance to walls, goals or balls that are within a certain range.
In the benchmark of continuous control problems (Duan et al., 2016) it was shown that algorithms that employ naive exploration strategies could not solve them. More advanced intrinsically motivated explorations (Houthooft et al., 2016) do achieve some progress, and we report our stronger results with the exact same setting in Appendix B. Our hyperparameters for the neural network architectures and algorithms are detailed in the Appendix A and the full code is available4.
7 RESULTS
We evaluate every step of the skill learning process, showing the relevance of the different pieces of our architecture and how they impact the exploration achieved when using them in a hierarchical fashion. Then we report the results5 on the sparse environments described above. We seek to answer the following questions:
• Can the multimodality of SNNs and the MI bonus consistently yield a large span of skills? • Can the pre-training experience improve the exploration in downstream environments? • Does the enhanced exploration help to efficiently solve sparse complex tasks?
7.1 SKILL LEARNING IN PRETRAIN
To evaluate the diversity of the learned skills we use “visitation plots”, showing the (x, y) position of the robot’s Center of Mass (CoM) during 100 rollouts of 500 time-steps each. At the beginning of every rollout, we reset the robot to the origin, always in the same orientation (as done during training). In Fig. 4(a) we show the visitation plot of six different feed-forward policies, each trained from scratch in our pre-training environment. For better graphical interpretation and comparison with the next plots of the SNN policies, Fig. 4(b) superposes a batch of 50 rollout for each of the 6 policies, each with a different color. Given the morphology of the swimmer, it has a natural preference for forward and backward motion. Therefore, when no extra incentive is added, the visitation concentrates heavily on the direction it is always initialized with. Note nevertheless that the proxy reward is general enough so that each independently trained policy yields a different way of advancing, hence granting a potentially useful skills to solve downstream tasks when embedded in our described Multi-policy hierarchical architecture.
Next we show that, using SNNs, we can learn a similar or even larger span of skills without training several independent policies. The number of samples used to train a SNN is the same as a single
4Code available at: https://github.com/florensacc/snn4hrl 5Videos available at: http://bit.ly/snn4hrl-videos
feed-forward policy from Fig. 4(a), therefore this method of learning skills effectively reduces the sample complexity by a factor equal to the numbers of skills being learned. With an adequate architecture and MI bonus, we show that the span of skills generated is also richer. In the two rows of Figs. 4(c)-4(d) we present the visitation plots of SNNs policies obtained for different design choices. Now the colors indicate the latent code that was sampled at the beginning of the rollout. We observe that each latent code generates a particular, interpretable behavior. Given that the initialization orientation is always the same, the different skills are truly distinct ways of moving: forward, backwards, or sideways. In the following we analyze the impact on the span of skills of the integration of latent variables (concatenation in first row and bilinear in the second) and the MI bonus (increasing coeficient αH towards the right). Simple concatenation of latent variables with the observations rarely yields distinctive behaviors for each latent code sampled. On the other hand, we have observed that 80% of the trained SNNs with bilinear integration acquire at least forward and backward motion associated with different latent codes. This can be further improved to 100% by increasing αH and we also observe that the MI bonus yields less overlapped skills and new advancing/turning skills, independently of the integration of the latent variables.
7.2 HIERARCHICAL USE OF SKILLS
The hierarchical architectures we propose have a direct impact on the areas covered by random exploration. We will illustrate it with plots showing the visitation of the (x − y) position of the Center of Mass (CoM) during single rollouts of one million steps, each generated with a different architecture.
On one hand, we show in Fig. 5(a) the exploration obtained with actions drawn from a Gaussian with µ = 0 and Σ = I , similar to what would happen in the first iteration of training a Multi-Layer Perceptron with normalized random initialization of the weights. This noise is relatively large as the swimmer robot has actions clipped to [−1, 1]. Still, it does not yield good exploration as all point reached by the robot during the one million steps rollout lay within the [−2, 2] × [−2, 2] box around the initial position. This exploration is not enough to reach the first reward in most of the downstream sparse reward environments and will never learn, as already reported by Duan et al. (2016).
On the other hand, using our hierarchical structures with pretrained policies introduced in Sec. 5.4 yields a considerable increase in exploration, as reported in Fig. 5(b)-5(d). These plots also show a possible one millions steps rollout, but under a randomly initialized Manager Network, hence outputting uniformly distributed one-hot vectors every T = 500 steps. The color of every point in the CoM trajectory corresponds to the latent code that was fixed at that time-step, clearly showing the “skill changes”. In the following we describe what specific architecture was used for each plot.
The rollout in Fig. 5(b) is generated following a policy with the Multi-policy architecture. This hierarchical architecture is our strong baseline given that it has used six times more samples for pretrain because it trained six policies independently. As explained in Sec. 5.4, every T = 500 steps it uses the sampled one-hot output of the Manager Network to select one of the six pre-trained policies. We observe that the exploration given by the Multi-policy hierarchy heavily concentrates around upward and downward motion, as is expected from the six individual pre-trained policies composing it (refer back to Fig. 4(a)).
Finally, the rollouts in Fig.5(c) and 5(d) use our hierarchical architecture with a SNN with bilinear integration and αH = 0 and 0.01 respectively. As described in Sec. 5.4, the placeholder that during the SNN training received the categorical latent code now receives the one-hot vector output of the Manager Network instead. The exploration obtained with SNNs yields a wider coverage of the space as the underlying policy usually has a larger span of skills. Note how most of the changes of latent code yield a corresponding change in direction. Our simple pre-train setup and the interpretability of the obtained skills are key advantages with respect to previous hierarchical structures.
7.3 MAZES AND GATHER TASKS
In Fig. 6 we evaluate the learning curves of the different hierarchical architectures proposed. Due to the sparsity of these tasks, none can be properly solved by standard reinforcement algorithms (Duan et al., 2016). Therefore, we compare our methods against a better baseline: adding to the downstream task the same Center of Mass (CoM) proxy reward that was granted to the robot in the pre-training task. This baseline performs quite poorly in all the mazes, Fig. 6(a)-6(c). This is due to the long time-horizon needed to reach the goal and the associated credit assignment problem. Furthermore, the proxy reward alone does not encourage diversity of actions as the MI bonus for SNN does. The proposed hierarchical architectures are able to learn much faster in every new MDP as they effectively shrink the time-horizon by aggregating time-steps into useful primitives. The benefit of using SNNs with bilinear integration in the hierarchy is also clear over most mazes, although pretraining with MI bonus does not always boost performance. Observing the learned trajectories that solve the mazes, we realize that the turning or sideway motions (more present in SNNs pretrained with MI bonus) help on maze 0, but are not critical in these tasks because the robot may use a specific forward motion against the wall of the maze to reorient itself. We think that the extra skills will shine more in other tasks requiring more precise navigation. And indeed this is the case in the Gather task as seen in Fig. 6(d), where not only the average return increases, but also the variance of the learning curve is lower for the algorithm using SNN pretrained with MI bonus, denoting a more consistent learning across different SNNs. For more details on the Gather task, refer to Appendix B.
It is important to mention that each curve of SNN or Multi-policy performance corresponds to a total of 10 runs: 2 random seeds for 5 different SNNs obtained with 5 random seeds in the pretrain task. There is no cherry picking of the pre-trained policy to use in the sparse environment, as done
in most previous work. This also explains the high variance of the proposed hierarchical methods in some tasks. This is particularly hard on the mazes as some pretrained SNN may not have the particular motion that allows to reach the goal, so they will have a 0 reward. See Appendix C.4 for more details.
8 DISCUSSION AND FUTURE WORK
We propose a framework for first learning a diverse set of skills using Stochastic Neural Networks trained with minimum supervision, and then utilizing these skills in a hierarchical architecture to solve challenging tasks with sparse rewards. Our framework successfully combines two parts, firstly an unsupervised procedure to learn a large span of skills using proxy rewards and secondly a hierarchical structure that encapsulates the latter span of skills and allows to re-use them in future tasks. The span of skills learning can be greatly improved by using Stochastic Neural Networks as policies and their additional expressiveness and multimodality. The bilinear integration and the mutual information bonus are key to consistently yield a wide, interpretable span of skills. As for the hierarchical structure, our experiments demonstrate it can significantly boost the exploration of an agent in a new environment and we demonstrate its relevance for solving complex tasks as mazes or gathering.
One limitation of our current approach is the switching between skills for unstable agents, as reported in the Appendix D.2 for the “Ant” robot. There are multiple future directions to make our framework more robust to such challenging robots, like learning a transition policy or integrating switching in the pretrain task. Other limitations of our current approach are having fixed sub-policies and a fixed switch time T during the training of downstream tasks. The first issue can be alleviated by introducing end-to-end training, for example using the new straight-through gradient estimators for Stochastic Computations Graphs with discrete latent variables (Jang et al., 2017; Maddison et al., 2017). The second issue is not critical for static tasks like the ones used here, as studied in Appendix C.3. But in case of becoming a bottleneck for more complex dynamic tasks, a termination policy could be learned by the Manager, similar to the option framework. Finally, we only used feedforward architectures and hence the decision of what skill to use next only depends on the observation at the moment of switching, not using any sensory information gathered while the previous skill was active. This limitation could be eliminated by introducing a recurrent architecture at the Manager level.
ACKNOWLEDGMENTS
This work was supported in part by DARPA, by the Berkeley Vision and Learning Center (BVLC), by the Berkeley Artificial Intelligence Research (BAIR) laboratory, by Berkeley Deep Drive (BDD), and by ONR through a PECASE award. Carlos Florensa was also supported by a La Caixa Ph.D. Fellowship, and Yan Duan by a BAIR Fellowship and a Huawei Fellowship.
A HYPERPARAMETERS
All policies are trained with TRPO with step size 0.01 and discount 0.99. All neural networks (each of the Multi-policy ones, the SNN and the Manager Network) have 2 layers of 32 hidden units. For the SNN training, the mesh density used to grid the (x, y) space and give the MI bonus is 10 divisions/unit. The number of skills trained (ie dimension of latent variable in the SNN or number of independently trained policies in the Mulit-policy setup) is 6. The batch size and the maximum path length for the pre-train task are also the ones used in the benchmark (Duan et al., 2016): 50,000 and 500 respectively. For the downstream tasks, see Tab. 1.
B RESULTS FOR GATHER WITH BENCHMARK SETTINGS
To fairly compare our methods to previous work on the downstream Gather environment, we report here our results in the exact settings used in Duan et al. (2016): maximum path length of 500 and batch-size of 50k. Our SNN hierarchical approach outperforms state-of-the-art intrinsic motivation results like VIME (Houthooft et al., 2016).
The baseline of having a Center of Mass speed intrinsic reward in the task happens to be stronger than expected. This is due to two factors. First, the task offers a not-so-sparse reward (green and red balls may lie quite close to the robot), which can be easily reached by an agent intrinsically motivated to move. Second, the limit of 500 steps makes the original task much simpler in the sense that the agent only needs to learn how to reach the nearest green ball, as it won’t have time to reach anything else. Therefore there is no actual need of skills to re-orient or properly navigate the environment, making the hierarchy useless. This is why when increasing the time-horizon to 5,000 steps, the hierarchy shines much more, as can also be seen in the videos.
C SNAKE
In this section we study how our method scales to a more complex robot. We repeat most experiments from Section 7 with the ”Snake” agent, a 5-link robot depicted in Fig. 8. Compared to Swimmer, the action dimension doubles and the state-space increases by 50%. We also perform a further analysis on the relevance of the switch time T and end up by reporting the performance variation among different pretrained SNNs on the hierarchical tasks.
C.1 SKILL LEARNING IN PRETRAIN
The Snake robot can learn a large span of skills as consistently as Swimmer. We report in Figs. 9(a)9(e) the visitation plots obtained for five different random seeds in our SNN pretraining algorithm from Secs. 5.1-5.3, with Mutual Information bonus of 0.05. The impact of the different spans of skills on the later hierarchical training is discussed in Sec. C.4.
C.2 HIERARCHICAL USE OF SKILLS IN MAZE AND GATHER
Here we evaluate the performance of our hierarchical architecture to solve with Snake the sparse tasks Maze 0 and Gather from Sec. 6. Given the larger size of the robot, we have also increased the size of the Maze and the Gather task, from 2 to 7 and from 6 to 10 respectively. The maximum path length is also increased in the same proportion, but not the batch size nor the switch time T (see Sec. C.3 for an analysis on T ). Despite the tasks being harder, our approach still achieves good results, and Figs. 10(a)-10(b), clearly shows how it outperforms the baseline of having the intrinsic motivation of the Center of Mass in the hierarchical task.
C.3 ANALYSIS OF THE SWITCH TIME T
The switch time T does not critically affect the performance for these static tasks, where the robot is not required to react fast and precisely to changes in the environment. The performance of very different T is reported for the Gather task of sizes 10 and 15 in Figs. 11(a)-11(b). A smaller environment implies having red/green balls closer to each other, and hence more precise navigation is expected to achieve higher scores. In our framework, lower switch time T allows faster changes between skills, therefore explaining the slightly better performance of T = 10 or 50 over 100 for the Gather task of size 10 (Fig. 11(a)). On the other hand, larger environments mean more sparcity as the same number of balls is uniformly initialized in a wider space. In such case, longer commitment to every skill is expected to improve the exploration range. This explains the slower learning of T = 10 in the Gather task of size 15 (Fig. 11(b)). It is still surprising the mild changes produced by such large variations in the switch time T for the Gather task. For the Maze 0 task, Figs. 11(c)-11(d) show that the difference between different T is important but not critical. In particular, radical increases of T have little effect on the performance in this task because the robot will simply keep bumping against a wall until it switches latent code. This behavior is observed in the videos attached with the paper2. The large variances observed are due to the performance of the different pretrained SNN. This is further studied in Sec. C.4.
C.4 IMPACT OF THE PRE-TRAINED SNN ON THE HIERARCHY
In the previous section, the low variance of the learning curves for Gather indicates that all pretrained SNN preform equally good in this task. For Maze, the performance depends strongly on the pretrained SNN. For example we observe that the one obtained with the random seed 50 has a visitation with a weak backward span (Fig. 9(e)), which happens to be critical to solve the Maze 0 (as can be seen in the videos2). This explains the lack of learning in Fig. 12(a) for this particular pretrained SNN. Note that the large variability between different random seeds is even worse in the baseline of having the CoM reward in the Maze, as seen in Fig. 12(b). 3 out of 5 seeds never reach the goal and the ones that does have an unstable learning due to the long time-horizon.
D ANT
In this section we report the progress and failure modes of our approach with more challenging and unstable robots, like Ant (described in Duan et al. (2016)). This agent has a 27-dimensional space and an 8-dimensional action space. It is unstable in the sense that it might fall over and cannot recover. We show that despite being able to learn well differentiated skills with SNNs, switching between skills is a hard task because of the instability.
D.1 SILL LEARNING IN PRETRAIN
Our SNN pretraining step is still able to yield large span of skills covering most directions, as long as the MI bonus is well tuned, as observed in Fig. 14. Videos comparing all the different gaits can be observed in the attached videos6.
D.2 FAILURE MODES FOR UNSTABLE ROBOTS
When switching to a new skill, the robot finds itself in a region of the state-space that might be unknown to the new skill. This increases the instability of the robot, and for example in the case of Ant this might leads to falling over and terminating the rollout. We see this in Fig. 15, where the 5 depicted rollouts terminate because of the ant falling over. For more details on failure cases with Ant and the possible solutions, please refer to the technical report: http://bit.ly/snn4hrl-antReport
6Videos available at: http://bit.ly/snn4hrl-videos

*Edited the score 6->7.
The paper presents a method for hierarchical RL using stochastic neural networks. The paper has introduced using information-theoretic measure of option identifiability as an additional reward for learning a diverse mixture of sub-policies. One nice result in the paper is the comparison with strong baseline which directly combines the intrinsic rewards with sparse rewards and shows that this supposedly smooth reward can’t solve tasks. Besides the argument made from the authors on difficulty on long-term credit assignment/benefits from hierarchical abstraction, one possible explanation for this might be the diversity requirement imposed in sub-policy training, which is assumed to be off in the baseline case. Wonder if this can shed insights into improving the baseline and proposing new end-to-end hierarchical policy learning as hierarchical REPS/option-critic etc. papers do. Nice visualizations.
The paper presents a promising direction, and it may be strengthened further by possibly addressing some of the following points.
1) Limited diversification of sub-policies: Both concatenation and bilinear integration allow only minimal differentiations in sub-policies through first hidden weight, which is not a problem in the tested tasks because they essentially require same locomotion policies with minimal diversification, but such limitation can be more obvious in other tasks where ideal sub-policies are more diverse. Thus it is interesting to see it apply on harder, non-locomotion domains, where ideal sub-policies are not that similar, e.g. for manipulation, solving some task from one state can be very different from solving it from another state.
2) Limitation on hierarchical policies: Manager network is trained while the sub-policies are fixed. Furthermore, the time steps for sub-policies are fixed. This requires “intrinsic” rewards and their learned sub-policies to be very good for solving down-stream tasks. It would be nice to see some more discussions/results on handling such cases, ideally connecting to end-to-end hierarchical policy learning.
3) Intrinsic/unsupervised rewards seem domain-specific/supervised rewards: Because of (2), this seems unavoidable.