summaries: This paper proposes a classifier-agnostic method for saliency map extraction. In order to address the dependence of saliency map extraction on the classifier, the authors propose to learn a saliency mapping by considering all possible classifiers (i.e., a certain classifier structure w.r.t. the space of all its parameters). The goal is to find the relevant features in the data that work with all possible classifiers. The proposed framework is formulated as a min-max game between two players: a mask m corresponding to the saliency mapping, and a function f sampled from a set of classifiers with the same structure but different parameters. The mapping m is optimized to maximize the masked-out classification error (such that m captures all relevant features whose removal can maximally confuse the classifier), while f is optimized to minimize the mask-out classification error.

summaries: This paper focuses on the extraction of high-quality model-agnostic saliency maps. The authors argue that when an extracted saliency map is directly dependent on a model, then it might not be useful for a different classifier and thus not general enough. To overcome this problem, they consider all the possible classifiers weighted by their posterior probabilities. This problem cannot be solved explicitly, and the authors suggest a scheme to approximate the solution using two networks. That is, pretrain an initial classifier and then, following an adversarial training procedure, one network is trying to confuse the classifier and the other one to maximize its accuracy. Using this formulation, the authors report state-of-the-art results for salience map extraction.

abstractText: Extracting saliency maps, which indicate parts of the image important to classification, requires many tricks to achieve satisfactory performance when using classifier-dependent methods. Instead, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance. We observe that the proposed approach extracts higher quality saliency maps and outperforms existing weakly-supervised localization techniques, setting the new state of the art result on the ImageNet dataset.

authors:
Chunshui Cao
Xianming Liu
Yi Yang
Yinan Yu
Jiang Wang
Zilei Wang
Yongzhen Huang
Liang Wang
Chang Huang
Wei Xu

title: Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks

venue: In International Conference on Computer Vision,

authors:
Piotr Dabkowski
Yarin Gal

title: Real time image saliency for black box classifiers

venue: In Neural Information Processing Systems,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: In Computer Vision and Pattern Recognition,

authors:
Lijie Fan
Shengjia Zhao
Stefano Ermon

title: Adversarial localization network. In Learning with limited labeled data: weak supervision and beyond

venue: NIPS Workshop,

authors:
Ruth C Fong
Andrea Vedaldi

title: Interpretable explanations of black boxes by meaningful perturbation

venue: arXiv preprint arXiv:1704.03296,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

authors:
Aravindh Mahendran
Andrea Vedaldi

title: Salient deconvolutional networks

venue: In European Conference on Computer Vision. Springer,

authors:
Stephan Mandt
Matthew D Hoffman
David M Blei

title: Stochastic gradient descent as approximate bayesian inference

venue: The Journal of Machine Learning Research,

authors:
Leonid I Rudin
Stanley Osher
Emad Fatemi

title: Nonlinear total variation based noise removal algorithms

venue: Physica D: Nonlinear Phenomena,

authors:
Ramprasaath R Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In ICCV,

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: arXiv preprint arXiv:1409.1556,

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency maps

venue: arXiv preprint arXiv:1312.6034,

authors:
Jost Tobias Springenberg
Alexey Dosovitskiy
Thomas Brox
Martin Riedmiller

title: Striving for simplicity: The all convolutional net

venue: arXiv preprint arXiv:1412.6806,

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

venue: arXiv preprint arXiv:1312.6199,

authors:
Christian Szegedy
Wei Liu
Yangqing Jia
Pierre Sermanet
Scott Reed
Dragomir Anguelov
Dumitru Erhan
Vincent Vanhoucke
Andrew Rabinovich

title: Going deeper with convolutions

title: An image inpainting technique based on the fast marching method

venue: Journal of graphics tools,

authors:
Max Welling
Yee W Teh

title: Bayesian learning via stochastic gradient langevin dynamics

venue: In International Conference on Machine Learning,

authors:
Matthew D Zeiler
Rob Fergus

title: Visualizing and understanding convolutional networks

venue: In European Conference on Computer Vision,

authors:
Jianming Zhang
Zhe Lin
Jonathan Brandt
Xiaohui Shen
Stan Sclaroff

title: Top-down neural attention by excitation backprop

authors:
Bolei Zhou
Aditya Khosla
Agata Lapedriza
Aude Oliva
Antonio Torralba

title: Learning deep features for discriminative localization

heading: 1 INTRODUCTION

text:
The success of deep convolutional networks for large-scale object recognition Krizhevsky et al. (2012); Simonyan & Zisserman (2014); Szegedy et al. (2015); He et al. (2016) has spurred interest in utilizing them to automatically detect and localize objects in natural images. Pioneering this direction, Simonyan et al. (2013) and Springenberg et al. (2014) demonstrated that the gradient of the class-specific score of a given classifier could be used for extracting a saliency map of an image. Such classifier-dependent saliency maps can be utilized to analyze the inner workings of a specific network. However, as only the part of the image that is used by a given model is highlighted, these methods are not identifying all “evidence” in a given image. They also tend to be noisy, covering many irrelevant pixels and missing many relevant ones. Therefore, much of the recent work has focused on introducing regularization techniques of correcting such classifier-dependent saliency maps. For instance, Selvaraju et al. (2017) propose averaging multiple saliency maps created for perturbed images to obtain a smooth saliency map. We argue, however, that applying tricks and tweaks on top of methods that were designed to analyze inner workings of a given classifier is not a principled way to get saliency maps that focus on all useful evidence.
In this work, we aim to find saliency maps indicating pixels which aid classification, i.e. we want to find pixels in the input image such that if they were masked, it would confuse an unknown classifier. Assuming we were given a classifier, a naive approximate solution would be to train a generative model to output a mask (a saliency map) confusing that classifier. That can be achieved using a simple GAN-like approach (Goodfellow et al., 2014) where the classifier acts as a fixed discriminator. Unfortunately, as we prove experimentally, this solution suffers from the same issues as prior approaches. We argue that the strong dependence on a given classifier lies at the center of the problem. To tackle this directly we propose to train a saliency mapping that is not strongly coupled with any specific classifier. Our approach, a class-agnostic saliency map extraction, can be formulated as a practical algorithm that realizes our goal.
Our focus on classifier-agnostic saliency maps is not our objective per se, it is a remedy that resolves the core problem. The proposed approach results in a neural network based saliency mapping that only depends on an input image. We qualitatively find that it extracts higher quality saliency maps compared to classifier-dependent methods, as can be seen in Fig. 2. Extracted saliency maps show all the evidence without using any symptom-masking methods: difficult to tune regularization penalties (such as total variation), exotic activation functions, complex training procedures or image preprocessing tricks (such as superpixels), etc. We also evaluate our method quantitatively by using the extracted saliency maps for object localization. We observe that the proposed approach outperforms the existing weakly-supervised techniques setting the new state of the art result on the ImageNet dataset and closely approaches the localization performance of a strongly supervised model. Furthermore, we experimentally validate that the proposed approach works reasonably well even for classes unseen during training.
Our method has many potential applications, in which being classifier-agnostic is of primary importance. For instance, in medical image analysis, where we are interested not only in class prediction but also in indicating which part of the image is important to classification. Importantly, however, it is criticial to indicate all parts of the image, which can influence diagnosis, not just ones used by a specific classifier.

text: In this paper, we tackle a problem of extracting a salient region of an input image as a problem of extracting a mapping m : RW×H×3 → [0, 1]W×H over an input image x ∈ RW×H×3. Such a mapping should retain (= 1) any pixel of the input image if it aids classification, while it should mask (= 0) any other pixel.

heading: 2.1 CLASSIFIER-DEPENDENT SALIENCY MAP EXTRACTION

text:
Earlier work has largely focused on a setting in which a classifier f was given (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017). These approaches can be implemented as solving the following maximization problem:
m = argmax m′
S(m′, f), (1)
where S is a score function corresponding to a classification loss. That is,
S(m, f) = 1
N N∑ n=1 [ l(f((1−m(xn)) xn), yn) +R(m(xn)) ] , (2)
where denotes elementwise multiplication (masking), R(m) is a regularization term and l is a classification loss, such as cross-entropy. We are given a training set D = {(x1, y1), . . . , (xN , yN )}. This optimization procedure could be interpreted as finding a mapping m that maximally confuses a given classifier f . We refer to it as a classifier-dependent saliency map extraction.
A mapping m obtained with a classifier f may differ from a mapping m′ found using f ′, even if both classifiers are equally good in respect to a classification loss for both original and masked images, i.e. L(0, f) = L(0, f ′) and L(m, f) = L(m′, f ′), where
L(m, f) = 1
N N∑ n=1 l(f((1−m(xn)) xn), yn). (3)
This property is against our definition of the mapping m above, which stated that any pixel which helps classification should be indicated by the mask (a saliency map) with 1. The reason why this is possible is these two equally good, but distinct classifiers may use different subsets of input pixels to perform classification.
An example This behaviour can be intuitively explained with a simple example, illustrating an extreme special case. Let us consider a data set in which all instances consist of two identical copies of images concatenated together, that is, for all xn, xn = [x′n;x ′ n], where x ′ n ∈ RW/2×H×3. For such a data set, there exist at least two classifiers, f and f ′, with the same classification loss. The classifier f uses only the left half of the image, while f ′ uses the other half. Each of the corresponding mappings, m and m′, would then indicate a region of interest only on the corresponding half of the image. When the input image does not consist of two concatenated copies of the same image, it is unlikely that two equally good classifiers will use disjoint sets of input pixels. Our example is to show an extreme case when it is possible.

heading: 2.2 CLASSIFIER-AGNOSTIC SALIENCY MAP EXTRACTION

text:
In order to address the issue of saliency mapping’s dependence on a single classifier, we propose to alter the objective function in Eq. (1) to consider not only a single fixed classifier but all possible classifiers weighted by their posterior probabilities. That is,
m = argmax m′
Ef [S(m′, f)] , (4)
where the posterior probability, p(f |D,m′), is defined to be proportional to the exponentiated classification loss L, i.e., p(f |D,m′) ∝ p(f) exp(−L(m′, f)). Solving this optimization problem is equivalent to searching over the space of all possible classifiers, and finding a mapping m that works with all of them. As we parameterize f as a convolutional network (with parameters denoted as θf ), the space of all possible classifiers is isomorphic to the space of its parameters. The proposed approach considers all the classifiers and we call it a classifier-agnostic saliency map extraction.
In the case of the simple example above, where each image contains two copies of a smaller image, both f and f ′, which respectively look at one and the other half of an image, the posterior probabilities of these two classifiers would be the same1. Solving Eq. (4) implies that a mapping m must minimize the loss S for both of these classifiers.
2.3 ALGORITHM Algorithm 1: Classifier-agnostic saliency map extraction
input :an initial classifier f (0), an initial mapping m(0), dataset D, number of iterations K output : the final mapping m(K)
Initialize a sample set F (0) = { f (0) } .
for k ← 1 to K do θf(k) ← θf(k−1) − ηf∇θfL(m (k−1), f (k−1))
F (k) ← F (k−1) ∪ { f (k) } f ′ ← Sample(F (k)) θm(k) ← θm(k−1) + ηm∇θmS(m (k−1), f ′) F (k) ← Thin(F (k))
The optimization problem in Eq. (4) is, unfortunately, generally intractable. This arises from the intractable expectation over the posterior distribution. Furthermore, the expectation is inside the optimization loop for the mapping m, making it even harder to solve.
Thus, we approximately solve this problem by simultaneously estimating the mapping m and the expected objective. First, we sample one f (k) with the posterior probability p(f |D,m(k−1)) by taking a single step of stochastic gradient descent (SGD) on the classification loss with respect to θf (classifier f parameters) with a small step size:
θf(k) ← θf(k−1) − ηf∇θfL(m(k−1), f (k−1)). (5)
This is motivated by earlier work (Welling & Teh, 2011; Mandt et al., 2017) which showed that SGD performs approximate Bayesian posterior inference.
We have up to k + 1 samples2 from the posterior distribution F (k) = { f (0), . . . , f (k) } . We sample3 f ′ ∈ F (k) to get a single-sample estimate of m in in Eq. (4) by computing S(m(k−1), f ′). Then, we use it to obtain an updated θm (mapping m parameters) by
θm(k) ← θm(k−1) + ηm∇θmS(m(k−1), f ′). (6)
We alternate between these two steps until m(k) converges (cf. Alg. 1). Note, that our algorithm resembles the training procedure of GANs (Goodfellow et al., 2014), where mapping m takes the role of a generator and the classifier f can be understood as a discriminator. Fan et al. (2017) also applied adversarial training in order to achieve better saliency maps. The relation to the work is discussed in details in Section 6.
Score function The score function S(m, f) estimates the quality of the saliency map extracted by m given a data set and a classifier f . The score function must be designed to balance the precision and recall. The precision refers to the fraction of relevant pixels among those marked by m as relevant, while the recall is the fraction of pixels correctly marked by m as relevant among all the relevant pixels. In order to balance these two, the score function often consists of two terms.
The first term is aiming to ensure that all relevant pixels are included (high recall). As in Eq. (2), a popular choice has been the classification loss based on an input image x masked out by 1−m(x). In our preliminary experiments, however, we noticed that this approach leads to obtaining masks with adversarial artifacts. We hence propose to use the entropyH(f((1−m(x)) x)) instead. This makes generated masks cover all salient pixels in the input, avoiding masks that may sway the class
1We assume a flat prior, i.e., p(f) = c. 2A usual practice of “thinning” may be applied, leading to a fewer than k samples. 3We set the chance of selecting f (k) to 50% and we spread the remaining 50% uniformly over F (k−1).
prediction to a different, but semantically close class. For example, from one dog species to another. The second term, R(m), excludes a trivial solution, m simply outputting an all-ones saliency map, which would achieve maximal recall with low very precision. In order that, we must introduce a regularization term. Some of the popular choices include total variation (Rudin et al., 1992) and L1 norm. For simplicity, we use the latter only.
In summary, we use the following score function for the class-agnostic saliency map extraction:
S(m, f) = 1
N N∑ n=1 [ H ( f((1−m(xn)) xn) ) − λR‖m(xn)‖1 ] , (7)
where λR > 0 is a regularization coefficient.
Thinning As the algorithm collects a set of classifiers, f (k)’s, from the posterior distribution, we need a strategy to keep a small subset of them. An obvious approach would be to keep all classifiers but this does not scale well with the number of iterations. We propose and empirically evaluate a few strategies. The first three of them assume a fixed size of F (k). Namely, keeping the first classifier only, denoted by F (F (k) = { f (0) } ), the last only, denoted by L (F (k) = { f (k) } ) and the first and
last only, denoted by FL (F (k) = { f (0), f (k) } ). As an alternative, we also considered a growing set of classifiers where we only keep one every 1000 iterations (denoted by L1000) but whenever |F (k)| = 30, we randomly remove one from the set. Analogously, we experimented with L100. Classification loss Although we described our approach using the classification loss computed only on masked images, as in Eq. (3), it is not necessary to define the classification loss exactly in this way. In the preliminary experiments, we noticed that the following alternative formulation, inspired by adversarial training (Szegedy et al., 2013), works better:
L(m, f) = 1
2N N∑ n=1 [l(f((1−m(xn)) xn), yn) + l(f(xn), yn)] . (8)
We thus use the loss as defined above in the experiments. We conjecture that it is advantageous over the original one in Eq. (3), as the additional term prevents the degradation of the classifier’s performance on the original, unmasked images while the first term encourages the classifier to collect new pieces of evidence from the images that are masked.

heading: 3 EXPERIMENTAL SETTINGS

text:
Dataset: ImageNet Our models were trained on the official ImageNet training set with ground truth class labels Deng et al. (2009). We evaluate them on the validation set. Depending on the experiment, we use ground truth class or localization labels.
Reproducibility We made our code publicly available at MASKED.

heading: 3.1 ARCHITECTURES

text:
Classifier f and mapping m We use ResNet-50 (He et al., 2016) as a classifier f in our experiments. We follow an encoder-decoder architecture for constructing a mapping m. The encoder is implemented also as a ResNet-50 so its weights can be shared with the classifier or it can be separate. We experimentally find that sharing is beneficial. The decoder is a deep deconvolutional network that ultimately outputs the mask of an input image. The overall architecture is shown in Fig. 1. Details of the architecture and training procedure are in the appendix.
Regularization coefficient λR As noticed by Fan et al. (2017), it is not trivial to find an optimal regularization coefficient λR. They proposed an adaptive strategy which gets rid of the manual selection of λR. We, however, find it undesirable due to the lack of control on the average size of the saliency map. Instead, we propose to control the average number of relevant pixels by manually setting λR, while applying the regularization term R(m(x)) only when there is a disagreement between f(x) and f((1−m(x)) x). We then set λR for each experiment such that approximately 50% of pixels in each image are indicated as relevant by a mappingm. In the preliminary experiments, we further noticed that this approach avoids problematic behavior when an image contains small objects, earlier observed by Fong & Vedaldi (2017).
We also noticed that the training of mapping m is more stable and effective when we use only images that the classifier is not trivially confused on, i.e. predicts the correct class for the original images.

heading: 3.2 EVALUATION

text:
In our experiments we only use a single architecture explained in subsection 3.1. We use the abbreviation CASM (classifieragnostic saliency mapping) to denote the final model obtained using the proposed method. Our baseline model (Baseline) is of the same architecture and it is trained with a fixed classifier (classifier-dependent saliency mapping) realized by following thinning strategy F.
Following the previous work (Cao et al., 2015; Fong & Vedaldi, 2017; Zhang et al., 2016) we discretize our mask by computing
bij(x) = { 1, if mij(x) ≥ αm(x) 0, otherwise ,
where m(x) is the average mask intensity and α is a hyperparameter. We simply set α to 1, hence the average of pixel intensities is the same for the input mask m(x) and the discretized binary mask b(x). To focus on the most dominant object we take the largest connected component of the binary mask to obtain the binary connected mask.
Visualization We visualize the learned mapping m by inspecting the saliency map of each image in three different ways. First, we visualize the masked-in image b(x) x, which ideally leaves only the relevant pixels visible. Second, we visualize the masked-out image (1− b(x)) x, which highlights pixels irrelevant to classification. Third, we visualize the inpainted masked-out image using an inpainting algorithm (Telea, 2004). This allows us to inspect whether the object that should be masked out cannot be easily reconstructed from nearby pixels.
Classification by multiple classifiers In order to verify our claim that the proposed approach results in a classifier-agnostic saliency mapping, we evaluate a set of classifers4 on the validation sets of masked-in images, masked-out images and inpainted masked-out images. If our claim is correct, we expect the inpainted masked-out images created by our method to break these classifiers, while the masked-in images would suffer minimal performance degradation.
Object localization As the saliency map can be used to find the most dominant object in an image, we can evaluate our approach on the task of weakly supervised localization. To do so, we use the ILSVRC’14 localization task. We compute the bounding box of an object as the tightest box that covers the binary connected mask.
We use three metrics to quantify the quality of localization. First, we use the official metric (OM) from the ImageNet localization challenge, which considers the localization successful if at least one ground truth bounding box has IOU with predicted bounding box higher than 0.5 and the class prediction is correct. Since OM is dependent on the classifier, from which we have sought to make our mapping independent, we use another widely used metric, called localization error (LE), which only depends on the bounding box prediction Cao et al. (2015); Fong & Vedaldi (2017). Lastly, we evaluate the original saliency map, of which each mask pixel is a continuous value between 0 and 1, by the continuous F1 score. Precision P and recall R are defined as the following:
P = ∑ (i,j)∈B∗(x)mij(x)∑
ijmij(x) and R =
∑ (i,j)∈B∗(x)mij(x)
|B∗(x)| ,
4We train twenty ResNet-50 models with different initial random sets of parameters in addition to the classifiers from torchvision.models (https://pytorch.org/docs/master/torchvision/models. html): densenet121, densenet169, densenet201, densenet161, resnet18, resnet34, resnet50, resnet101, resnet152, vgg11, vgg11_bn, vgg13, vgg13_bn, vgg16, vgg16_bn, vgg19 and vgg19_bn.
(c) Baseline
where B∗(x) is the ground truth bounding box. We compute F1 scores against all the ground truth bounding boxes for each image and report the highest one among them as its final score.

heading: 4 RESULTS AND ANALYSIS

text:
Visualization and statistics We randomly select seven consecutive images from the validation set and input them to two instances of CASM (each using a different thinning strategy – L or L100) and Baseline. We visualize the original (clean), masked-in, masked-out and inpainted masked-out images in Fig. 2. The proposed approach produces clearly better saliency maps, while the classifier-dependent approach (Baseline) produces so-called adversarial masks (Dabkowski & Gal, 2017).
We further compute some statistics of the saliency maps generated by CASM and Baseline over the validation set. The masks extracted by CASM exhibit lower total variation (2.5× 103 vs. 7.0× 103), indicating that CASM produced more regular masks, despite the lack of explicit TV regularization. The entropy of mask pixel intensities is much smaller for CASM (0.05 vs. 0.21), indicating that the mask intensities are closer to either 0 or 1 on average. Furthermore, the standard deviation of the masked out volume is larger with CASM (0.19 vs. 0.14), indicating that CASM is capable of producing saliency maps of varying sizes dependent on the input images.
Classification As shown on the left panel of Figure 3, the entire set of classifiers suffers less from the masked-in images produced by CASM than those by Baseline. We, however, notice that most of the classifiers fail to classify the masked-out images produced by Baseline, which we conjecture is due to the adversarial nature of the saliency maps produced by Baseline approach. This is confirmed by the right panel which shows that simple inpainting of the masked-out images dramatically increases the accuracy when the saliency maps were produced by Baseline. The inpainted masked-out images by CASM, on the other hand, do not benefit from inpainting, because it truly does not maintain any useful evidence for classification.
Localization We report the localization performance of CASM, Baseline and prior works in Table 1 using two different metrics. Most of the existing approaches, except for Fan et al. (2017), assume the knowledge of the target class, unlike our work. CASM performs better than all prior approaches including the classifier-dependent Baseline. The difference is statistically significant. For ten separate training runs with random initialization the worst scores 36.3 and the best 36.0 with the average of 36.1. The fully supervised approach is the only approach that outperforms CASM.
Thinning strategies In Table 2 (a–e), we compare the five thinning strategies described earlier, where F is equivalent to the Baseline. According to LE and OM metrics, the strategies L100 and L1000 perform better than the others, closely followed by L. These three strategies also perform the best in term of F1. Sharing the encoder and classifier As clear from Fig. 1, it is not necessary to share the parameters of the encoder and classifier. Our experiments, however, reveal that it is always beneficial to share them as shown in Table 2.
Score function Unlike Fan et al. (2017), we use separate score functions for training the classifier and the saliency mapping. We empirically observe in Table 2 that the proposed use of entropy as a score function results in a better mapping in term of OM and LE. The gap, however, narrows as we use better thinning strategies. On the other hand, the classification loss is better for F1 as it makes CASM focus on the dominant object only. Because we take the highest score for each ground truth bounding box, concentrating on the dominant object yields higher scores.

heading: 5 UNSEEN CLASSES

text:
Since the proposed approach does not require knowing the class of the object to be localized, we can use it with images that contain objects of classes that were not seen during training neither by the classifier f (k) nor the mapping m. We explicitly test this capability by training five different CASMs on five subsets of the original training set of ImageNet.
We first divide the 1000 classes into five disjoint subsets (denoted as A, B, C, D, E and F) of sizes 50, 50, 100, 300, 300 and 200, respectively. We train our models (in all stages) on 95% images (classes in B, C, D, E and F), 90% images (classes in C, D, E and F), 80% images (classes in D, E and F), 50% images (classes in E and F) and finally on 20% of images only (classes in F only). Then, we test each saliency mapping on all the six subsets of classes independently. We use the thinning strategy L for computational efficiency in each case.
All models generalize well and the difference between their accuracy on seen or unseen classes is negligible (with exemption of the model trained on 20% of classes). The general performance is a little poorer which can be explained by the smaller training set. In Table 3, we see that the proposed approach works well even for localizing objects from previously unseen classes. The gap in the localization error between the seen and unseen classes grows as the training set shrinks. However, with a reasonably sized training set, the difference between the seen and unseen classes is small. This is an encouraging sign for the proposed model as a class-agnostic saliency map.

heading: 6 RELATED WORK

text:
The adversarial localization network Fan et al. (2017) is perhaps the most closely related to our work. Similarly to ours, they simultaneously train the classifier and the saliency mapping which does not require the object’s class at test time. There are four major differences between that work and ours. First, we use the entropy as a score function for training the mapping, whereas they used the classification loss. This results in obtaining better saliency maps as we have shown earlier. Second, we make the training procedure faster thanks to tying the weights of the encoder and the classifier, which also results in a much better performance. Third, we do not let the classifier shift to the distribution of masked-out images by continuing training it on both clean and masked-out images. Finally, their mapping relies on superpixels to build more contiguous masks which may miss small details due to inaccurate segmentation and makes the entire procedure more complex. Our approach solely works on raw pixels without requiring any extra tricks.
Dabkowski & Gal (2017) also train a separate neural network dedicated to predicting saliency maps. However, their approach is a classifier-dependent method and, as such, a lot of effort is devoted to preventing generating adversarial masks. Furthermore, the authors use a complex training objective with multiple hyperparameters which also has to be tuned carefully. On a final note, their model needs a ground truth class label which limits its use in practice.

heading: 7 CONCLUSIONS

text: In this paper, we proposed a new framework for classifier-agnostic saliency map extraction which aims at finding a saliency mapping that works for all possible classifiers weighted by their posterior probabilities. We designed a practical algorithm that amounts to simultaneously training a classifier and a saliency mapping using stochastic gradient descent. We qualitatively observed that the proposed approach extracts saliency maps that cover all the relevant pixels in an image and that the masked-out images cannot be easily recovered by inpainting, unlike for classifier-dependent approaches. We further observed that the proposed saliency map extraction procedure outperforms all existing weakly supervised approaches to object localization and can also be used on images containing objects from previously unseen classes, paving a way toward class-agnostic saliency map extraction.


summaries: This paper proposes a new set of heuristics for learning a NN for generalising a set of NNs trained for more specific tasks. This particular recipe might be reasonable, but the semi-formal flavour is distracting. The issue of model selection (clearly the main issue here) is not addressed. A quite severe issue with this report is that the authors don't report relevant learning results from before (+-) 2009, and empirical comparisons are only given w.r.t. other recent heuristics. This makes it for me not possible to advice publication as is.

summaries: This paper presents a method for distilling multiple teacher networks into a student, by linearly combining feature representations from all networks at multiple intermediate layers, and gradually forcing the student to "take over" the learned combination.  Networks to be used as teachers are first pretrained on various initial tasks.  A student network is then trained on a target task (possibly different from any teacher task), by combining corresponding hidden layers from each teacher using learned linear remappings and weighted combinations.  Learning this combination allows the system to find appropriate teachers for the target task; eventually, a penalty on the combination weights forces all weight onto the student network, resulting in the distillation.

abstractText: A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other ‘knowledge exchange’ methods.

authors:
Yoshua Bengio
Jérôme Louradour
Ronan Collobert
Jason Weston

title: Curriculum learning

venue: In Proc. ICML,

title: pytorch-playground. https://github.com/aaron-xichen/ pytorch-playground, 2017

authors:
Z. Chen
B. Liu

title: Lifelong Machine Learning

authors:
Adam Coates
Andrew Ng
Honglak Lee

title: An analysis of single-layer networks in unsupervised feature learning

venue: In Proc. AISTATS,

authors:
Gregory Cohen
Saeed Afshar
Jonathan Tapson
André van Schaik

title: EMNIST: an extension of MNIST to handwritten letters

venue: arXiv preprint arXiv:1702.05373,

authors:
Chrisantha Fernando
Dylan Banarse
Charles Blundell
Yori Zwols
David Ha
Andrei A. Rusu
Alexander Pritzel
Daan Wierstra

title: Pathnet: Evolution channels gradient descent in super neural networks

venue: arXiv preprint arXiv:1701.08734,

authors:
Tommaso Furlanello
Jiaping Zhao
Andrew M. Saxe
Laurent Itti
Bosco S. Tjan

title: Active long term memory networks

venue: arXiv preprint arXiv:1606.02355,

venue: In Proc. CVPR,

authors:
Geoffrey E. Hinton
Oriol Vinyals
Jeffrey Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
Gao Huang
Zhuang Liu
Kilian Q. Weinberger

title: Densely connected convolutional networks

authors:
Heechul Jung
Jeongwoo Ju
Minju Jung
Junmo Kim

title: Less-forgetting learning in deep neural networks

venue: arxiv,

title: Learning multiple layers of features from tiny images

venue: Technical report, University of Toronto,

authors:
Zhizhong Li
Derek Hoiem

title: Learning without forgetting

venue: In Proc. ECCV,

authors:
Mingsheng Long
Yue Cao
Jianmin Wang
Michael I. Jordan

title: Learning transferable features with deep adaptation networks

authors:
T. Mitchell
W. Cohen
E. Hruscha
P. Talukdar
J. Betteridge
A. Carlson
B. Dalvi
M. Gardner
B. Kisiel
J. Krishnamurthy
N. Lao
K. Mazaitis
T. Mohammad
N. Nakashole
E. Platanios
A. Ritter
M. Samadi
B. Settles
R. Wang
D. Wijaya
A. Gupta
X. Chen
A. Saparov
M. Greaves
J. Welling

title: Never-ending learning

venue: In Proc. AAAI,

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis

title: Human-level control through deep reinforcement learning

venue: In Nature,

authors:
Volodymyr Mnih
Adria Puigdomenech Badia
Mehdi Mirza
Alex Graves
Timothy P. Lillicrap
Tim Harley
David Silver
Koray Kavukcuoglu

title: Asynchronous methods for deep reinforcement learning

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
Sinno Jialin Pan
Qiang Yang

title: A survey on transfer learning

venue: IEEE Trans. on Knowl. and Data Eng.,

authors:
Emilio Parisotto
Jimmy Lei Ba
Ruslan Salakhutdinov

title: Actor-mimic: Deep multitask and transfer reinforcement learning

venue: In Proc. ICLR,

authors:
Vishal M. Patel
Raghuraman Gopalan
Ruonan Li
Rama Chellappa

title: Visual domain adaptation: A survey of recent advances

venue: IEEE Signal Process. Mag.,

authors:
Andrei A. Rusu
Sergio Gomez Colmenarejo
Çaglar Gülçehre
Guillaume Desjardins
James Kirkpatrick
Razvan Pascanu
Volodymyr Mnih
Koray Kavukcuoglu
Raia Hadsell

title: Policy distillation

authors:
Andrei A. Rusu
Neil C. Rabinowitz
Guillaume Desjardins
Hubert Soyer
James Kirkpatrick
Koray Kavukcuoglu
Razvan Pascanu
Raia Hadsell

title: Progressive neural networks

venue: In arXiv preprint arXiv:1606.04671,

authors:
Paul Ruvolo
Eric Eaton

title: Ella: An efficient lifelong learning algorithm

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms

venue: arXiv preprint arXiv:1707.06347,

authors:
Yee Teh
Victor Bapst
Wojciech M. Czarnecki
John Quan
James Kirkpatrick
Raia Hadsell
Nicolas Heess
Razvan Pascanu

title: Distral: Robust multitask reinforcement learning

venue: In Proc. NIPS,

title: Analysis and optimization of convolutional neural network architectures

venue: arXiv preprint arXiv:1707.09725,

title: Lifelong learning algorithms

venue: In Learning to Learn. Springer US,

authors:
Eric Tzeng
Judy Hoffman
Trevor Darrell
Kate Saenko

title: Simultaneous deep transfer across domains and tasks

venue: In Proc. ICCV,

authors:
Yu-Xiong Wang
Deva Ramanan
Martial Hebert

title: Growing a brain: Fine-tuning by increasing model capacity

authors:
Yuhuai Wu
Elman Mansimov
Shun Liao
Roger B. Grosse
Jimmy Ba

title: Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation

authors:
Junbo Jake Zhao
Michaël Mathieu
Ross Goroshin
Yann LeCun

title: Stacked what-where autoencoders

venue: arXiv preprint arXiv:1506.02351,

heading: 1 INTRODUCTION

text:
Research communities have amassed a sizable number of deep net architectures for different tasks, and new ones are added almost daily. Some of those architectures are trained from scratch while others are fine-tuned, i.e., before training, their weights are initialized using a structurally similar deep net which was trained on different data.
Beyond fine-tuning, particularly in reinforcement learning, teachers have also been considered in one way or another by Rusu et al. (2016b); Fernando et al. (2017); Wang et al. (2017); Li & Hoiem (2016); Bengio et al. (2009); Patel et al. (2015); Chen & Liu (2016); Teh et al. (2017); Parisotto et al. (2016). For instance, progressive neural net (Rusu et al., 2016b) keeps multiple teachers during both training and inference, and learns to extract useful features from the teachers for a new target task. PathNet (Fernando et al., 2017) uses genetic algorithms to choose pathways from a giant network for learning new tasks. ‘Growing a Brain’ (Wang et al., 2017) fine-tunes a neural network while growing the network’s capacity (wider or deeper layers). Actor-mimic (Parisotto et al., 2016) pre-trains a big model on multiple source tasks, then the big model is used as a weight initialization for a new model which will be trained on a new target task. Knowledge distillation (Hinton et al., 2015) distills knowledge from a large ensemble of models to a smaller student model.
However, all the aforementioned techniques have limitations. For example, progressive neural net models (Rusu et al., 2016b) grow with the number of teachers. This large number of parameters limits the number of teachers a progressive neural net can handle, and largely increases the training and testing time. In PathNet (Fernando et al., 2017), searching over a big network for pathways is computationally intensive. For fine-tuning based methods such as ‘Growing a Brain’ (Wang et al., 2017) and actor-mimic (Parisotto et al., 2016), only one pretrained model can be used at a time. Hence, their performance heavily relies on the chosen pretrained model.
To address these shortcomings, we develop knowledge flow which moves ‘knowledge’ of multiple teachers when training a student. Irrespective of how many teachers we use, the student is guaranteed to become independent at the final stage of training and the size of the resulting student net remains constant. In addition, our framework makes no restrictions on the deep net size of the teacher and student, which provides flexibility in choosing teacher models. Importantly, our approach is applicable to a variety of tasks from reinforcement learning to fully-supervised training.
We evaluate knowledge flow on a variety of tasks from reinforcement learning to fully-supervised learning. In particular, we follow Rusu et al. (2016b); Fernando et al. (2017) and compare on the same
Atari games. In addition, we also observed significant top-1 error rate improvements on supervised learning datasets, i.e., CIFAR-10, and CIFAR-100.

heading: 2 BACKGROUND

text:
Knowledge flow is applicable to a variety of settings from supervised learning to reinforcement learning, which we briefly review to introduce notation.
Supervised Learning recovers the parameters θ of a mapping fθ : X → Y from data space X to output space Y . To this end, a dataset D = {(xi, yi)}ni=1 containing n pairs (xi, yi) (assumed to be sampled i.i.d.) is used, where xi ∈ X and yi ∈ Y . Given this dataset, the parameters θ of the mapping fθ are learned by minimizing a loss function `(x,y)(θ) composed of a regularization term R(θ) and an empirical risk `(y, fθ(x)) which compares groundtruth label y and prediction fθ(x). The parameters θ are obtained by optimizing the following program:
min θ E(x,y)∼D[`(x,y)(θ)] := E(x,y)∼D[`(y, fθ(x))] +R(θ). (1)
Hereby, the mapping fθ is obtained by maximizing the logits or a corresponding probability distribution f̂θ(y|x), i.e., fθ = argmaxy∈Y f̂θ(y|x). Here and below let the hat (‘̂·’) indicate probability distributions over appropriate domains.
Reinforcement Learning considers an agent interacting with an environment according to a policy πθπ : X → A which maps a state xt ∈ X to an action at ∈ A at time t. The policy depends on the parameters θπ. After performing action at, the agent observes the next state xt+1 and receives a scalar reward rt. The discounted return at time t is defined as Rt = ∑∞ k=0 γ
krt+k, where γ is the discount factor. The expected future reward when observing state x and when following policy πθπ is defined as V
πθπ (xt) = Eτ∼πθπ [Rt|xt], where τ = {(xt, at, rt), (xt+1, at+1, rt+1), . . .} is a trajectory generated by following πθπ from state xt.
The goal of reinforcement learning is to find a policy that maximizes the expected future reward from each state xt. Without loss of generality, in this paper, we follow the asynchronous advantage actor-critic (A3C) formulation (Mnih et al., 2016). In A3C, the policy mapping πθπ (x) = argmaxa∈A π̂θπ (a|x) is obtained from a probability distribution over states, where π̂θπ (a|x) is modeled by a deep net with parameters θπ . The value function is also approximated by a deep net Vθv (x), having parameters θv .
To optimize the policy parameters θπ given a state xt, a loss function based on a scaled negative log-likelihood and a negative entropy regularizer is common:
`τπ(θπ) = 1 |τ | ∑ t∈τ [− log π̂θπ (at|xt)(Rt − Vθv (xt))− βH(π̂θπ (·|xt))] .
Hereby, Rt = ∑k−1 i=0 γ irt+i + γ kVθv (xt+k) is the empirical k-step return obtained when starting in state xt, and |τ | is the length of the trajectory τ generated by following πθπ . The scalar β ≥ 0 is a user-specified constant, and H(π̂θπ (·|xt)) is the entropy function, which encourages exploration by favoring a uniform probability distribution π̂θπ (a|x). To optimize the value function Vθv , it is common to use the squared loss `τv(θv) = 1 2|τ | ∑ t∈τ (Rt − Vθv (xt))2.
By minimizing the empirical expectation of `τπ(θπ) and ` τ v(θv), i.e., by addressing
min θπ
Eτ∼πθπ [` τ π(θπ)], and min θv Eτ∼πθπ [` τ v(θv)], (2)
alternatingly, we learn a policy and a value function that maximize expected return.

heading: 3 KNOWLEDGE FLOW

text: Instead of optimizing the programs given in Eq. (1) and Eq. (2) from scratch, the aforementioned warm-start techniques (see Sec. 5 for more) are applicable. To address their mentioned shortcomings, we propose knowledge flow, a framework that moves ‘knowledge’ from an arbitrary number of deep nets, henceforth referred to as ‘teachers’ to a deep net under training, called the ‘student.’

heading: 3.1 OVERVIEW

text:
Knowledge flow is outlined on example deep nets in Fig. 1 (a,b). We train the parameters of the student net which are randomly initialized. To this end we take advantage of teachers, whose parameters are fixed and obtained from pre-trained models on different source tasks by different algorithms. For example, for reinforcement learning, we may consider teachers trained by A3C (Mnih et al., 2016), A2C (Dhariwal et al., 2017) or DQN (Mnih et al., 2015).
‘Knowledge’ of multiple teachers is transferred to a student by adding transformed and scaled intermediate representations from the teacher deep nets to the student net. To achieve this, we modify the student net, i.e., fθ in the supervised setting and πθπ (a|x), Vθv (x) in the reinforcement learning case. We add teacher representations which are transformed by multiplication with a trainable matrix Q and scaled via a weight pw that is normalized to sum to one for each student layer and parameterized via trainable parameters w. The normalized weights encode which of the teachers’ or the student’s representation to trust at every layer of the student net. Note that a teacher can help the student at different levels of abstraction with input from different levels of its net.
Importantly, after training, the student model should perform well on the target task without relying on teachers. To achieve this, as training progresses, we increasingly encourage a high normalized weight on the student representation, which forces the student to eventually capture all the ‘knowledge.’ Due to the trainable scaling, at an early stage of training, we observe the student to rely heavily on the ‘knowledge’ of the teacher to quickly obtain better performance. However, as training proceeds, the student is encouraged to become more and more independent. During final stages of training, the student will no longer be able to rely on teachers, which ensures that the student has learned to master the desired task on its own. This is observed in Fig. 1 (c).
To formally encourage this successive transfer we introduce two additional loss functions. The first, referred to as the dependency loss `dep(w), captures how much a student relies on teachers. It depends on the weight vector w which encodes the strength of the coupling. The second one ensures that a student’s behavior doesn’t change rapidly when the teachers’ influence decreases. We use loss `KL(·, ·) to capture the change. By combining student net modifications and additional loss terms, for the supervised task we obtain
min θ,w,Q
E(x,y)[˜̀(x,y)(θ, w,Q) + λ1`dep(w) + λ2`KL( ˜̂ fθ, ˜̂ fθold)], (3)
and for reinforcement learning the transformed program reads as follows:{ minθπ,w,Q Eτ∼π̃θπ [˜̀ τ π(θπ, w,Q) + λ1`dep(w) + λ2` τ KL( ˜̂πθπ , ˜̂πθπold )]
minθv,w,Q Eτ∼π̃θπ [˜̀ τ v(θv, w,Q)]
. (4)
Loss ˜̀··(θ, w,Q) originates from the original loss ` · ·(θ) (Eqs. (1)-(2)) by transforming the deep net to include cross-connections, hence its dependence on w,Q. The tilde (‘̃·’) denotes this dependence, also for probability distribution ˜̂f and policy distribution ˜̂π. Parameters from the current and a previous iteration are referred to via θ and θold respectively.
For both supervised and reinforcement learning, λ1 and λ2 control the strength which is used to decrease the influence of the teacher. A low λ1 allows the student to rely on teachers. Close to the end of training, the student should be independent. Therefore, we set λ1 to a small value at the beginning, and gradually increase its value as training progresses.
Note that we don’t make any assumptions about teachers and student’s objective. If a teacher’s and student’s objective differ, negative transfer may occur initially. However, the proposed method quickly decreases the weight for teacher layers to reduce this effect. Despite differences, students could potentially still benefit from the low level representation of the teachers. We do observe this low level knowledge transfer in our experiments.
In the following we first describe how to modify the deep nets, before we detail the loss functions `dep and `KL, which are used to successively decrease the influence of the teachers.

heading: 3.2 DEEP NET TRANSFORMATION AND LOSS TERMS

text:
Deep Net Transformation: Knowledge flow enhances the student by adding transformed and scaled intermediate representations from teacher models. To perform the transformation, intermediate representations from teachers are first multiplied by transformation matrices Q. Then the transformed representations from teachers and representations from the student are linearly combined. The weights for this linear combination are determined by a weight pw which is normalized to sum to one for each student layer.
Let index m = 0 denote the student model and let θ(0) refer to its parameters. Further, let θ(m), m ∈ {1, . . . ,M} denote teacher models. We use lim to refer to deep net layer i of teacher m, with i ∈ {1, . . . , Lm} and Lm the number of layers in teacher m. We define layer j of the student model to be lj0, where j ∈ {1, . . . , L0} and L0 the number of deep net layers in the student model. The output of layer lkm right before and after an activation unit is denoted z(l k m) and h(l k m) respectively.
To align a teacher’s layer lim with a student’s layer l j 0, we introduce a learnable transformation matrix Qj(lim) ∈ Rdim(l j 0)×dim(l i m), where dim(·) gives the number of elements in the corresponding layer. The matrix multiplication Qj(lim)z(l i m) aligns the representation from layer i of teacher m with the representation of layer j of the student.
For each layer j in the student model, we define a candidate set Lj , which contains lj0 and all the teachers’ layers to be considered. For example, in Fig. 1 (a), layer one of the student model is combined with layer one of teacher one and layer two of teacher two. Therefore, the candidate set of layer one of the student model is given by L1 = {l10, l11, l22}. To decide which teachers’ or the student’s representation to trust at every layer of the student net, we introduce a normalized weight pjw(l) for all j ∈ {1, . . . , L0}, where l ∈ Lj , summing to one for each layer j in the student deep net, i.e.,∑
l∈Lj pjw(l) = 1, ∀j ∈ {1, . . . , L0}.
To obtain the combined intermediate representation of layer j for the student model, we use
h(lj0) = σ  ∑ l∈Lj\lj0 pjw(l)Q j(l)z(l) + pjw(l j 0)z(l j 0)  , where pjw(l i m) determines how much the student layer j relies on transformed representations of layer i from the m-th teacher. Intuitively, if the transformed representation of the m-th teacher layer i is helpful, pjw(l i m) will be close to one. We visualize the deep net transformation in Fig. 1 (b).
Note that the intermediate representations of teachers are not changed in our framework. To obtain the output of layer lkm we apply the original activation unit to the original representation z(l i m), i.e., h(lim) = σ(z(l i m)), ∀m ∈ {1, . . . ,M}, j ∈ {1, . . . Lm}.
The maximal number of introduced matrices Q in our framework is ∑M i=1 LiL0. In practice, we don’t link a student’s layer to every layer of a teacher network. Intuitively, a teachers’ bottom layer
features are very likely irrelevant to a student’s top layer features. Indeed, we observed that linking a teachers’ bottom layer to a student’s top layer generally doesn’t yield improvements. Therefore, in practice, we recommend to link one teacher layer to one or two student layers, in which case we introduce on the order of ML0 matrices Q. Also note that while additional trainable parameters Q and w are introduced in our framework, Q and w are not part of the resulting student network since we ensure pjw(l) ≡ 0 ∀l ∈ Lj\l j 0 at the end of training as discussed next. Hence, the additional parameters function as auxiliary knobs that help the student learn faster. In the final stage of training, the student will be independent (see Fig. 1 (c)) and does no longer rely on Q, w, or any transformed representations from teachers.
Decreasing Teachers’ Influence: We successively decrease the influence of the teachers during training by gradually encouraging the normalized weight pjw(l j 0) to increase to a value of 1 ∀j ∈ {1, 2, . . . , L0}. To capture how much the student relies on teachers, we introduce the dependence cost as the negative log probability:
`dep(w) = − 1
L0 ∑ j∈{1,2,...,L0} log pjw(l j 0). (5)
By minimizing `dep(w), we encourage weights for the layers of the student to increase. Hence we encourage the student to become more and more independent. During the final stage of training, pjw(l j 0) approaches one for all j ∈ {1, . . . , L0}, making the student independent of the transformed representation obtained from teachers.
Empirically, we found that a fast decrease of the influence of the teacher can degrade the performance. This is intuitive as it requires some time to find good transformations Q. Moreover, decreasing the influence of a teacher too fast may change the output distribution over labels or actions of the student model too much, and thus lead to performance loss. To prevent changing a student’s output distribution too fast, we found a Kullback-Leibler (KL) regularizer to yield good results. More specifically, in the case of supervised learning we use
`KL( ˜̂ fθ, ˜̂ fθold) = DKL[ ˜̂ fθ(·|x)|| ˜̂fθold(·|x)]. (6)
Hereby, θ is the set of current parameters, and θold are the previous ones. In the reinforcement learning case we use DKL[˜̂πθ(·|xt)||˜̂πθold(·|xt)].

heading: 4 EXPERIMENTAL RESULTS

text: In the following we evaluate knowledge flow on reinforcement and supervised learning tasks. Results are reported by using only the student model to avoid even the smallest influence from any teacher nets.

heading: 4.1 REINFORCEMENT LEARNING

text:
We evaluate knowledge flow on reinforcement learning using Atari games that were used by Rusu et al. (2016b); Fernando et al. (2017). Following existing work, the input to our agent are raw images from the environment. The agent learns to predict actions only based on the rewards and the input
images from the environment. The agent chooses an action every four frames, and the last action is repeated on the skipped four frames. For all teacher models and the student model, we use the fully forward architecture of A3C (Mnih et al., 2016). The model has three hidden layers. The first layer is a convolutional layer with 16 filters of size 8x8 and stride 4. The second layer is a convolutional layer with 32 filters of size 4x4 and stride 2. The third layer is a fully connected layer with 256 hidden units. Following the third hidden layer are two sets of output. One is a softmax output that provides a probability distribution over all valid actions. The other one is a scalar output that provides the estimated value function. We use the same hyper-parameter settings as Mnih et al. (2016) except for the learning rate. Mnih et al. (2016) use RMSProp with shared statistics while we use Adam with shared statistics, which we found to give better results when training the baselines. The learning rate is set to 10−4 and gradually decreased to zero for all experiments. To select λ1 and λ2 in our framework, we follow progressive neural net (Rusu et al., 2016b): randomly sample λ1 ∈ {0.05, 0.1, 0.5} and λ2 ∈ {0.001, 0.01, 0.05}. Note that λ1 is set to zero at the beginning of training, and linearly increased to the sampled value at the end of training. Following Rusu et al. (2016b), we repeat each experiment 25 times with different random seeds and randomly sampled λ1 and λ2. The results of the top three out of 25 runs are reported. As A3C, we run 16 agents on 16 CPU cores in parallel.
Evaluation Metrics: We follow the evaluation procedure of Mnih et al. (2015). The trained student models are evaluated by playing each game for 30 episodes. We also follow the ‘no-op’ procedure: at the beginning of each testing episode, the agents perform up to 30 ‘no-op’ actions.
Results: We first compare our framework with PathNet (Fernando et al., 2017) and progressive neural net (PNN) (Rusu et al., 2016b), which are state-of-the-art transfer reinforcement learning frameworks, using their experimental settings. The comparison is summarized in Table 1. The state-of-the-art results (Mnih et al., 2016; Schulman et al., 2017; Wu et al., 2017) on Atari games are also included in Table 1 for reference. Compared to PathNet, a student model trained using our transfer framework with one teacher achieves higher scores in 11 out of 14 experiments. Compared with PNN, for a two-teacher framework, our trained student model has only 0.7M parameters and PNN has 16M parameters. Nonetheless we observe higher scores in five out of the seven experiments. The results demonstrate that knowledge flow effectively transfers knowledge from teachers to the student. Table 1 also indicates that, in our framework, when the number of teachers increases from one to two, the student’s performance improves significantly across all experiments. The training curves for the experiments are shown in Fig. 2. The curve is the average of the top three out of 25 runs. We observe our approach to generally perform very well.

heading: 4.2 SUPERVISED LEARNING

text:
For supervised learning, we use a variety of image classification benchmarks, including CIFAR10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), and EMNIST (Cohen et al., 2017). The parameters λ1 for the dependent cost and λ2 for the KL cost are determined using the validation set of each dataset.
Evaluation Metrics: To evaluate the trained student model we report top-1 error rate on the test set of each dataset. All plots and reported numbers are the average of three runs obtained using different random seeds.
CIFAR-10/CIFAR-100: CIFAR-10 and CIFAR-100 datasets consist of colored images of size 32× 32. CIFAR-10 (C10) has 10 classes and CIFAR-100 (C100) has 100 classes. For both dataset, the training and test sets contain 50,000 and 10,000 images respectively. We perform all experiments on CIFAR-10 and CIFAR-100 with standard data augmentation (Huang et al., 2017).
We use Densenet (Huang et al., 2017) (depth 100, growth rate 24) as a baseline and follow their hyper-parameter settings to train our baseline, teacher and student models. For our approach, we first train teachers on CIFAR-10, CIFAR-100, and SVHN (Netzer et al., 2011). We then train the student model using a different combination of teachers. We compare our results to fine-tuning and the baseline model. As shown in Table 3 (a), for the CIFAR-10 target task, fine-tuning from the CIFAR-100 expert improves 4% over the baseline. Fine-tuning from the SVHN expert performs worse than the baseline model. Intuitively, for the CIFAR-10 target task, the CIFAR-100 deep net is a good teacher while a deep net trained with SVHN isn’t. Presented with both good and inadequate teachers, knowledge flow improves by 13% over the baseline. This demonstrates that knowledge flow can not only leverage a good teacher’s ‘knowledge,’ but it can also avoid misleading influence. As detailed in Table 3 (b), the results are similar on the CIFAR-100 dataset.
To further demonstrate the properties of knowledge flow, additional results are in the appendix.

heading: 5 RELATED WORK

text:
As mentioned before, ‘knowledge’ transfer has been considered using a variety of techniques. We briefly discuss related work in contrast to our approach in the following and defer details to Sec. 8. PathNet (Fernando et al., 2017) enables multiple agents to train the same deep net while reusing parameters and avoiding catastrophic forgetting. In contrast to this formulation we consider availability of multiple pre-trained teacher nets. Progressive Net (Rusu et al., 2016b) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features. Our discussed method uses similar lateral connections. However, in contrast to Rusu et al. (2016b), our method ensures independence of the student upon training, addressing a limitation in (Rusu et al., 2016b) where only a fraction of the capacity of the student is eventually utilized. Distral a neologism combining ‘distill & transfer learning’ (Teh et al., 2017) considers joint training of multiple tasks. Multiple tasks share a ‘distilled’ policy which encodes common behavior between different tasks. While each worker addresses its own task, a shared policy encourages consistency between the policies. Different from Distral, which is a multi-task learning framework, knowledge flow addresses a single task, while in multi-task learning, multiple tasks are addressed at the same time. Hence, common for multi-task learning and knowledge flow is a transfer of information. However, in multi-task learning, information extracted from different tasks are shared to boost performance, while, in knowledge flow, the information of multiple teachers is leveraged to help a student learn better a single, new, previously unseen task.
Other related work includes actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016), growing a brain (Wang et al., 2017), policy distillation (Rusu et al., 2016a), domain adaptation (Pan & Yang, 2010; Long et al., 2015; Tzeng et al., 2015), knowledge distillation (Hinton et al., 2015) or lifelong learning (Chen & Liu, 2016). A more detailed discussion on related work is provided in Sec. 8 of the supplementary material.

heading: 6 CONCLUSION

text: We developed a general knowledge flow approach that permits to train a deep net from any number of teachers. We showed results for reinforcement learning and supervised learning, demonstrating improvements compared to training from scratch and to fine-tuning. In the future we plan to learn when to use which teacher and how to actively swap teachers during training of a student.

heading: 7 APPENDIX

heading: 7.1 SUPERVISED LEARNING

text:
Comparison with Knowledge Distillation: We follow knowledge Distillation (KD) (Hinton et al., 2015) to distill knowledge from a larger model (teacher) to a smaller model (student). The student models have 50% - 5% parameters of the teacher models. Following their setup, we conduct experiments on MNIST, MNIST with digit ‘3’ missing in the training set, CIFAR-100, and ImageNet. For MNIST and MNIST with digit ‘3’ missing, following KD, the teacher model is an MLP with two hidden layers of 1200 hidden units, and the student model is an MLP with two hidden layers of 800 hidden units. For CIFAR-100, we use the model from Chen (2017) as teacher model. The student model follows the structure of the teacher, but the number of output channels of each convolutional layer is halved. For ImageNet, the teacher model is a 50-layer ResNet (He et al., 2016), and the student model is a 18-layer ResNet. The test error of the distilled student model are summarize in Table 4. Our framework has consistently better performance than KD, because the student model in our framework benefits not only from the output layer behavior of the teacher but also from intermediate layer representations of the teacher.
EMNIST:
The ‘EMNIST Letters’ dataset consists of images of size 28× 28 pixels showing handwritten letters. It has 26 balanced classes. Each class contains lower and upper case letters. The training and test sets contain 124,800 and 20,800 images respectively. The ‘EMNIST Digits’ dataset consists of images of size 28× 28 pixels showing handwritten digits. It has 10 balanced classes. The training and test sets contain 240,000 and 40,000 images respectively.
In this case we use the MNIST model from Chen (2017) as a baseline, teacher and student model. We trained teachers on EMNIST Digits, EMNIST Letters, and EMNIST Letters with only 13 classes. Our target task is EMNIST Letters. The student model is trained with different teachers and the results are compared to fine-tuning, the baseline model, and the state-of-the-art results on EMNIST. The results are summarized in Table 5. Compared to the baseline and fine-tuning, student learning in our framework with expert teacher (EMNIST Letters), semi-expert teacher (Half EMNIST Letters), and non-expert teacher (EMNIST Digits) all have better performance. In Fig. 4 we illustrate the accuracy over epochs for training of different models.
STL-10:
The STL-10 dataset consist of colored images of size 96× 96 pixels. It has 10 balanced classes. The training set contains 5,000 labeled images and 100,000 unlabeled images. The test set contains 8,000 images. In our experiment, we only use the 5,000 labeled images for training.
We use the STL-10 model from Chen (2017) as our baseline, teacher and student model. We trained teachers on CIFAR-10 and CIFAR-100. We compare our results to fine-tuning and the baseline in Table 6. Note that STL-10 is very similar to CIFAR-10 and CIFAR-100. Therefore, both CIFAR-10 and CIFAR-100 are very good teachers. As shown in Table 6, compared to the baseline, fine-tuning a
model using weights pretrained on CIFAR-10 and CIFAR-100 reduce test errors by more than 10%. Compared with fine-tuning, student model training in our framework further reduces the test error by 3%. Note that we only train on the labeled data while other approaches use this data for testing of semi-supervised approaches. Hence our results are obtained using fewer data and may not be directly comparable. We still list their results in Table 6 for reference. In Fig. 5 we illustrate the accuracy over the epochs of training.

heading: 7.2 REINFORCEMENT LEARNING

text: We also compare to Distral (Teh et al., 2017), which is the state-of-the-art multi-task reinforcement learning framework. We used ‘KL + ent 1 col’, which has a central model (m0), and a task model (mi) for each task. We perform the experiments on Atari games. In the experiments, we have three tasks (task 1, task 2, task 3). The teachers of task 2 (m2) and task 3 (m3) are provided for our framework. Distral is trained for 120M steps (40M steps/task), and our model is trained for 40M steps. For fair comparison, we report results of Distral’s task 1 model (m1), which is better than its center model (m0). The results are summarized in Table 7. Distral is suboptimal, because it aims to learn a multi-task agent. In addition, identical action and state space is assumed. When the target task is very different from the source tasks, Distral cannot decrease the teacher influence. In contrast, our framework can decrease a teacher’s influence, and thus reduce negative transfer.

heading: 7.3 VISUALIZATION OF NORMALIZED WEIGHTS OF TEACHERS AND STUDENT

text: Following the reviewer’s suggestion, we plot the averaged normalized weight (pw) for teachers and the student in the C10 experiment, where C100 and SVHN experts are teachers. Intuitively, the C100 teacher should have a higher pw value than the SVHN teacher, because C100 is more relevant to C10. The plot verifies this intuition. As shown in Fig. 7, pw of the C100 teacher is higher than that of the SVHN teacher over the entire training. Note, both teachers’ normalized weights approach zero at the end of training.

heading: 7.4 ABLATION STUDIES

heading: 7.4.1 UNTRAINED TEACHER MODELS

text: To verify that the student really benefits from the knowledge of teachers, we conduct an ablation study suggested by a reviewer. We use teacher models that haven’t been trained at all. Intuitively, learning with untrained teachers should have worse performance than learning with knowledgeable teachers. Our experiments verify this intuition. In Fig. 8 (a), where the target task is hero, learning with untrained teachers (‘w/ untrained teachers’) achieves an average reward of 15934. Learning with knowledgeable teachers (‘Ours with seaquest and riverraid teacher’) achieves an average reward of 30928. More results are presented in Figs. 8 (b, c). The results show that knowledge flow achieves higher rewards than training with untrained teachers in different environments and teacher-student settings.

heading: 7.4.2 TRAINING WITHOUT KL TERM

text: The KL term prevents the student’s output distribution over actions or labels from drastic changes when the teachers’ influence is decreasing. To investigate the importance of the KL term, we conduct an ablation study where the KL coefficient (λ2) is set to zero. The result is summarized in Fig. 9. Considering Fig. 9 (a), where the target task is MsPacman and the teachers are Riverraid and Seaquest experts. Without the KL term, when a teacher’s influence decreases, the rewards drop drastically. In contrast, with a KL term, we don’t observe performance drops. At the end of training, learning with the KL term achieves an average reward of 2907 and learning without the KL term achieves an average reward of 1215. More results are presented in Fig. 9 (b, c), which shows that training with the KL term achieves higher reward than training without the KL term.

heading: 7.5 TEACHERS WITH DIFFERENT ARCHITECTURE THAN STUDENT

text:
In additional experiments, following the suggestion of a reviewer, we use architectures for the teacher which differ from the student model. More specifically, we use the model of Mnih et al. (2015) as a teacher model. The teacher model consists of 3 convolutional layers, which have 32, 64, and 64 filters, followed by a hidden fully connected layer which has 512 ReLUs. We use the model of Mnih et al. (2016) as the student model. The student model consists of 2 convolutional layers, which have 16 and 32 filters respectively, followed by a hidden fully connected layer which has 256 ReLUs. Both models’ fully connected layers are followed by two output layers for actions and values. In the experiments, we link each teacher’s first convolutional layer to the student’s first convolutional layer. Moreover, we link each teacher’s third convolutional layer to the student’s second convolutional layer, and each teacher’s fully connected layer to the student’s fully connected layer. In the experiment, the target task is KungFu Master, and the teachers are experts for Seaquest and Riverraid. The results are summarized in Fig. 10. We observed that learning with teachers, whose architecture differs from the student, to have similar performance as learning with teachers which have the same architecture. Consider as an example Fig. 10 (a), where the target task is KungFu Master, and the teachers are experts for Seaquest and Riverraid. At the end of training, learning with teachers of different architectures achieves an average reward of 37520, and learning with teachers of the same architecture achieves an average reward of 35012. More results are shown in Fig. 10 (b, c). The results show that knowledge flow can enable higher rewards, even if the teachers and the student architectures differ.
7.6 AVERAGE NETWORK AS θold
For the parameters θold an average network can be used. To investigate how usage of an average network to obtain the parameters θold affects the performance, we conduct an experiment where θold is computed using the exponential running average of the model weight. More specifically, θold is updated as follows: θold ← α · θold + (1 − α) · θ, where α = 0.9. The results are summarized in Fig. 11. We observe that using an exponential average to compute θold results in very similar performance as using a single model. Consider Fig. 11 (a), where the target task is Boxing and the teacher is a Riverraid expert. At the end of training, using an average network to obtain θold achieves an average reward of 96.2 and using a single network to obtain θold achieves an average reward of 96.0. More results on using an average network are shown in Fig. 11 (b, c).

heading: 8 RELATED WORK

text:
As mentioned before, variants of ‘knowledge’ transfer have been considered using a variety of techniques, for instance, fine-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016). Also related are techniques on transfer learning and lifelong learning. We discuss those methods and contrast them to our approach in the following. PathNet (Fernando et al., 2017) enables multiple agents to train the same giant deep net while reusing parameters and avoiding catastrophic forgetting. To this end, agents embedded in the neural net discover which weights can be reused for new tasks and restrict application of gradients to those parameters. In contrast to this formulation we consider availability of multiple teacher nets, which are trained. Progressive Net (Rusu et al., 2016b) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features. Our discussed method uses similar lateral
connections. However, in contrast to Rusu et al. (2016b), we introduce scaling with normalized weights. This ensures independence of the student upon training, addressing a limitation in (Rusu et al., 2016b) where only a fraction of the capacity of the student is eventually utilized. Distral a neologism combining ‘distill & transfer learning’ (Teh et al., 2017) considers joint training of multiple tasks. Multiple tasks share a ‘distilled’ policy which encodes common behavior between different tasks. While each worker addresses its own task, a shared policy encourages consistency between the policies. Different from Distral, which is a multi-task learning framework, knowledge flow addresses a single task, while in multi-task learning, multiple tasks are addressed at the same time. Hence, common for multi-task learning and knowledge flow is a transfer of information. However, in multi-task learning, information extracted from different tasks are shared to boost performance, while, in knowledge flow, the information of multiple teachers is leveraged to help a student learn better a single, new, previously unseen task. Knowledge distillation (Hinton et al., 2015) distills information form a larger deep net into a smaller one. It assumes both nets are trained on the same dataset. In contrast, our technique allows knowledge transfer between different source and target domains. Actor-mimic (Parisotto et al., 2016) enables an agent to learn how to address multiple tasks simultaneously and generalize the extracted knowledge to new domains. A single policy net learns how to act in a set of tasks following the guidance of several expert teachers. A combination of feature regression and cross entropy loss is used to encourage the student to produce similar actions and representations. Our proposed technique differs in that we take advantage of a teachers representation at the beginning of training, Learning without forgetting (Li & Hoiem, 2016) permits to add a new task to a deep net without forgetting the original capabilities. Importantly, only data from the new task is used and the old capabilities are retained by first recording the old networks output on the new data. Similar techniques have been developed by Furlanello et al. (2016); Jung et al. (2016). In contrast, we transfer ‘knowledge’ from teacher networks more explicitly.
Growing a Brain (Wang et al., 2017) analyzes the parameters which change during fine-tuning and points out that more natural model adaptation is obtained when increasing the model capacity, by either extending width or depth. Appropriate normalization is essential to significantly outperform classical fine-tuning. Since this technique is based on fine-tuning, it differs from our student-teacher based approach.
Other related work includes policy distillation (Rusu et al., 2016a), domain adaptation (Pan & Yang, 2010; Long et al., 2015; Tzeng et al., 2015) or lifelong learning (Chen & Liu, 2016; Thrun, 1998; Mitchell et al., 2015; Ruvolo & Eaton, 2013).

title: KNOWLEDGE FLOW: IMPROVE UPON YOUR TEACH-