Likelihood-based generative models are a promising resource to detect out-ofdistribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

# ==== Summary of the paper ====
This paper proposes a new criterion for out-of-distribution (OOD) detection for
generative modelling of images (absence of labels). The OOD detection task is defined as follows: given a generative model p(x) trained on some data $X \sim p^*$ (where $p^*$ is the true unknown data generating distribution), and a test point y, how can we make use of p to detect that y is not drawn from $p^*$ (i.e., out of distribution)? The first natural idea is to use the likelihood (density) given by p, which, by now, is known to be problematic since the likelihood can be high when evaluated on data points from a completely different domain.
This paper contributes the following results:
1. Show that "simple" images (e.g., constant color) tend to give high likelihood, whereas complex images (e.g., noise) tend to give low likelihood. See Figure 2. The paper further quantifies this complexity with the normalized size of the compressed input images, as given by a compression algorithm. The paper shows that there is a negative correlation between the compressed size and the likelihood (models trained on CIFAR 10. See Figure 4).
2. To address this issue with using only the likelihood for OOD detection, the paper proposes a new measure $S(x)$ given by the negative log likelihood minus the normalized compressed size. See Eq 1. The paper connects this measure to Bayesian model selection.
Empirical results on more than 10 image datasets show that the proposed measure works better than the negative log likelihood in most cases.
# ==== Review ====
The paper is easy to follow. The finding that simple images tend to give high likelihood is interesting. My concerns are:
1. How much does the conclusion that "simple images tend to give high likelihood" depend on the complexity of the model? As far as I can see, only a few models are studied here: Glow and PixelCNN++. What is the quality of the learned models? Do they generate realistic images? What happens if you consider simpler models (say, reduce the number of layers in Glow.)?
2. Given a model p and a test input image y, how exactly do you tell if y is out of distribution? Is there a threshold? If so, what is the threshold? I understand that by AUROC used in Table 1, you vary the threshold. For each value of the threshold, you compute the true positive and false positive rates, and plot the ROC curve. The reported numbers in Table 1 are areas under the curve. Is this correct? But this does not explain how to perform OOD detection given one input image.
3. What is the reason for the poor AUROC in the case of TinyImageNet in Table 1? This is an interesting case since it may suggest that there are other hidden factors (besides the complexity of images) that can affect the OOD detection with the likelihood.
4. Have you tried to run the experiment in Table 1 with a model trained on Constant (simplest images) or Noise (most complex images)?  What happens? Also, why not also include CIFAR10 (used to train the model) in the list of datasets in Table 1?
Overall, my main concerns are with the thoroughness of the experiments, and that the two main contributions (summarized above) may not be enough. I will consider my evaluation again after seeing responses from the authors.
# ==== Minor. Did not affect the score ====
* The bottom margin seems off. Please check whether the Latex template is used correctly.
* The paragraph before Section 3: "... could replace the computed likelihood values for the negative of our complexity estimate ..." I think it is too soon to make this conclusion.
* The term "likelihood ratio test" used in Section 3.2 is very misleading. There is no hypothesis testing there.
* Might be better to briefly describe the meaning of AUROC at the beginning of section 5.


We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two finegrained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.

The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.
I am giving a score of 3. The engineering effort in the paper is appreciated, but the novelty is lacking. The losses introduced in the paper, broadly classified as contrastive loss, has been around since (Frome et al., 2013). See also (Karpathy et al., 2014; Wang et al., 2016) and in particular (Xian et al., 2016). The subtle differences between all these losses are whether there is a margin, whether the loss is smooth (i.e., a softmax as opposed a max), and how the negative samples are selected. I am surprised that the loss function being used by many are considered a contribution in the paper. The property of metric scaling is certainly interesting, but it does not answer why metric scaling is needed in the first place. Overall, the novelty is limited.
Below are some minor points and questions for the paper.
... anchor embeddings learned in one modality as prototypes ...
--> the word "prototype" is used extensively in the rest of the paper. i know this is a common term in the zero-shot learning community, but it might be good to give a formal definition early or at least give an informal definition.
Algorithm 1
--> what is the choice of d in the experiments?
... the probability of image v_i and text t_j to belong to the same object distance
--> isn't this the constrastive loss? what does it mean for the two to belong to the same object distance?
equations (2)
--> it seems that v_i and t_i are paired, but it is unclear from the text.
equation (1) and (2)
--> have you considered other (distributions of) negative samples?
... P{\hat{y} \in Y^{tr} | y_v \in Y^{ts}} is significantly greater than P{\hat{y} \in Y^{tr} | y_v \in Y^{tr}}
--> what's the definition of P{\hat{y} | y_v}? the notation is confusing and hinders the understanding of the rest of the discussion. it's probably easier to just talk about distances and use the nearest neighbor argument.
Figure 2
--> what's different between the 10 different runs? random seeds?
Figure 3
--> why is the performance significantly lower when lambda=0? can this be related to how negative samples are selected?
References:
DeViSE: a deep visual-semantic embedding model
A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov
Neurips, 2013
Deep fragment embeddings for bidirectional image sentence mapping
A. Karpathy, A. Joulin, F. Li
Neurips, 2014
Learning deep structure-preserving image-text embeddings
L. Wang, Y. Li, S. Lazebnik
CVPR, 2016
Latent embeddings for zero-shot classification
Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele
CVPR, 2016


We present network embedding algorithms that capture information about a node from the local distribution over node attributes around it, as observed over random walks following an approach similar to Skip-gram. Observations from neighborhoods of different sizes are either pooled (AE) or encoded distinctly in a multiscale approach (MUSAE). Capturing attribute-neighborhood relationships over multiple scales is useful for a diverse range of applications, including latent feature identification across disconnected networks with similar attributes. We prove theoretically that matrices of node-feature pointwise mutual information are implicitly factorized by the embeddings. Experiments show that our algorithms are robust, computationally efficient and outperform comparable models on social, web and citation network datasets.
Figure 1: Phenomena affecting and inspiring the design of the multi-scale attributed network embedding procedure. In Figure 1a attributed nodes D and G have the same feature set and their nearest neighbours also exhibit equivalent sets of features, whereas features at higher order neighbourhoods differ. Figure 1b shows that as the order of neighbourhoods considered (r) increases, the product of the adjacency matrix power and the feature matrix becomes less sparse. This suggests that an implicit decomposition method would be computationally beneficial.
Our key contributions are:
1. to introduce the first Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods, both pooled (AE) and multi-scale (MUSAE), and their counterparts that attribute distinct features to each node (AE-EGO and MUSAE-EGO);
2. to theoretically prove that their embeddings approximately factorize PMI matrices based on the product of an adjacency matrix power and node-feature matrix;
3. to show that popular network embedding methods DeepWalk (Perozzi et al., 2014) and Walklets (Perozzi et al., 2017) are special cases of our AE and MUSAE;
4. we show empirically that AE and MUSAE embeddings enable strong performance at regression, classification, and link prediction tasks for real-world networks (e.g. Wikipedia and Facebook), are computationally scalable and enable transfer learning between networks.
We provide reference implementations of AE and MUSAE, together with the datasets used for evaluation at https://github.com/iclr2020/MUSAE.

This paper introduces Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods. Algorithm 1 and 2 shows that in fact they propagate randomly selected node features to neighbors. The reviewer doesn’t think this random-walk way for selecting node feature is appropriate.  Node features describe node content. The features of neighboring nodes may complement each other. However, there is no benefit to select random features and then propagate, given that there already many approaches smartly combining node content in neighborhood.
The proof part follows Qiu et al (2018). But it is unclear, why c^{−1}A is the stationary joint distribution over consecutive nodes p(wj , wj+1).  and c^{−1}DF describes the stationary joint distribution p(f,wj) over nodes and features. There needs more explanation.
The Remark 1 and 2 discuss the case AE with F=I_|V|,  which is in fact the case when there is no node attributes. In this case, the AE process naturally goes back to plain network embedding, where DeepWalk and WalkLets are proposed for. Therefore, these remarks are done by Qiu et al (2018) already, not make no much new contribution here.
Figure 2 shows that the presented two approaches just slightly better or equivalent, or sometimes worse than baseline methods. As mentioned earlier, it is not beneficial to randomly select features to propagate.
The experiments presented in Section 5.2 evaluate whether the learned embedding can be used for label inference in a different graph. But it is unknown how success the transferring is. There is no F1-score of a solution that does embedding of the target network itself independently, and then classify the target network nodes.


Decision forests (DF), in particular random forests and gradient boosting trees, have demonstrated state-of-the-art accuracy compared to other methods in many supervised learning scenarios. In particular, DFs dominate other methods in tabular data, that is, when the feature space is unstructured, so that the signal is invariant to permuting feature indices. However, in structured data lying on a manifold-such as images, text, and speech-neural nets (NN) tend to outperform DFs. We conjecture that at least part of the reason for this is that the input to NN is not simply the feature magnitudes, but also their indices (for example, the convolution operation uses "feature locality"). In contrast, naïve DF implementations fail to explicitly consider feature indices. A recently proposed DF approach demonstrates that DFs, for each node, implicitly sample a random matrix from some specific distribution. Here, we build on that to show that one can choose distributions in a manifold aware fashion. For example, for image classification, rather than randomly selecting pixels, one can randomly select contiguous patches. We demonstrate the empirical performance of data living on three different manifolds: images, time-series, and a torus. In all three cases, our Manifold Forest (MO R F) algorithm empirically dominates other state-of-the-art approaches that ignore feature space structure, achieving a lower classification error on all sample sizes. This dominance extends to the MNIST data set as well. Moreover, both training and test time is significantly faster for manifold forests as compared to deep nets. This approach, therefore, has promise to enable DFs and other machine learning methods to close the gap with deep nets on manifold-valued data.

====== Post Rebuttal ======
No rebuttal was provided and all reviewers have raised issues. Therefore, I will maintain the original rating.
====== Summary======
The paper puts forward a potential issue with the standard decision tree and tries to remedy it. The issue is that standard decision trees, by independently picking the split feature(s), virtually discard the structure of the input features (if any). In this work, structure essentially refers to ordered input features such as sequences, signals, and images. The work is motivated by the fact that convolutional networks (ConvNet) exploit this structure thanks to its local convolution operation. The idea of this work is to bridge this gap by constraining the split feature selection to follow the local structures of the input where each split feature would correspond to a local bounding box of the input feature. Applying this idea to random forests, it demonstrates significant improvements over unstructured split features, on a few synthesized datasets as well as MNIST dataset. The performance is also compared to one ConvNet architecture which demonstrates similar performance on synthesized datasets while being considerably slower than the proposed random forests.
====== Strengths and Weaknesses ======
+ The motivation to make random forests respect the input structure similar to ConvNets is well-grounded and important since random forests are still being used in certain applications where computational complexity and/or interpretability are crucial factors.
+ It proposes a simple technique to add locality to the split criterion which brings significant improvements over the standard random forest.
- I believe the paper’s title should be closing the gap to *convolutional* neural networks since fully-connected networks are not local in the first place.
- continuing on the previous point, the proposed method pushes the decision forests closer to “locally-connected” networks where, although the features are local, they are not shared across different locations in the input. The additional “sharing” property which takes locally-connected networks to convolutional networks is an important property of ConvNets since it brings translation equivariance for the representations. The proposed random forest method is not translation-equivariant by design and is only local.
- regarding the previous point, a larger difference between ConvNet and the proposed method is more imminent if one goes to datasets with “non-aligned” observations. This already becomes more evident in the MNIST (which contains mostly aligned digits) where ConvNet clearly outperform MORF but would likely become more significant when going to real-world datasets, e.g. CIFAR.
- the proposed method is very similar to patch-based random forest image recognition methods. For instance, several variants exist that are used for object or part detection in a given image where a patch is selected from the image for the split criterion. This patch will respect locality in a similar way to the proposed MORF’s bounding boxes. For instance, see “tomography scans” example of Criminisi et al. 2012 (section 4.5).
- The paper is missing to provide many important details
- what are the actual hyperparameter (hp) values used for the different methods in figure 1,2, and 3. This includes the hp relevant for the random forests including the number of decision trees, the stopping criterion, the random data partitioning method, number of random projections, as well as h_min, h_max, w_min, and w_max. It also does not discuss the hyper parameters of the baseline methods including k for KNN, distance measure for KNN, penalty cost for SVM, variance for the RBF kernel, ConvNet architecture, etc.
- more importantly, it is not mentioned how these hp are optimized for the different baselines as well as the proposed method. What algorithm has been used (e.g. grid search)? How much hp optimization budget is used for different baselines? Is there a validation set put aside for hp optimization?
- from the description in the start of page 4, it seems that the atoms for the proposed MORF are vectors of binary dimensions while for the general SPORF each atom’s element can be -1 as well (page 3). Why is this choice made despite the fact that it reduces the capacity of the model?
- the bounding box sampling seems biased as presented. That is, bounding boxes closer (than h_max and/or w_max) to the right and/or lower borders are more likely since the number of valid boxes will be lower.
====== Final Decision ======
I think it will be very interesting to bridge the gap between ConvNets and random forests since the latter comes with attractive properties. While I find all the concerns that are listed above important, my rating is mainly due to the novelty of this work compared to the prior patch-based random forest techniques for image analysis.
====== Points of improvements ======
I believe it’s important to disentangle the two main properties of convolution operation in ConvNets being shared and local parameters. Then, accordingly propose strategies to bring these properties to a decision tree.


State-of-the-art deep learning methods for outlier detection make the assumption that outliers will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result in an overall penalty if they are mapped to low-probability regions. We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than δ, and outliers to regions past this δ, δ being the inverse Chi Squared CDF evaluated at 1 − α with α the percentage of outliers in the dataset. Our experiments confirm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets.

This work proposes an outlier detection method based on WAE framework. WAE is trained to ensure that 1) latent distribution follows a prior distribution 2) weighted reconstruction error is low where prior PDF is used to weight the reconstruction error.
Positives
------------
1.I liked the intuition behind the proposed method and I felt its worth exploring. Paper points out that in previous works, there is no mechanism to prevent outliers from getting mapped to high probability areas in the model [21]. Authors claim that their method will over-come this issue. I believe this is the core contribution of the paper.
2.I agree with authors point that WAE is a better choice than VAE for outlier detection because, former "encourages the latent representations as a whole to match the prior ".
3.I agree training with a distributional divergence loss along with a weighed reconstruction loss will be helpful for learning a robust representation (as outliers in the training dataset would be assigned a lower weight).
4.Authors have compared the performance of their method on OOD dataset where they have  compared against three other baseline methods, where they obtain better performance in majority of cases.
Negatives
--------------
A. Outlier detection and anomaly/novelty detection are two very different problems.  Outliers are 'bad eggs' coming from the same class as normal data. On the other hand, anomaly/novelty are unexpected data possibly coming from other classes. This is the taxonomy followed by majority of works. In my understanding this work is about 'outlier detection'. I hope authors will use the term 'outlier detection' consistency through the paper.
B. Authors have not done a good survey on existing outlier detection methods.  Eg:
    I. Chong You, Rene Vidal, Provable Self-Representation Based Outlier Detection in a Union of Subspaces, CVPR 17
    II. Yan Xia, Xudong Cao, Fang Wen, Gang Hua, Jian Sun, Learning Discriminative Reconstructions for Unsupervised Outlier Removal, ICCV 15
   III. Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli, Adversarially Learned One-Class Classifier for Novelty Detection, CVPR 18. (they have experiments on outlier detection)
C. Although existing methods have not explicitly stated the problem identified in (1) above, their proposals are indirectly solving this problem. Therefore, authors should have compared with papers listed in (B) to demonstrate the effectiveness of their method for a meaningful comparison.
D. This is the first time I'm seeing the OOD dataset used in the paper. Have other works published their results on this dataset? Can they be included in your paper?. If not, consider reporting results on standard datasets used in papers (B). I believe reporting results on at least two datasets is necessary to demonstrate the generalizability of the method.
Other comments
------------------------
a. What is the dimensionality used in the latent space? I believe a larger latent space may be required in modeling more complex data such as images. Is the weighting mechanism effective when a very large latent space is used due to the curse of dimensionality.
b. I don't think the synthetic dataset experiment is giving any interesting insights. This space is better used if an additional dataset is used instead.
In conclusion, I like the idea presented in this paper; however, I believe experimental results needs to be improved significantly to demonstrate the effectiveness of the proposed method. I cannot recommend to accept the paper in its present condition.
Post Rebuttal:
Authors have partially addressed my concerns. In light of new experiments provided, I'm changing my decision to weak accept.


Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either retards the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes will be made public upon the publication.

The authors propose a simple but effective strategy that aims to alleviate not only overfitting, but also feature degradation (oversmoothing) in deep graph convolutional networks (GCNs).  Inspired by dropout in traditional MLPs and convnets, the authors clearly motivate their contribution in terms of alleviating both overfitting and oversmoothing, which are problems established both in previous literature as well as validated empirically by the authors. Ultimately, the authors provide solid empirical evidence that, while a bit heuristic, their method is effective at alleviating at least partially the issues of overfitting and oversmoothing.
I vote weak-accept in light of convincing empirical results, some theoretical exploration of the method's properties, but limited novelty.
Pros:
Simple, intuitive method
Draws from existing literature relating to dropout-like methods
Little computational overhead
Solid experimental justification
Some theoretical support for the method
Cons:
Method is somewhat heuristic
Mitigates, rather than solves, the issue of oversmoothing
Limited novelty (straightforward extension of dropout to graphs edges)
Unclear why dropping edges is "valid" augmentation
Followup-questions/areas for improving score:
It would be nice to have a principled way of choosing the dropout proportion; 0.8 is chosen somewhat arbitrarily by the authors (presumably because it generally performed well). There is at least a nice interpretation of choosing 0.5 for the dropout proportion in regular dropout (maximum regularization).
As brought up in the comments, edges to drop out to the graph's properties is an interesting direction to explore. While the authors state that they would like to keep the method simple and general, the method is ultimately devised as an adaptation of dropout to graphs, so exploiting graph-specific properties seems reasonable and a potential avenue to further improving performance.
p2: "First, DropEdge can be considered as a data augmentation technique" Why are these augmentations valid; why should the output of the network be invariant to these augmentations? I would like to see some justification for why the proposed random modification of the graph structure is valid; intuitively, it seems like it might make the learning problem impossible in some cases.
Deeper analysis of the (more interesting, I think) layer-independent regime would be nice. (As a side-note, the name "layer-independent" for this regime is a bit confusing, as the edges dropped out *do* depend on the layer here, whereas in the "layer dependent" regime, edges dropped out do *not* depend on the layer).
Comments:
Figure 1 could probably be re-organized to better highlight the comparison between GCNs with and without DropEdge; consolidating the content into 2 figures instead of 4 might be more easily parsable. Adding figure-specific captions and defining the x axis would also be nice.
Use "reduce" in place of "retard"
p2 " With contending the scalability" improve phrasing
p2 "By recent," -> "Recently,"
p2 "difficulty on" -> "difficulty in"
p2 " deep networks lying" -> "deep networks lies"
p3 "which is a generation of the conclusion" improve phrasing
p3 " disconnected between" -> "disconnected from"
p4 "adjacent matrix" -> "adjacency matrix"
p4 "severer " -> "more severe"
p5 "but has no help" -> "but is no help"
p5 "no touch to the adjacency matrix" -> improve phrasing


SNOW is an efficient learning method to improve training/serving throughput as well as accuracy for transfer and lifelong learning of convolutional neural networks based on knowledge subscription. SNOW selects the top-K useful intermediate feature maps for a target task from a pre-trained and frozen source model through a novel channel pooling scheme, and utilizes them in the task-specific delta model. The source model is responsible for generating a large number of generic feature maps. Meanwhile, the delta model selectively subscribes to those feature maps and fuses them with its local ones to deliver high accuracy for the target task. Since a source model takes part in both training and serving of all target tasks in an inference-only mode, one source model can serve multiple delta models, enabling significant computation sharing. The sizes of such delta models are fractional of the source model, thus SNOW also provides model-size efficiency. Our experimental results show that SNOW offers a superior balance between accuracy and training/inference speed for various image classification tasks to the existing transfer and lifelong learning practices.

*** Increased to Accept from Weak Accept after author rebuttal and changes to the paper ***
This paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning. SNOW starts with a pre-trained, frozen source model, and trains delta models for target tasks which, at each layer, concatenate a small number of task-specific features with the top-K most useful subset of features in the corresponding layer in the source model. As long as the target tasks are sufficiently related to the source task, it allows for small delta models and a small additional parameter overhead in the form of one weight per source model feature map. While there are (i) some issues with the presentation of results for training efficiency, (ii) some question marks over the sensitivity of the model to hyper-parameters, and (iii) several grammatical errors / typos in the manuscript, if these can be addressed I recommend the paper for acceptance because it seems to strike a superior balance of efficiency (regarding memory usage and inference speed) and accuracy when compared to a number of baselines, and to my knowledge it is a novel approach.
Detailed comments:
* Section 2.2 - How is sigma (the exploratory noise added for feature selection during training) chosen and how sensitive is the approach to its value? It seems like it was fine-tuned, given that a different sigma is chosen for the Action dataset (several orders of magnitude difference). In practice, tuning sigma could significantly increase training time.
* It seems like the performance of only one run was plotted per hyperparameter setting - it would be informative to see a mean and standard deviation especially since the approach seems like it could be unstable for the wrong hyperparameter settings.
* Related to the previous point, how much do the top-K feature selections change throughout training? One would have thought that this could cause instability during training for a high sigma. If sigma is too low, you could end up with suboptimal feature selection.
* Figure 4 graphs are a bit misleading because the throughput on the x-axis is reported per GPU and the larger models all need 2 or more GPUs. While this is mentioned in the main text, it is still optically deceptive and the results are GPU-dependent - presumably if the GPUs had a larger memory, the larger models would not seems as slow. I think it would be clearer to plot images/sec on the x-axis or to rerun the experiments just using a single GPU.
* It is stated that  “[d]etermining K […] has a critical impact on both size and target accuracy in the target models”, where K is the number of feature maps in the source model that the delta model subscribes to in each layer. How sensitive is the accuracy exactly? Can this be quantified or discussed in more detail?
* Furthermore, how sensitive is the performance to the number of target-model-specific features at each layer?
* Different learning rate schedules were used for SNOW and baselines - initial lr for SNOW is 1.0, while for all other models it is 0.1. Was it checked whether the baselines improve when they are run with an initial lr of 1.0? Was this hyperparameter more heavily tuned for SNOW than for the baselines?
* Since the source model is fixed, the applicability of the approach to lifelong learning is heavily dependent on the usefulness of the source model to subsequent tasks. If it is not, then one will have to incorporate large delta models. Furthermore, there can be no transfer between the tasks trained in the delta models.

Grammatical errors / suggestions:
* Page 1, first line: “hallmark” doesn’t make sense in this context - maybe “key objective” or “goal”?
* Page 1, 2nd paragraph, first line: “wee” -> “we”.
* Page 1, 2nd paragraph, line 6: “best top-K” -> either “K best” or “top K"
* Page 2, last paragraph, first line: “three folds” -> “threefold"
* Section 2.1, line 2: “pooing”->”pooling”. Same typo on Page 4, last line.
* Page 6, line 1: “training from the scratch” -> “training from scratch"
* Page 6, line 9: “more 6x than” -> “6x more than"
* Overall, the manuscript needs to be proofread a few times.


We propose a novel algorithm for learning fair representations that can simultaneously mitigate two notions of disparity among different demographic subgroups. Two key components underpinning the design of our algorithm are balanced error rate and conditional alignment of representations. We show how these two components contribute to ensuring accuracy parity and equalized false-positive and false-negative rates across groups without impacting demographic parity. Furthermore, we also demonstrate both in theory and on two real-world experiments that the proposed algorithm leads to a better utility-fairness trade-off on balanced datasets compared with existing algorithms on learning fair representations.

Update:
Thanks for providing additional results on the Adult dataset. I have increased my score. However I'd be nice to also see the balanced accuracy (i.e. sum of TPRs for each class divided by 2) results and compare to baseline trained with oversampling or re-weighted loss.
I would suggest authors to add more extensive comparisons to other methods using Adult dataset. There are quite a lot of papers in the fairness literature that experiment with the Adult dataset. They focus on different metrics and there is probably no method that is uniformly the best. Your paper demonstrated that your approach can succeed in achieving accuracy parity, but it would be good to also show tradeoffs with other metrics (in addition to DP). I was able to do some back-of-the-envelop calculations and your results seem fine, but a clear comparison would be good.
Below are some examples of the papers studying Adult dataset from the fairness angle:
[1] Mitigating unwanted biases with adversarial learning. Zhang et al., 2018. (already cited, but no comparison)
[2] What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes. Romanov et al., 2019.
[3] Learning fair predictors with Sensitive Subspace Robustness. Yurochkin et al., 2019.
---------------------------------------------------------------------------------------------------------------------------------------------------------
This paper proposes an adversarial representation learning approach. The key difference with prior work is that the objective function is built around balanced error rates, one for classes, that is eventually used for classification, and two adversarial for predicting each of the protected attributes. Authors argue that proposed approach can simultaneously achieve accuracy parity and equalized odds.
The notion of accuracy parity does not seem to be very meaningful. For example, predicting uniformly at random seems like an intuitively fair classifier with EO gap 0 and DP gap 0. However it will not necessarily have error gap of 0 (i.e. satisfy accuracy parity), making me wonder if the notion of accuracy parity makes much sense.
I am not really sure what is the Err0 + Err1 metric used in Figures 1 and 2. Is it not normalized and can vary between 0 and 2? In which case it seems counterintuitive for performance quantification. If it is normalized, then results on COMPAS do not make sense. Err0 + Err1 of all methods is above 60%, which is worse than predicting uniformly at random.
Please report your TPRs for classes grouped by protected attribute when reporting the results in the context of group fairness. Further for Adult dataset, reporting balanced TPR as a measure of accuracy seems to make more sense given the class imbalance.


Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space. These difficulties can be addressed by second-order approaches that apply a preconditioning matrix to the gradient to improve convergence. Unfortunately, such algorithms typically struggle to scale to high-dimensional problems, in part because the calculation of specific preconditioners such as the inverse Hessian or Fisher information matrix is highly expensive. We introduce first-order preconditioning (FOP), a fast, scalable approach that generalizes previous work on hypergradient descent (Almeida et al., 1998;Maclaurin et al., 2015;Baydin et al., 2017) to learn a preconditioning matrix that only makes use of first-order information. Experiments show that FOP is able to improve the performance of standard deep learning optimizers on visual classification and reinforcement learning tasks with minimal computational overhead. We also investigate the properties of the learned preconditioning matrices and perform a preliminary theoretical analysis of the algorithm.

This paper studies hypergradient descent for precondition matrices. The goal is to learn an adaptable preconditioning for the task while training. Specifically, they take the gradient of the loss wrt the precondition matrix and update the precondition matrix to decrease the loss. They reparametrize the precondition matrix to ensure it is positive-definite and provide low-rank approximations and they provide cheap approximations for CNNs.
Pros:
- Figure 3 and 4 show promising results on cifar10 with a 9-layer cnn.
- Figure 4 shows FOP can improve the accuracy for particular hyper-parameters. In cases improving by 2%.
Cons:
- Results on imagnet are not particularly good. The improvement is not significant.
- Why positive-definite precondition matrix rather than positive-semi-definite?
- Section 5: why is a degenerate precondition matrix bad? Fisher and Hessian for deep networks can be highly ill-conditioned.
- Theo 1 seems to have errors. The term M_t in the update rule should show up in the bound on P as an exponential term in the first upper bound.
- Figure 2: On mnist after 20 epochs the model has not reached 1% test error. Not clear if we can make any conclusions from this figure.
After rebuttal:
I keep my rating as weak reject. I reiterate that results look promising. However, the quality and accuracy of the writing are not acceptable for a paper on optimization. In my original review I only named a few problematic statements. I have to clarify that I do not think fixing only those few is enough.
I am also not convinced about the proof of Theorem 1. Basically, section 6 looks very much like section 5 from Baydin et al. 2018. Even the wording is mostly the same. Theorem 5.1 in Baydin et al. 2018 is based on their update rule in Eq 6 in the form of alpha_t = alpha_{t-1} - beta nabla^T nabla, where alpha does not appear in the second term. However, in this paper, the update rule on line 7 in Algorithm 1 is M_t = M_{t-1} + rho * eps *(.) M_{t-1}, where M_t appears in the second term. Hence, the first bound in Theorem 1 in this paper cannot simply be the same as in Baydin et al. 2018.


We argue that symmetry is an important consideration in addressing the problem of systematicity and investigate two forms of symmetry relevant to symbolic processes. We implement this approach in terms of convolution and show that it can be used to achieve effective generalisation in three toy problems: rule learning, composition and grammar learning.

======================================== Update after rebuttal =============================================
I have now read the author rebuttal, but my concerns about the paper remain. The training details are not described in anywhere near sufficient detail (optimizer?, batch size?, learning rate?, initialization?, etc). The baseline architectures “recurrent net” or the “multi-layer perceptron” are not described at all, despite my explicit request to that effect. I had also requested to see the source code for the experiments as this would perhaps have illuminated a lot of the details left out in the paper, but the authors have not provided it. I understand that the authors are not required to provide their code, but this should have been a relatively straightforward request in this case given the simplicity of the experiments and as I mentioned in my initial review, it would have been very useful in evaluating the paper.
In their rebuttal, the authors also claimed that the results in Fig. 1 and Table 1 are training results (that even though the architecture is "innate", the weights are learned), but I'm concerned about this claim. I happen to be doing some experiments along these lines at the moment, and it is not trivial at all to get such crisp results as those shown in Fig. 1 & Table 1 in these kinds of experiments (even when the architecture is correctly specified). Again, it would have been very helpful if the authors had either provided their source code or had described their experimental setup in sufficient detail to allow the reproduction of these results.
Given these concerns, I have decided to keep my score as it is.
========================================================================================================
This paper addresses an important problem: systematic generalization in neural networks. However, the paper is very confusing and I have some serious concerns about the models and the results presented in sections 3 and 4. Here are the main issues:
1) In section 3, there are only 10x10=100 possible combinations in this composition task. Yet, the paper says “we randomly sample 1000 such translation pairs, choose three combinations and remove all instances of them from the training data and then exclusively test on unseen pairings of command and modifier.” How can you sample 1000 pairs out of a possible 100 combinations? Also being able to generalize to 3 held-out combinations out of 100 is not very impressive. On the contrary, it is almost trivial.
2) No details are given about the “recurrent net” or the “multi-layer perceptron” baselines in section 3. What are these models? The fact that they have exactly zero accuracy is a bit suspicious, especially given the almost trivial nature of the task in section 3. Previous works reported perfect or near perfect accuracy with similar baselines in similar tasks (see e.g., Lake & Baroni, 2018).
3) I'm afraid the proposed model in section 3 also doesn’t make sense to me. It is explicitly acknowledged (Appendix B) that y is an M+1-dimensional vector, g is an Nx(M+1) matrix. Then by all accounts, the convolution of these should be an N-dimensional vector. Yet, somehow, h_t+1 in Equation 2 manages to be an NxM matrix. How? Please clarify this. If possible, making the source code available would be very helpful.
4) What is the semantics of x and y in section 3? What exactly are they supposed to be doing? This is not explained in the paper beyond a vague description.
5) Similar problems arise in section 4. The task is not explicitly described in the text. We only learn from Appendix C that it is actually to predict the next symbol. The task description mentions “strings of length 15, 17, 19, 21, 23 and 24,...” (p. 5). But, the grammar in Fig. 3 can only generate odd length strings, it cannot generate a string of length 24. Is this a typo?
6) Again in section 4, I have no idea how the proposed model is actually supposed to work. The motivation for the model and its description are not clear at all.
7) The model in section 2 is hand-coded. It is not shown that it can actually learn this solution from data. What happens if the sequences are longer or if the rules are different, for example? Then you have to hand-code a completely different architecture.
8) Which brings me to another important issue I have with this paper (and with similar papers): this whole set-up is very misguided in my mind. I think the real problem is not to come up with an architecture that would generalize systematically in a very specific (and usually toy) problem. It is to come up with an architecture that would learn to generalize systematically in a much broader set of problems. The learning aspect in sections 3-4 is a step in the right direction, but there’s no evidence that the models proposed there can learn to generalize in any task other than the very specific tasks they were designed for (if they can actually do that).


Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success. This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior. The synthetic likelihood ratio term also shows instability during training. We propose a novel objective with a "Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.

“Best of Many Samples” Distribution matching
Summary:
This paper proposes to a novel VAE-GAN hybrid which, during training, draws multiple samples from the reparameterized latent distribution for each inferred q(z|x), and only backpropagates reconstruction error for the resulting G(z) which has the lowest reconstruction. The authors appear to use the AAE method instead of analytically matching KL(p||q) for enforcing that the latents q(z|x) match the prior. The authors present results on MoG toy datasets, CIFAR-10, and CelebA, and compare  against several other models.
My take:
The idea in this paper is moderately interesting, well-founded, has plenty of precedent in the literature (while still being reasonably novel),  but the results present only a minimal improvement (a 5% relative improvement in FID over the baseline model from Rosca et al on CIFAR, especially when including SN [which is not a contribution of this paper]) and come at a substantial compute cost, requiring up to 30 extra samples per batch in order to attain this minimal increase. While I think the idea is interesting, the change in results over Rosca et. al does not seem to justify its increased computational expense (which is also not characterized in sufficient thoroughness). I am pretty borderline on this paper ( about a 5/10) but under the 1-3-6-8 scoring constraint I tend to lean reject because while I like the idea, I do not think the results are significant enough to support its adoption; I think the relative compute and implementation cost limit this method’s potential impact. I am keen to discuss this paper with the other reviewers.
Notes:
-The results on the 2D MoG toy datasets are good but are also suspect—the authors state that they use a 32-dimensional latent space, but the original code provided for VEEGAN uses a 2-dimensional latent space. The authors should re-run the experiment for BMS-VAE-GAN  using a 2D latent space (this should be very easy and take less than an hour on a GPU to get several runs in).
-“again outperforming by a significant margin (21.8 vs 22.9 FID)” This is not a significant margin; this is less than a 5% margin and, at those FID scores, represents an imperceptible change in sample quality.
-The authors seem to suggest that applying spectral norm to the GAN of Rosca et. al. is somehow a contribution (e.g. having “ours” next to this model in the tables); I would advise against even appearing to suggest this as it is clearly not a contribution.
-Characterize the increase in compute cost. “. We use as many samples during training as would fit in GPU memory so that we make the same number of forward/backward passes as other approaches and minimize the computational overhead of sampling multiple samples” is a qualitative description; I would like to see this quantitatively described. How do the runtimes differ between your baseline and the T=10 and T=30 runs? If they don’t differ, why? Are the authors e.g. reducing the batch size by a factor of 10 or 30 to make this computationally tractable?
-The latent space discriminator D_L should be referred to in section 3; its formal introduction is deferred to later in the paper, hampering the presentation and flow.
-CelebA is not multimodal; it is in fact, highly constrained, and primarily only has textural variation (virtually no pose variation).
-ALI and BiGAN are listed under Hybrid VAE-GANs. These models are not VAE-GAN hybrids. Additionally, this section states that BiGAN builds upon ALI. This is not true, these papers are in fact proposing the same thing and were released at nearly the exact same time.  Do not mischaracterize or incorrectly summarize papers. Please re-read both papers and refer to them correctly.
-Mode collapse (when many points in z map to an unexpectedly small region in G(z)) is a different phenomenon from mode dropping (when many points in x are not represented in G(z), i.e. no point in z maps to a cluster of x’s, as is the case if e.g. a celebA model generates frowning and neutral faces but no smiling faces). While these phenomena often co-occur (especially during complete training collapse), they are not the same thing, and this paper conflates them in several places.
Minor:
Section 3, paragraph 2: “The GAN (Gθ,DI…” There’s a close parenthesis missing here.
Section 3.3: “The network is traiend…”
Please thoroughly proofread your paper for typos and grammatical mistakes.


With the increasing adoption of Deep Learning (DL) models in various applications, testing DL models is vitally important. However, testing DL models is costly and expensive, especially when developers explore alternative designs of DL models and tune the hyperparameters. To reduce testing cost, we propose to use only a selected subset of testing data, which is small but representative enough for quick estimation of the performance of DL models. Our approach, called DeepReduce, adopts a two-phase strategy. At first, our approach selects testing data for the purpose of satisfying testing adequacy. Then, it selects more testing data in order to approximate the distribution between the whole testing data and the selected data leveraging relative entropy minimization. Experiments with various DL models and datasets show that our approach can reduce the whole testing data to 4.6% on average, and can reliably estimate the performance of DL models. Our approach significantly outperforms the random approach, and is more stable and reliable than the state-of-the-art approach.

The paper presents a new approach to create subsets of the testing examples that are representative of the entire test set so that the model can be tested quickly during the training and leaving the check on the full test set only at the end to validate its validity.
The key idea is to create the smaller possible subset with the same or similar coverage (in the paper the neurons coverage is considered) and output distribution, maintaining the difference below a (small) threshold.
In particular, the output distribution is approximated by dividing the output range of each neuron into K intervals. In this way, an estimate of the output distribution is considered during the extraction of the representative subset of the testing data.
The whole process is divided into two phases, the first is to create a first subset using HGS, the second refines this subset in order to achieve the desired precision in the output distribution.
The paper also presents a good experimental campaign that shows good performances.
The paper is really well written and enjoyable, clear in its description and in the objectives it aims to achieve. To improve the readability a bit further, I would suggest trying to move equation 1 after or close to its reference, or at least to describe before it what KL is.
Moreover, a similar problem is present for TI, which is introduced in Algorithm 1 and described later. I would add, for example in the input section of the algorithm, a sentence stating that TI is used in getCandidate and described later.
The first sentence on page 5 seems to be incomplete as it is written. I would suggest rephrasing the sentence.
In Algorithm 2, about the two "for" cycles for i in 1,m and foreach k in 2,K, I suggest unifying them and use the same cycle (only for reasons of readability). Moreover, I think that using braces instead of parentheses would be more correct in these cycles.
It is interesting to note that in the first line for VGG19, last column of Table 3, the accuracy is lower with KL < 0.001 than with KL < 0.005. I would expect monotonicity here. Do the authors have any idea of the reasons for this?
Finally, it would be interesting to know the runtime to obtain the subsets of the test data of Table 2 required by the considered systems.
Typos:
On page 6, penultimate paragraph, the word "the" is repeated twice in the parentheses.
On page 8, at the beginning of the first sentence after Table 3, there is a comma that seems to be useless after the word "that".
On page 8, "When the termination crite gets stricter", crite should be corrected in criterion.
There is a typo in the README of the github project linked in the paper: "coveraeg data" instead of "coverage data".


With the recent proliferation of the usage of reinforcement learning (RL) agents for solving real-world tasks, safety emerges as a necessary ingredient for their successful application. In this paper, we focus on ensuring the safety of the agent while making sure that the agent does not cause any unnecessary disruptions to its environment. The current approaches to this problem, such as manually constraining the agent or adding a safety penalty to the reward function, can introduce bad incentives. In complex domains, these approaches are simply intractable, as they require knowing apriori all the possible unsafe scenarios an agent could encounter. We propose a model-based approach to safety that allows the agent to look into the future and be aware of the future consequences of its actions. We learn the transition dynamics of the environment and generate a directed graph called the imaginative module. This graph encapsulates all possible trajectories that can be followed by the agent, allowing the agent to efficiently traverse through the imagined environment without ever taking any action in reality. A baseline state, which can either represent a safe or an unsafe state (based on whichever is easier to define) is taken as a human input, and the imaginative module is used to predict whether the current actions of the agent can cause it to end up in dangerous states in the future. Our imaginative module can be seen as a "plug-and-play" approach to ensuring safety, as it is compatible with any existing RL algorithm and any task with discrete action space. Our method induces the agent to act safely while learning to solve the task. We experimentally validate our proposal on two gridworld environments and a self-driving car simulator, demonstrating that our approach to safety visits unsafe states significantly less frequently than a baseline.

This paper presents a model-based approach to safety in RL, where the agent uses a transition model to plan ahead to avoid actions that can lead it to unsafe states. They call the planning component an imaginative module. The agent takes the baseline state as input - that can be used to define either a safe or unsafe state, that is used in the planning component. The authors claim that using these two techniques they can tackle both the safe exploration (not violating safety constraints during learning) and irreversible side-effects (unintended irreversible behavior due to poorly designed reward-function).  They validate their approach on two grid world environments and self-driving car simulators.

This paper should be rejected because of the assumptions it makes goes against the very task they are trying to solve. In the sense, the task is trivial given the assumptions they have.
1) The inconsistent assumption regarding the access to trajectories to learn a model.
The authors start with the assumption that the agent does not have access to the model (Sec 3) , and they explicitly learn the model. However, in the very next section (Sec 4), the authors assume that they can deploy a number of agents that interact with the environment randomly and collect that data to learn a complete transition model. Note that this assumption is wrong because:
If the random data agents are “safe”, i.e., don’t violate any safety constraint or cause any harmful behavior in the environment, then it is equivalent to assuming the agent having access to all the data to learn the model. This is a very big assumption that essentially says the agent has access to the model, which defeats the purpose of the safe-exploration problem.
If the random agents are “unsafe”, i.e., they can violate the safety constraint, then it goes against the very claim made about their method being able to respect the constraints throughout the learning process.
2) The assumption about the baseline state(s).
This is also a pretty big assumption to have, that is not acknowledged in the paper. If the agent already has the set of all the states it needs to avoid (or the set of states that are safe), then along with the assumption regarding access to the model, solving reversibility is significantly easier task then the general safe exploration problem [1, 2]
3) The results reported in Figure 4 are not statistically significant. The experiments are only run over 3 random seeds [3]
4) Can you give a few more details about the assumptions? In terms of how realistic they are or how essential they are to the method.
Things to improve the paper that did not impact the score:
- The negative side-effects problem that this work address is only based on reversibility criteria.
Claim about learning the dynamics model is sample efficient is unsupported.
References:
[1] Berkenkamp, Felix, et al. "Safe model-based reinforcement learning with stability guarantees." Advances in neural information processing systems. 2017.
[2] Dalal, Gal, et al. "Safe exploration in continuous action spaces." arXiv preprint arXiv:1801.08757 (2018).
[3] Henderson, Peter, et al. "Deep reinforcement learning that matters." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.


A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multiagent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.

The paper presents a new method CM3 for multi-goal multi-agent settings where agent must care about the rewards of others as well as theirs. The method restructures the problem into two-stage curriculum where individual agents solve the problem in the individual setting of the environment in the first stage and then more agents are introduced in the multi-agent setting in second stage. The method also uses function augmentation to only learn parameters which are necessary for single agent in first stage and then more parameters are introduced in the second stage. The results are shown on three environments which show that CM3 outperforms the baselines.
The paper is well-written, motivated and clear to read. A lot of important stuff has been pushed into appendix and experiments/setup require more details but overall I believe paper has significant contributions and results. Therefore, I assign a rating of weak accept which I am happy to raise if clarity of paper can be improved.
The paper doesn’t comment on parameter counts of CM3 compared with baselines which is also an important factor in choosing one method over the other. I would like to see more quantitative analysis as presented in IAC to understand what is happening behind the scenes. It is very surprising to see that IAC is outperforming COMA in most of the tasks which is opposite of what COMA paper suggested.
On the note of curriculum [1] and [2] uses curriculum in traffic junction settings to improve overall performance as agent increases. This can be compared to moving from stage 1 to stage 2 but instead we move from less number of agents to more. [2] also suggests that using individualized rewards help in better credit assignment. There is also an assumption in the setup that private observations from all other agents are readily available. It would be interesting to see what would happen if agents have to communicate their private state. So, the paper is also missing discussion on communication protocols (discrete, continuous, through critic) [1][2].
It is hard to directly compare through the charts. So, tables in Appendix E and F should be moved to the main text. The experiment section needs to be extended and the main model section needs to be decreased in the content amount.
It seems like you missed 1/n in advantage function’s equation in Section 3, Multi-agent credit assignment section. Figure 6 has been mentioned in 6 second paragraph but doesn’t appear until the last page.
[1] Sukhbaatar, Sainbayar, and Rob Fergus. "Learning multiagent communication with backpropagation." In Advances in Neural Information Processing Systems, pp. 2244-2252. 2016.
[2] Singh, Amanpreet, Tushar Jain, and Sainbayar Sukhbaatar. "Learning when to communicate at scale in multiagent cooperative and competitive tasks." arXiv preprint arXiv:1812.09755 (2018).


In this work we propose Generative Latent Flow (GLF), an algorithm for generative modeling of data distributions. GLF uses an auto-encoder to learn latent representations of the data, and a normalizing flow to map the distribution of the latent variables to that of a standard Gaussian. GLF can be seen as a variational auto-encoder, with normalizing flow prior, and a vanishing limit of the pixel-wise variance of the data. We carefully study this relationship and the pros and cons of using an auto-encoder vs. a variational auto-encoder. In contrast to a number of other auto-encoder based generative models, which use various regularizers to encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs a mapping between these two distributions, leading to better density matching while avoiding over regularizing the latent variables. We compare our model with several related techniques, and show that under standard quantitative evaluations, our method achieves state-of-the-art sample quality and diversity among AE based models on commonly used datasets and is competitive with GANs' benchmarks.

The paper proposes a deterministic Auto-Encoder with a trained marginal distribution over latent variables, p(z), to be able to sample from the model. For this purpose, the authors propose to use a flow-based model for p(z), and regularize the AE objective (i.e., MSE) with a cross-entropy between q(z) = 1/N \sum_n E(x_n) and p(z). In general, I find the idea quite interesting. The construction of the objective and motivation behind is rather clear. The experiments are rather convincing (however, the FID metric is subjective, but it's impossible to calculate the log-likelihood scores). The main disadvantage of the paper is its language. There are many typos and difficult to follow sentences. But besides that, the main ideas are well explained. Please find more specific remarks below.
Remarks
- The language in the paper is a bit off. There are sentences that sound very peculiar, e.g., "Deep generative models can be roughly classified into explicit and implicit models. The former class assumes explicit parametric specification of the distribution, whereas the latter does not."
The introduction should be re-written. Similarly, Section 2 is hard to follow.
There are places where a word or a punctuation mark is missing, for instance:
* The first line misses a word: "(...) on deep learning."
* First paragraph, Section 2: "This distribution is unknown and possibly The model also has a predefined prior distribution p(z) on Z."
* Section 3.2, below Eq. 2: "(...) assumptio (...)".
* Section 3.4, last paragraph: "Our work is differs in two ways (...)".
- The authors should refer to the following very relevant paper:
* Xu, H., Chen, W., Lai, J., Li, Z., Zhao, Y., & Pei, D. (2019). On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder. arXiv preprint arXiv:1905.13452.
- It is unclear how the authors dequantize image data that are typically represent as integers from 0 to 255 (or 0 and 1 in the binary case). The authors mention in Section 3.2 that they use the MSE loss for \mathcal{L}_{recon}. This corresponds to taking a Gaussian decoder in the VAE framework. However, the manner how the images are dequantized is extremely important to properly evaluate all results. For instance, if the authors add a uniform noise, then it highly influences the final quality of a model.
- I do not fully follow why the authors call the objective part for learning the marginal distribution over latents "the NLL loss". It is rather the cross entropy between q(z) = 1/N \sum_n E(x_n) and p(z). This follows from the notation and the description that we stop the gradient for E(.). I find it confusing.
- Figure 4 is extremely important for understanding why we should stop gradient. However, jumping between Section 4.2 and E is extremely annoying. The authors can use up to 10 pages, so adding a half of a page would not make a difference, but it would help a reader a lot.
- The main difference between the objective in Eq. 2 and the ELBO lies in considering a deterministic encoder and entails skipping the entropy term for the variational posterior. Could the authors comment why this is so important to choose a deterministic encoder? I can easily imagine taking q(z|x) = Normal(z | mu(z), a1), i.e., fixing the variance to some value a (e.g., a=1), that would result in Entropy[q] = const. Hence, we can skip entropy from the objective, but still we use a stochastic encoder.
===== AFTER REBUTTAL =====
I would like to thank the authors for their response and new version of the paper. I must admit that I had a very hard nut to crack. From one side, I really appreciate all the effort the authors put to improve the paper. However, on the other hand, I tend to agree with the reviewer #3 that the paper is interesting from the engineering perspective and it lacks novelty. Eventually, I decided to sustain my score. There are two reasons for that. First, more analysis of the model would be helpful. For instance, analyzing the latent space would be interesting (a suggestion from the reviewer #3). Second, considering a pure AE with a trainable flow-based prior is interesting, but it is also very limiting in the sense of obtaining nicely looking pictures instead of being able to provide (approximate) probability of an image. In my opinion, from the decision making perspective, having a probability of an object is much more important  than being able to generate a crisp image.


Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge , gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of  to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are label-agnostic, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

This paper aims at exploring the properties of neural network training during the early phase. By some studies on the lottery ticket hypothesis, something important happens during the early phase of training so rewinding the network should go to these early phases instead of the initial phase. So, what is important during training? The paper explores this problem from four aspects through empirical studies:
1. By showing the various statistics through different training iterations, especially the gradient magnitude, the training phase is divided into three parts, and each part has different behaviors.
2. The paper explores what is more important for the early phase of training: signs of the weights or magnitude of the weights.
3. The paper explores what is more important for the early phase of training if we redistribute the weights, signs or magnitudes.
4. The paper explores how training depends on data. Giving weak information such as self-supervised information may work but giving wrong information such as random labels will hurt the performance.
This paper studies the properties of deep neural networks. Through a series of carefully designed experiments, the paper shows what is important for the weights (magnitude or signs), and what is important for the data (weak information or random information). It enables a deep understanding of neural networks and may motivate new neural network compression methods to be proposed. Generally, the paper is well-written, although some parts can be improved. I would vote for acceptance of the paper.
Some questions/suggestions to make the paper clearer:
1. It is better to have a table summarizing various results of the paper, to give readers an overall impression after going through so many detailed experimental results.
2. The results of Fig. 4 and Fig.6 can be inconsistent. In Figure 4, it says that signs are less important than magnitudes. In Figure 6, it says that signs are more important than magnitudes if shuffling filters and keep signs. Any explanation on the inconsistency?
3. There is no explanation of the “weight trace” in Figure 3.
4. It is not clear what is the difference between “Init” and “Final” in “L2 Dist” and “Cosine Similarity” in Figure 3.
Finally,  it is said in the “call for papers”, “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages”. This paper is nearly nine pages, and higher standards should be applied.
--------------------------------------
I am satisfied with the rebuttal. Since the paper is now within the 8 pages limit, I would not apply a high standard and increase my score.


The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

This paper proposes a low-rank tensor decomposition model (Tensor Train-TT [Oseledets et al, 2011]) to parameterize the embedding matrix in Natural Language Processing (NLP). It shows that TT allows for a compression of the network and sometimes even a slight increase of test accuracy. The paper is well written and easy to follow.
I found the idea as a natural consequence of many recent papers proposing tensor decomposition to parameterize deep learning networks. However, I think this is the first time that the concept has been applied to learning an embedding matrix, which is an important problem in the field.
The authors reported several experimental results on different tasks and datasets for NLP such as: Sentiment Analysis, Neural Machine Translation and Language Modeling; and an application to the click through rate prediction problem.
I think the paper is of limited novelty but includes interesting experimental results that helps to better understand the potential and limitations of tensor decompositions in deep learning architectures.
Below, I summarize the issues I found, and I would like the authors to address them in their responses:
Major issues:
-	In Page 4 (and Appendix B), the authors show a comparison with Tensor Ring (TR) and conclude that TT marginally outperforms TR in terms of BLEU measure for a fixed number of parameters in both models. I found this comparison incomplete, weak and misleading because of the following reasons:
o	TR is a more general model including TT as a particular case when the first and last ranks are equal to one (Zhao et al, 2016). In fact, in this experiment, the authors chose all intermediate ranks set at the same value R with the first/last ranks set to 1 and R for TT and TR, respectively, which is not a fair comparison. Shown results suggest that first/last ranks contain less information than intermediate ranks, but it would be necessary to explore other combinations of rank values without keeping them constant to explore the generalization power of the TR model including TT as a particular case.
o	The authors compares TT and TR only in case of the NMT, Transformer-big on WMT‘14 English-to-German dataset, where the results are not good for TT and TR. It is noted that baseline model (Big) attains a Sacre BLEU = 28.84, TT1 = 28.53 and TR1 = 28.07 and the compression rate is only 210/179=1.17 for TT1 and TR1 and the Iteration time is larger than in the baseline model. In this case, there is no a clear advantage of using TT or TR.
In my opinion, to improve the paper, I think the authors could:
o	To avoid the sentence “In our experiments, however, the resulting TR embeddings performed slightly worse than TT–embeddings with the same number of parameters” unless more conclusive and exhaustive experiments are performed comparing TT and TR.
o	To add some comparison results between TT and TR for the rest of datasets such as Sentiment Analysis, Language Modeling and the click through rate prediction problem.
o	Highlight that TT is a particular case of TR so considering the first and last ranks equal to one reduce the number of parameters but can affect the generalization power of the model.
-	The approach of the paper is mostly intuitive. A theoretical result about why the low rank TT is able to catch the useful information of an optimal or suboptimal embedding matrix is missing.
Minor issues:
-	In last paragraph of section 3.2: The number of parameters is computed on the 3D tensor cores only. I think the size of the first/last 2D cores should be added. Please revise the equation.
-	The pseudocode for the mapping one index to multiple indices is trivial and could be avoided. If it is kept, I think the reverse operation should be also included, i.e. how to map multiple indices i1, …., iN to one index i.
-	The discussion and Figure 2 about the Gaussianity of the values in the higher order tensor based on Gaussian core tensors is not relevant. Maybe, the authors should better motivate why it is important to highlight that the distribution tends to a Gaussian density for increasing ranks.
-	In page 5, it is mentioned that “factors should be as close to each other as possible” but there is no a justification for it. Could you give some theoretical insight on why it is important to obtain uniform distribution of matrix size?
-	Section 4.1, reference to the Stanford sentiment treebank (SST) is missing.
On Nov 16th: I am satisfied with the responses provided by the Authors who made few changes to solve some identified minor issues. Thanks for taking the review report into account. I have raised the rating.


Role-Wise Data Augmentation for Knowledge Distillation
Knowledge Distillation (KD) is a common method for transferring the "knowledge" learned by one machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g., more parameters or higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -and this data is the only medium by which the teacher's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, a human teacher may demonstrate a piece of knowledge with individualized examples adapted to a particular student, for instance, in terms of her cultural background and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distillation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network. We find empirically that specially tailored data points enable the teacher's knowledge to be demonstrated more effectively to the student. We compare our approach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our results will be made publicly available.

The authors hypothesise that, in a Knowledge Distillation (KD) setting, the student network may benefit from learning from different training data than the teacher. The motivation comes from the fact that human teachers "adapt" the examples given their students, depending on their individual expertise on the subject, personal and cultural biases, and other factors.
More concretely, the paper proposes to use an evolutionary algorithm for data augmentation (PBA, already published) to train the teacher and student networks. The key aspect of the proposed method is that the teacher and the student use different augmentation schedules. The augmentation schedules improve results on both the teacher and student networks.
Orthogonally to this, they also propose to combine different KD objective functions and show that this also improves the results over each loss in isolation.
The datasets used in the experiments are CIFAR-10 and CIFAR-100. In most of the experiments the student has a (much) lower bit-width than the teacher, but they also apply the method with a student network with less parameters. Two different network architectures are used in the experiments: AlexNet and Resnet18, their method shows consistent improvements over the baselines in all cases (although some improvements may be considered marginal).
Overall the paper is well motivated, clearly written and, in the experiments section, they give enough details, and/or cite previous works that contain them, to reproduce the experiments.
Most importantly, the experiments are well designed to test the initial hypothesis:
1. The authors show that the student (and teacher) trained with the PBA data augmentation achieves a higher accuracy than the baseline method (Table 2). However, this is not enough to confirm/refute the hypothesis itself, since it is known that data augmentation generally helps.
2. They show that the two augmentation strategies are different, by using the teacher’s augmentation to train the student network (Table 3). The results show that it’s better to use a specific data augmentation schedule for the student network.
However, (the main criticism is that) the paper only partially confirms the hypothesis. For instance, it is not clear whether the hypothesis is true for other forms of KD, or it only applies when training a student with lower bit-width. In Section 5.6, they train a student with fewer parameters (less layers) than the teacher, instead of fewer bits/parameter. However, the improvements of their method over the baseline (II-KD) seem marginal there. And most importantly, as pointed out earlier, this alone is not enough to confirm the hypothesis. A similar table to Table 3 should be included in this section.
Secondly, the proposed KD loss, which is just a combination of previously published works, is an orthogonal improvement to the main method (as the authors admit), and it is not related at all to the subject of the study. It is obviously good to introduce more than one contribution in a paper, but this second (and minor) contribution should be well motivated in its own, in order to avoid "distracting" the reader from the main contribution.
In addition, I would suggest to perform statistical tests to give additional robustness to the conclusions drawn from the experiments. They perform experiments on different architectures (AlexNet and Resnet18) and different KD losses, and the conclusions are always consistent with the hypothesis, but it’s not clear whether the improvements are statistically significant or not. In this regard, it would also be appreciated to include results with other datasets, since only CIFAR-10 and CIFAR-100 (which are very similar datasets) were used.
Beyond statistical significance, it’s also not clear how important are these results for researchers working outside the scope of Knowledge Distillation.
Finally, some minor comments:
- L_original in Eq. (9) is not defined. I'm assuming it's the L_{KD}^{soft} loss.
- Please, check consistency of pronouns: authors refer to the teacher network as “it” (e.g. “[...] teacher distills knowledge to a narrow/shallow student to improve its performance”, page 2), but to the student as “she” (e.g. “[...] to train the student better from her teacher”, page 2).
- “our methods clearly outperforms [...]” (page 6)  -> “our method clearly outperforms [...]”.
Score: Borderline accept, but I will increase the score if the authors address my concerns and provide better evidence that the hypothesis is confirmed.


Multi-task learning promises to use less data, parameters, and time than training separate single-task models. But realizing these benefits in practice is challenging. In particular, it is difficult to define a suitable architecture that has enough capacity to support many tasks while not requiring excessive compute for each individual task. There are difficult trade-offs when deciding how to allocate parameters and layers across a large set of tasks. To address this, we propose a method for automatically searching over multi-task architectures that accounts for resource constraints. We define a parameterization of feature sharing strategies for effective coverage and sampling of architectures. We also present a method for quick evaluation of such architectures with feature distillation. Together these contributions allow us to quickly optimize for parameter-efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining a high level of final performance.

This paper proposes a framework for learning multi-task convolutional neural networks. For each layer of the network, the proposed algorithm assigns a subset of the layer's channels to each of the tasks. This is in contrast to existing methods that assign whole layers to tasks. There are two key ideas here: (1) instead of searching in the space of binary assignments of layers to tasks, search in the continuous space of fractions of channels assigned to each layer, subject to some consistency constraints; this allows for using finite differences for gradient estimation which can be fed into a black-box optimization procedure; (2) the use of distillation to estimate the performance of a given assignment, rather than retraining many models. Experimentally, the proposed framework performs relatively well on the Visual Decathlon benchmark.
Overall, I do like the paper's key ideas. However, I am not convinced by the presentation of the paper and by the lack of multi-task learning baselines in the experiments. I will consider changing my score if the following comments are addressed.
- Testing your main premise: "we instead partition out individual feature channels within a layer. This offers a greater degree of control over both the computation required by each task and the sharing that takes place between tasks."
Actually, based on the paper, it is not clear that assigning channels rather than layers brings about additional gains. To prove that hypothesis, you should carry out experiments comparing your method to existing multi-task learning methods that you surveyed in the paper.
- "share all" performance: this baseline seems to do quite well in Table 2, beating es in 5/10 w.r.t. the average accuracy. Why does this happen?
- Too many hacks hidden in the appendix and it is not clear what works why. Have these values been found by cross-validation? The reader should be able to understand how your method works *exactly*. Examples of these magical values:
-- Number of samples for the ES
-- Number of parameter directions
-- "Distillation training is done for a brief 3000 iterations with a batch size of 4 and a learning rate of 1 which is dropped by a factor of 10 at iteration 2000."
Clarification questions:
- I don't understand what "strategy" means here: "Fixed vs Learned Strategies: Is a uniform strategy applied across tasks or is a task-specific solution learned?"
- Section 4.2 needs to be rewritten to give a more structured exposition of the distillation process. Please add pseudocode if needed. The Figure is not very informative.
- MIP: Generally, this is a very important component of your method, yet it is lacking in detail and left to the appendix. Some questions:
-- how long does it take to solve? Which commercial solver do you use? Please provide more details.
-- constraint (8) is for which k? Is it for all k? Please fix. Also for (10), it should be for all i and j as well as k.
-- what is the "non-linear constraint" that (8-9) are linearizing? Please provide the original MIP formulation (without slack or linearization), and explain the final formulation accordingly.
-- MIP: are you adding slack variables to allow for slightly infeasible solutions in case no M can be derived from \tilde{P}?
Minor:
- "cut down average feature dramatically": unclear, rephrase as needed.
- Figure 6 is low-resolution.


There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations. In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance. Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel. Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network to other commonly used classifiers and incorporate this into our method. The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases. In fact, in a couple of cases, we even approach the complex model's performance. We also conduct further experiments to validate assertions and intuitively understand why our method works. Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.

The paper proposes a method for transferring knowledge from a complex, high-performing model to a small/lower-performing one. The approach is based on reweighting the train examples according to the ratio of confidences of the complex and simple models, and then retraining the simple model on the reweighed dataset.
Pros:
* Some interesting ideas motivating the approach
Cons:
* An important error in the main result and various other issues (imprecise statements, lack of details) raise doubts on the theoretical contribution of the paper.
* Limited novelty. Contribution boils down to importance reweighting via a ratio of probabilities.
* Limited awareness of related work.
* Experimental framework lacks meaningful baselines and important details .
Based on the points above, I have to recommend rejection of this paper to ICLR.
Detailed Comments:
* The claim that per-sample hardness reweighting is novel to this work is a bit of a stretch. Most reweighting-based methods for distillation use some form of this idea, though how each method estimates “hardness” varies, ranging from raw confidence scores of the teacher model (e.g., classic distillation), to scaled versions of it, to a ratio of this and the student’s score (this work).
* More generally, there’s quite a few very relevant related works which are not cited here (see list below for a non-exhaustive list).
* I believe the inequality used to prove Lemma 3.2 is not true in general. It’s easy to come up with counterexamples for which w>=1 and x>1 yet log(wx) <= w log(x) does not hold. For example, for w=2, any x in (1,2) will not satisfy this. In general, this inequality is only true if x >= w^{1/(w-1)}. I would be prepared to pass this off as a minor error, were it not for the fact that the w and x in the proof of Lemma 3.2 could quite conceivably be outside this range, i.e., rendering the inequality (and therefore the bound) invalid.
* There are various other technical aspects of the paper that are either non-rigorous or lack details. For example, Lemma 3.1 requires more information. For which y does w>= hold? Is it just for the true class y*? The proof is equally obscure, and seems to use the same problematic inequality of Lemma 3.2 (for the cross-entropy loss).
* The statement of Lemma 3.2 is confusing. The last sentence should read loss smaller than *that* of theta*. Also, I don’t see the optimality of theta* being used anywhere. The statement could very well be phrased in terms of any two models, one of which has a lower loss than the other.
* Assuming the complex “teacher” classifiers are very accurate (which is assumed several times throughout the paper), re-weighting training examples by max(1, p_c / p_theta) does not provide too much information beyond the ground truth labels. This is a core problem in knowledge distillation, already identified as early as Hinton et al (2015), and which is often alleviated by some form of temperature annealing or by operating on pre-softmax logits rather than final probabilities. That doesn’t seem to be the case here, so I’m quite puzzled by the “intuitive justification” provided in this work.
* The discussion in the first paragraph of pg 4 is a well known dichotomy (namely, up-weighting easy or hard examples) in the distillation/boosting/instance-reweighting literature. This discussion would really benefit from awareness of previous work discussing this duality (e.g., (Bengio et al 2009; Ren et al 2019)
* Where and how is the ordering of the “graded” classifiers used? In step (3) of Algo 1 all \zeta’s seem to be weighted equally, so I’m missing why the classifiers need to be delta-graded.
* I find it quite surprising that in almost half of the datasets in Table 1, distillation yields the exact same performance as just training the Simple Model, and sometimes even degrades the performance. Section 4.1 points out that this is actually “an equivalent model to Distillation”, but no further explanation is given, so it is hard to conjecture what might be the problem. Futhermore, as the authors point out in the introduction, Distillation alla Hinton 2015 usually assumes both teacher and student models are soft predictors (e.g., neural nets), but the models used in the UCI experiments trees and SVMs, so I wonder how exactly they did this.
* For problems involving distillation into simple interpretable models (like trees), such as the UCI experiments, there are many reasonably strong baselines from previous work that could have been compared against (e.g. Frosst and Hinton 2017), which could have provided a more meaningful evaluation.
[1] Learning to Reweight Examples for Robust Deep Learning, Ren et al. ICML 2018
[2] Distilling a Neural Network Into a Soft Decision Tree, Frosst and Hinton, 2017
[3] Born-Again Neural Networks, Furlanello et al., ICML 2018
[4] Fidelity-Weighted Learning, Deghani et al., ICLR 2018