Discriminatively trained neural classifiers can be trusted, only when the input data comes from the training distribution (in-distribution). Therefore, detecting outof-distribution (OOD) samples is very important to avoid classification errors. In the context of OOD detection for image classification, one of the recent approaches proposes training a classifier called "confident-classifier" by minimizing the standard cross-entropy loss on in-distribution samples and minimizing the KL divergence between the predictive distribution of OOD samples in the low-density "boundary" of in-distribution and the uniform distribution (maximizing the entropy of the outputs). Thus, the samples could be detected as OOD if they have low confidence or high entropy. In this paper, we analyze this setting both theoretically and experimentally. We also propose a novel algorithm to generate the "boundary" OOD samples to train a classifier with an explicit "reject" class for OOD samples. We compare our approach against several recent classifier-based OOD detectors including the confident-classifiers on MNIST and Fashion-MNIST datasets. Overall the proposed approach consistently performs better than others across most of the experiments.

** post rebuttal start **
After reading reviews and authors' response, I decided not to change my score.
I recommend to strengthen their theoretical justification or make their method scalable to improve their work.
Detailed comments:
2. "Moreover, their results on some of the gray-scale experiments are significantly worse compared to ours."
-> If you are talking about the comparison in MNIST-variants, please note that experimental results on MNIST cannot be seriously taken unless there is a strong theoretical background; especially, MNIST-variants are too small to talk about the scalability of the method. It is hard to convince readers only with results in MNIST-variants, unless the method has a strong theoretical justification.
However, if your claim is true for general gray-scale images, e.g., preprocessing CIFAR to be in gray scale, then you may add supporting experiments about it.
4. Again, if the method is only applicable to MNIST-variants due to its computational complexity while it has no strong theoretical justification, I can't find benefits from it.
** post rebuttal end **
- Summary:
This paper proposes to improve confident-classifiers for OOD detection by introducing an explicit "reject" class. Although this auxiliary reject class strategy has been explored in the literature and empirically observed that it is not better than the conventional confidence-based detection, the authors provide both theoretical and empirical justification that introducing an auxiliary reject class is indeed more effective.
- Decision and supporting arguments:
Weak reject.
1. Though the analysis is interesting, it is not applicable to both benchmark datasets and real-world cases. Including the benchmark datasets they experimented, the input to the model is in general bounded, e.g., natural images are in RGB format, which is typically normalized to be bounded in [0,1]. Therefore, the polytopes would not be stretched to the infinity in most cases.
On the other hand, note that softmax classifiers produce a high confidence if the input vector and the weight vector of a certain class are in the same direction (of course feature/weight norm also matters, but let's skip it for simplicity). Therefore, if there is an auxiliary reject class, only data in the same direction will be detected as OOD; in other words, OOD is "modeled" to be in the same direction with the weight vector of the auxiliary reject class. However, the conventional confidence-based detection does not model OOD explicitly. Since OOD is widely distributed over the data space by definition, modeling such a wide distribution would be difficult. Thus, the conventional approach makes more sense to me.
2. The experiment is conducted only on MNIST variations, so it is unclear whether their claim is true on large-scale datasets and real-world scenario.
Why don't you provide some experimental results on other datasets commonly used in other OOD detection papers, such as CIFAR, SVHN, TinyImageNet, and so on?
- Comments:
1. In section 4, the authors conjectured the reason why the performance of reject class in Lee et al. (2018a) was worse is that the generated OOD samples do not follow the in-distribution boundaries well. I think Appendix E in the Lee et al.'s paper corresponds to this reasoning, but Lee et al. actually didn't generate OOD samples but simply optimized the confidence loss with a "seen OOD." Lee et al. didn't experiment on MNIST variations but many natural image datasets. So, it is possible that the auxiliary reject class strategy is only effective in MNIST variations. I suggest the authors to do more experiments on larger datasets to avoid this criticism.


Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved biasvariance trade-offs on standard benchmark tasks.

Summary:
As the title suggests the paper focusses mostly on a negative result:
Mutual information (MI) estimators obtained by variational methods have severe
limitations that make them potentially not useful for down stream tasks.
Besides highlighting the problems with variational MI estimators the authors
suggest a modification to slightly improve the performance of MI estimators
based on partition functions by reducing their variance when MI is high.
The authors give a good overview / introduction of various approaches to
variational MI estimation by discriminative and generative methods.
Generally, MI estimation involves the estimation of the KL divergence between
the joint distribution and the product of the marginals.
The authors present a unifying view on the different approaches that optimizes
the log density ratio required for the KL divergence over the space of
log density ratios.
Discriminative approaches model the density ratio directly (through e.g. neural
network models) and generative approaches model the separate densities
(as generative models where it is possible to evaluate the (conditional)
probabilities / likelihoods of the data generating process).
The authors prove that discriminative approaches that are based on the partition
function approach suffer from high variance where mutual information is high
(Theorem 2).
The estimator based on a finite sample has high variance even if the density
ratio approximation is correct.
(The partition function approach is a way of staying constrained to the log
density ratio function space.)
This high variance problem is something that has previously been observed
empirically and is the main theoretical point that is being made about
limitations of MI estimators.
In order to slightly alleviate the problem of high variance the authors suggest
a way of biasing MI estimators by clipping the density ratio estimates
through a constant chosen as a hyper-parameter.
They prove that their clipping approach reduces variance and therefore
introduces a bias variance tradeoff.
In their later experiments the clipped version of the discriminative approach
performs much better in terms of variance than without clipping and also better
than a generative approach.
In order to empirically evaluate the quality of MI estimators the authors
suggest three criteria that they call self-consistency:
(i) independence, (ii) data processing, (iii) additivity
Self-consistency is evaluated experimentally on images where mutual information
is computed between original image and image with part covered.
The authors claim and experimentally show that discriminative approaches fail
in (iii) and generative approaches fail in (i), (ii).
Overall, variational MI approaches do not satisfy self-consistency.
Evaluation:
I suggest to accept the paper.
The theoretical contribution of showing the variance limitation of
discriminative approaches seems significant.
That insight leads to the idea that clipping can be a useful bias that
significantly reduces variance without making the already biased anyways
results much worst in the experiments.
However, I also feel like the paper is not yet as focused as it could be.
It contains many concepts that could need a little bit more space.
Suggestions:
- Page 2: Nitpick, but in the definition of $L^p(Q)$ using $\colon$ twice is
	not super readable on the first read
- Page 2: In the definition of $I_{BA}$ clearify whether $p(x)$ is a marginal
	or a joint density (as $P$ is the cumulative joint)
- Page 3: In Theorem 1 what is the definition of $P \ll Q$?
	This suggests an order on the space of measures $\mathcal P$?
- Page 3: "Obtain an density ratio estimate" -> Obtain a density ratio estimate


Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped. In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code will be made publicly available.

In this work, the authors describe a technique for compressing neural networks by learning a so-called Epitome (E), and a transformation function (\theta) such that the weights for each layer can be constructed using \theta(E). The epitome and the transformation function can be learnt jointly while optimizing the network for the task specific loss.
The main idea of this paper is really interesting, and the experimental results which compare against other recent techniques also validate the proposed technique. However, while I think the main idea is relatively clear, I personally found the description of the proposed techniques -- particularly 3.2 and 3.4 -- to be somewhat hard to follow, particularly given that some details only appear in the Appendix. I would suggest that the authors try to revise this section by trying to move Figures 4 and 5 from the appendix into the main text. Some additional suggestions also appear below.
Overall, I would while I like the ideas in this paper, based on the current presentation I am inclined to rate the paper as a “weak reject”, though I would raise my rating if the paper was revised to improve the presentation.
Main comments:
1. The authors mention that “During inference, (the) routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.” It would be nice to include some results which indicate what the savings are with the proposed routing map.
2. Section 3.2: There were a few aspects of section 3.2 that I think could be improved for clarity.
A.) Personally, I found Figure 2 somewhat tricky to follow. I would suggest removing the 3x2 “Epitome” and “Generated Kernel” figures on the extreme right of the image, since I’m assuming the only goal of these is to indicate that “orange” and “blue” correspond to “Epitome” and “Kernel” respectively. I would also suggest mentioning the correspondence of the colors as the first sentence in the caption. Finally, if possible, I would suggest adding a small description alongside the (a), (b), (c) subcaptions: e.g., (a) straightforward but non-differentiable mapping, … , (c) Generated Weigh Kernel.
B.) I believe that the authors use a separate E for each layer, and that Epitomes are not shared across layers. I may have missed this in the text, but it would be useful to clarify this explicitly again in the section.
C.) The “parameterized transformation layer” is mentioned before Equation 3. I think it would be useful to mention that this is implemented using neural networks in your work for clarity. E.g.: “To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (implemented using a neural network in this work) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”
or
“To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (See Section 3.3) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”
D.) The exact structure of the “parameterized transformation layer” wasn’t exactly clear to me. In 3.3, the authors mention that it consists of “of two convolutional layer, followed by a sigmoid function … takes the feature map of the convolutional layer as input”. Please clarify exactly what is fed in as the input to this network e.g., (input feature map: F, and the indices i,j).
3. I personally also found Section 3.4 which discusses the Computation reduction was also somewhat hard to follow. Some clarification questions: Is the memory/computation cost of the storing/creating the routing map included in the compression calculations? I think it is important to factor these costs when computing the savings achieved by the model. Also, I was unclear on what R_{cin} and R_{cout} are, and why they appear in Equation 5. Could the authors please clarify.
Minor Comments:
1. Abstract: “Traditional compression methods … all assume that network architectures
and parameters should be hardwired.” What does it mean for them to be “hardwired” in this context?
2. Abstract: “Experiments demonstrate that, … with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio.” --> “Experiments demonstrate that, … with 25% MAdd reduction, and a 2.5% Madd reduction for AutoML for Model Compression (AMC) with nearly the same compression ratio.”
------- Update after Author Response --------
I would like to thank the authors for their responses and for the updates which strengthen the paper in my view. I have updated my score accordingly.


Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a post-local SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of local SGD variants.

This paper proposes a new distributed computation technique for SGD training of deep neural networks. The proposed method is a modification of the local SGD which updates models distributed to several workers in a parallel way and synchronize the model parameters at every few epochs. The local SGD shows a nice performance but it is not robust against large mini-batch size. The proposed method is called post-local SGD that starts the local SGD after some epochs of standard mini-batch SGD. This modification makes the local SGD robust against the large mini-batch size. The authors conducted thorough experiments to investigate the performance of the proposed method. The experiments reveal that the proposed method gives better performances than the mini-batch SGD and the vanilla local SGD.
Pros:
- As far as I checked, the numerical experiments are strong. They checked several points of view. Several settings of mini-batch sizes and number of workers are compared. For more detailed points, they compared the proposed method for not only the vanilla SGD but also other optimizers such as momentum SGD. Moreover, different choices of timing of starting local SGD (t') are compared.
- The proposed method is simple and easy to implement.
Cons:
- The local SGD itself is not new and has been already proposed (indeed, the authors are also explaining this point in the paper), and theoretical investigation of local SGD has also been well exploited. This method is just a combination of normal mini-batch SGD and local SGD. In that sense, there are not a so much eye-opening idea in the proposed method.
- SGLD interpretation is instructive, but it does not explain why the "post"-local SGD outperforms the local SGD. We can intuitively guess why post-local SGD is better but it is not a rigorous theory.
- The post local SGD algorithm has more freedom than existing method, which results in more tuning parameters. There are parameters such as the batch-size before and after switching, the number of workers, the timing of switching and so on. It could be disadvantage of the proposed method.
Minor comment:
- Please give the precise meaning of the dominant eigenvalue.
- I think the setting that the data are distributed to all workers is acceptable. It is also an interesting setting from the HPC point of view (on the other hand, it would be less interesting in terms of federated learning).


New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena; we make the case that discourse and pragmatics need to take center stage in the evaluation of natural language understanding. We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 discourse-focused evaluation datasets. DiscEval can be used as supplementary training data in a multitask learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets. Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which opens a new challenge for multi-task learning.

- Overview: This work presents a benchmark for language understanding centered on understanding pragmatics and discourse, in contrast to existing NLP benchmarks which focus on mostly semantic understanding. The stated contributions are
    - The DiscEval benchmark, with some estimates of human performance
    - Baseline performance with state of the art models
    - Comparisons between commonly used supervised training objectives (NLI) and discourse-oriented objectives
- Review: This work offers a complementary evaluation benchmark for NLU systems to what currently exists in the field, and also offers compelling evidence that the current methods used do not provide good signal for learning the types of phenomena assessed in this benchmark. I recommend (weak) accept.
    - The paper would benefit a lot from having more in-depth explanation of the tasks, what the classes are, and what the classes mean. It's not always clear from the examples or the labels in Table 1 what exactly is being tested.
        - Additionally, it'd be nice to get the source of the data for each task.
    - The rough grouping of tasks in nice.
    - I understand that estimates of human performance, especially for tasks with a high number of classes, can be tricky to obtain. I do feel that they are especially important to have for this dataset. As you said, some of these tasks rely on context that isn't explicitly provided in the utterance, so I wonder if for some of the tasks, there may be insufficient context in just the provided utterances for humans to perform well.
        - Additionally, some tasks seem to have quite subjective label definitions. For example, "eloquence" and "specificity" in Persuasion; "formality" and "informativeness" in Squinky; etc. Validating that humans are consistent on these tasks seems crucial.
        - For the estimates you do have, are these expert or lay annotators? For some of these annotations, it seems like you'd need trained annotators.
    - Have you given any thought to exploitable data artifacts that may occur in these datasets that might be driving the fairly high performance on some of these tasks (e.g. Verifiability)?
    - I think it's quite interesting that fine-tuning on MNLI doesn't lead to good performance on DiscEval, as MNLI, as the authors point out, is commonly taken to be a useful pretraining task. This discrepancy gives practical weight to the authors' claim that discourse and pragmatic phenomena are not being sufficiently studied or evaluated for in current NLP research, despite the fact that these handling these phenomena will be crucial for NLP systems.
- Things to Improve + Questions + Comments
    - I would hope that most people in the NLP community would not say that language understanding is a solved problem, but I agree that putting "universal" in model names and paper titles is a reach-y thing to do.
    - Tasks
        - What is the original citation for STAC?
    - S4.3, Table 2:
        - Any reason not to try combining all four pretraining/intermediate training tasks in a single model, or at least more combinations of DiscEval with other things?
        - Could you comment on the standard deviation of the scores (per task) given that you're averaging 6 runs?
    - Table 3
        - "The best overall unsupervised result is achieved with Discovery STILT": what does unsupervised mean here? It also doesn't look like BERT+Discovery is the best in any column here.
    - Tables 4 and 5 feel like a bit of a dump of numbers. It'd be more useful to the readers to extract trends (probably using the groupings and theoretical frameworks introduced earlier). The noting of BERT+MNLI being good at predicting absence of relations is nice.
    - Typos: There are a noticeable number of typos. Here are some I noted:
        - The formatting of the task descriptions in Section 3 is a bit inconsistent and awkward.
        - Table 1: "exemple"
        - P4: "We use a custom train/dev validation split"
        - P5: "that help *realize* speech acts' intentions."
        - P6: "Prediction of discourse markers based *off* the context clauses..."


Change-point detection problem consists of discovering abrupt property changes in the generation process of time-series. Most state-of-the-art models are optimizing the power of a kernel two-sample test, with only a few assumptions on the distribution of the data. Unfortunately, because they presume the samples are distributed i.i.d., they are not able to use information about the seasonality of a time-series. In this paper, we present a novel approach -ATR-CSPD allowing the detection of changes in the seasonal pattern of a time-series. Our method uses an autoencoder together with a temporal regularization, to learn the pattern of each seasonal cycle. Using low dimensional representation of the seasonal patterns, it is possible to accurately and efficiently estimate the existence of a change point using a clustering algorithm. Through experiments on artificial and real-world data sets, we demonstrate the usefulness of the proposed method for several applications.

Summary: The paper raises an alarm that state-of-the art change-point detection methods in the ML literature do not handle important practical aspects arising in time-series modeling, namely seasonality. Indeed, methods designed to detect changing distribution under an i.i.d. setting can fail dramatically when the assumption is violated, when the change happens in the seasonal component.  The paper proposes to use an auto-encoder to find the "main pattern" within each seasonal window, and to use total variation penalty (l1-norm on the change) of the hidden state in the auto-encoder to encourage a smooth state-sequence which allow breaks.  They use k-means clustering to partition data-points, and detect a change-point if two consequent hidden states don't end up in the same cluster.
While the proposal is sensible and the paper is reasonably readable, I find the paper lacking in several respects, and recommend to reject it. My main concerns are
(a) novelty: despite the claims in the paper -- the importance of seasonality is well known and appreciated in time-series literature, and the proposal to look for changes in seasonality is fairly obvious when dealing with practical time-series. I would suggest to do a comprehensive literature search and re-evaluate the novelty of the paper.
I believe that recent ML papers e.g. kernel two-sample tests and such, focus on the i.i.d. setting and ignore seasonality (and other messy aspects of practical TS) -- as it is the more challenging statistical problem.
(b) The paper considers a setting where the time-series consists of a seasonal component and an i.i.d. component (combined additively or multiplicatively). It doesn't attempt to model any kind of stochastic dynamics -- e.g. at least a simple auto-regressive model instead of iid, and non-stationarity (trends) in the time-series. So despite aiming to look at practical time-series, the paper still considers a simplified model.
(c) The paper's presentation is often sloppy in language use, assumptions, mathematical details, and simulations and needs to be significantly improved to be considered for ICLR (or related ML conferences).
Detailed comments:
a) The references are severely lacking. There is an extensive literature in modeling time-series with seasonality and classical methods such as SARIMA (seasonal ARIMA), or exponential smoothing can track the evolution and changes in seasonal components. Various nonlinear DL-approaches to TS with seasonality have also started to appear. Once time-series is decomposed into trend, seasonal and stochastic part (using any linear or nonlinear or deep model), it is straightforward to apply anomaly detection algorithms to each component separately. Please take a look at e.g. https://anomaly.io/blog/index.html (from salesforce.com), to see practical change-point or anomaly detection in time-series in practice which does pay attention to seasonality. Also papers by Rob Hyndman pay close attention to seasonality, see e.g. https://otexts.com/fpp2/.
"Changepoint Detection in Periodic and Autocorrelated Time Series", https://journals.ametsoc.org/doi/full/10.1175/JCLI4291.1
https://cran.r-project.org/web/packages/trend/vignettes/trend.pdf  (which has a section on seasonal change-point detection)
Harvey, Koopman, Penzer, "Messy Time Series: A Unified approach", Adv. in Econometrics, Vol. 13, pp. 103-143., https://www.stat.berkeley.edu/~brill/Stat248/messyts.pdf
Perhaps there's relatively less focus on these practical details of change-point detection in recent ML literature and the focus is on the stochastic component, as it is the most challenging for prediction. The use of l1-norm of differences in time-series to detect changes is a natural idea, and has been suggested many papers e.g.in: http://eeweb.poly.edu/iselesni/lecture_notes/TV_filtering.pdf,
"Time Series Clustering using the Total Variation Distance",
Stephen Boyd's trend filtering, https://web.stanford.edu/~boyd/papers/pdf/l1_trend_filter.pdf .
While I am not aware of a specific prior work on auto-encoder with temporal smoothness for CPD, most of the main ideas are well known, and in my view the contribution is very limited in novelty.
b) You're ignoring any memory or dynamics in the stochastic component of the time-series -- e.g. allowing something like a simple AR model rather than iid would be a good step. Detecting changes in the dynamics or correlation structure (temporal or cross-sectional) would make the paper more interesting.  Something closer to switching linear dynamical systems, see for example https://arxiv.org/abs/1610.08466.
c) The presentation has many issues in language / math / simulations and needs to be improved:
1. The setting is not described clearly / formally -- are you trying to detect change-points online or offline, what assumptions are you making on the segments after removing seasonality -- are these just iid / stationary, can they include trends, outliers, e.t.c.
2. Baseline methods for detecting seasonal patterns are naive -- clearly applying methods that are not aware of seasonality will fail when there is strong seasonal components. There is one basic attempt at removing the seasonal component by averaging, and applying iid kernel CPD methods -- where it does help.  I believe doing something a bit more realistic (like doing a seasonal decomposition) will make the baselines much stronger.
3. Citation format is inconsistent with ICLR.
4. ATR-CSPD is undefined in the abstract.
5. Intro:  i, j, k notation inconsistent -- you seem to use i both for i = j*p + k, and also to refer to window id.
6. What is a "generative function" of time-series? Do you mean the pdf / cdf? What do you mean by a product of generative functions (which is additive or multiplicative), do you mean adding / taking products of random variables coming from independent distributions? What do you mean that you do not differentiate between additive / multiplicative? Do you claim to handle both within the same model?
7. Definition 2 -- do you look for x_jo,k ~ Gk',  or x_j for j> j0 ~ Gk'?
8. You claim a multi-variate extension is easy -- but is it? How would you tackle e.g. changes in correlation structure?
9. "Autoencoders attempt to copy input to output" - isn't this trivial by using an identity function? You should mention some compression / bottleneck as well.
10. How do you optimize the total-variation (l1-norm) penalty in your formulation? Just throw it into SGD in keras?
11.  The discussion in 3.2. is confusing -- you talk about weekly series, but use daily-seasonality, however you then describe detecting weekdays vs. weekends? How can you associate separate weekends without a weakly seasonal model?
12. London electricity data-set -- why do you average all weeks within the time-series to find average customer week? This was very surprising. Don't you loose most of the interesting anomaly data this way?
13. Figures are not explained well. While there's nice use of color -- it's often hard to understand what is the description pointing at.
typos: Person -> Pearson,  Autencoder -> Autoencoder, and many others.


In artificial neural networks trained with gradient descent, the weights used for processing stimuli are also used during backward passes to calculate gradients. For the real brain to approximate gradients, gradient information would have to be propagated separately, such that one set of synaptic weights is used for processing and another set is used for backward passes. This produces the so-called "weight transport problem" for biological models of learning, where the backward weights used to calculate gradients need to mirror the forward weights used to process stimuli. This weight transport problem has been considered so hard that popular proposals for biological learning assume that the backward weights are simply random, as in the feedback alignment algorithm. However, such random weights do not appear to work well for large networks. Here we show how the discontinuity introduced in a spiking system can lead to a solution to this problem. The resulting algorithm is a special case of an estimator used for causal inference in econometrics, regression discontinuity design. We show empirically that this algorithm rapidly makes the backward weights approximate the forward weights. As the backward weights become correct, this improves learning performance over feedback alignment on tasks such as Fashion-MNIST, SVHN, CIFAR-10 and VOC. Our results demonstrate that a simple learning rule in a spiking network can allow neurons to produce the right backward connections and thus solve the weight transport problem.

The paper introduces a training mechanism for spiking neural nets that employs a causal inference technique, called RDD, for adjustment of backward spiking weights. This technique induces the backward influence strengths to be reciprocal to the forward ones, bringing desirable symmetry properties.
Pros:
 * The relationship between causal inference and biologically plausible learning is very interesting. This relationship is also important and impactful for the machine learning community, as we are on the quest of new deep learning technologies.
 * Application of the RDD method to spiking neural net training is novel. The reciprocal relationship of the causal effect to the synaptic strength is a very intuitive and elegant solution to the weight transport problem.
Cons:
 * From the reported results, it is not possible to decide whether RDD really outperforms Feedback Alignment (FA). The comparison is performed on only two data sets and each algorithm is better on one. Could the authors report results on at least two more data sets (however small or simple) during the rebuttal?
 * Fig and Table 1 report the same outcome. One of the two need to be removed.
Further Questions:
 * The Conv Net illustrated in Fig 2 panel A shares its weights with the biologically plausible net on panel B. Further, these two nets communicate for pre-training. How does the paper then isolate the contribution of the biologically plausible net to the prediction accuracy from the vanilla ConvNet? What would happen if we trained only the LIF net without a contact with the conv net?
 * Eq. 1 proposes induction of symmetry to solve the weight transform. At the extreme, this regularizer would make W and Y identical, boiling down to  a vanilla artificial neural net, which the ML community already knows wella nd performs with excellence. Would not having the biologically  implausible artificial neural model as the extreme solution contradict with the goal of biologically plausible learning? This would in the end make one conclude that the biological brain only performs a broken gradient descent.
Overall, this is a decent piece of work with some potential. My initial vote is a weak reject, as I  am at present missing sufficient evidence that the improved symmetry properties introduced by the causal inference scheme also brings an accuracy improvement over the vanilla feedback alignment method. I am open to improve to an accept if this evidence is provided and my aforementioned concerns primarily on the role of ConvNet are properly addressed during rebuttal.
--
Post-rebuttal: My only major concern was the lack of sufficient empirical evidence to support the idea. The updated version of the manuscript has properly addressed this issue by reporting results on additional data sets. The authors have also given enlightening clarifications to some of the open points I have raised earlier. Hence, I'm happy to increase my score.


We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyperparameters.

I have read the authors' rebuttal and satisfied with their response. Novelty is a little on the lower side, but thorough writing, results, and insightful comparisons make up for this in my opinion. I have updated my score to 8: Accept.
=====
This paper proposes an approach to perform image translation called U-GAT-IT. In image translation, the goal is to learn a mapping from images in a source domain to corresponding images in a target domain. Contemporary image translation approaches are able to transfer local texture but struggle to handle shape transfer. To address this concern, the authors introduce an attention mechanism based on CAM [1] and an adaptive normalization layer into a GAN-based image translation framework. Results indicate favorable quantitative and qualitative performance relative to a number of baselines.
Specific contributions include:
* Introduction of a normalization layer called AdaLIN that can interpolate between instance normalization and layer normalization based on the input.
* Introduction of an attention mechanism based on CAM [1] that allows the model to focus on specific parts of the image when either generating or discriminating.
* Collection and release of a selfie-to-anime dataset.
* Release of U-GAT-IT code.

In my opinion this paper is borderline, leaning towards weak accept. The experiments are thorough and the paper is well-written. I have concerns about the novelty and significance of the work, but overall the paper feels very close to being a finished piece of work in spite of its (relatively minor) flaws.

Strong points of this work include the writing and experiments. The paper is clearly organized and feels polished. It cites many relevant works, giving the reader a sense of the contemporary approaches for image translation. There is a thorough description of model architecture, dataset and tuning parameters in the appendix. In addition, code and the selfie-to-anime dataset have been released by the authors. In terms of experiments, the authors provide many qualitative visualizations comparing the proposed model to baselines on various datasets. Quantitative evaluation includes KID and a perceptual evaluation on human subjects.
Weak points include novelty and significance. The proposed approach combines two ideas already applied to image translation (adaptive normalization [3] and attention [4]).  It therefore synthesizes these ideas into an effective algorithm rather than directly adding something new. It is unclear to me how others can build on top of this work to further advance state-of-the-art in image translation. Are more sophisticated normalization and attention mechanisms truly the key to improving image translation in the future?
Specific comments:
* The formulation of AdaLIN in Equation (1) is vague. The text states "parameters are dynamically computed by a fully connected layer from the attention map", but it's not clear what those parameters are in the equation. Explicitly writing \gamma and \beta as functions of the fully-connected layer and \mu_I, \sigma_I, \mu_L, \sigma_L as the corresponding mean and standard deviation expressions would make things more clear.
* The motivation for using layer normalization was discussed in 2.1.1 but I still do not understand why it is beneficial.
* The term "importance weights" has a specific meaning in the context of Monte Carlo methods. I would suggest choosing a different term here.
Questions for the authors:
* How does U-GAT-IT compare to TransGaGa [2]? One of the stated goals of U-GAT-IT is to better handle shape when performing image translation. TransGaGa has a similar motivation and so I would have liked to see an experimental comparison or at the very least a description of how U-GAT-IT differs. What sorts of shape transfer could U-GAT-IT handle that TransGaGa couldn't and vice versa?
* What are the shortcomings of the model and how could they possibly be addressed?

[1] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A., 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2921-2929).
[2] Wu, W., Cao, K., Li, C., Qian, C. and Loy, C.C., 2019. Transgaga: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8012-8021).
[3] Huang, X. and Belongie, S., 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1501-1510).
[4] Mejjati, Y.A., Richardt, C., Tompkin, J., Cosker, D. and Kim, K.I., 2018. Unsupervised attention-guided image-to-image translation. In Advances in Neural Information Processing Systems (pp. 3693-3703).


This work provides the first unifying theoretical framework for node (positional) embeddings and structural graph representations, bridging methods like matrix factorization and graph neural networks. Using invariant theory, we show that the relationship between structural representations and node embeddings is analogous to that of a distribution and its samples. We prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa. We also show that the concept of transductive and inductive learning is unrelated to node embeddings and graph representations, clearing another source of confusion in the literature. Finally, we introduce new practical guidelines to generating and using node embeddings, which fixes significant shortcomings of standard operating procedures used today.

This paper tries to clarify some confounding concepts around graph/node representation learning by providing a unifying theoretical framework based on invariant theory. The authors define node embedding and structural graph representation based on a minimal number of requirements, which is, in turn, used to derive some useful properties and unify two seemly different concepts. Based on these, a new graph neural network framework is proposed. The experiments on multiple tasks with multiple datasets validate the main claim that a single node embedding is insufficient to capture the structural representation of a graph.
Overall, this paper tries to suggest a solution to a somewhat confusing and controversial problem that everyone recognises but hard to figure out a clear resolution. In that sense, this paper would serve as a good starting point by providing some theoretical baselines. I would like to see more discussion on this topic in future.
In many graph problems where each node is endowed with a certain feature set, we often observe the case where two subsets of nodes are isomorphic in terms of link structure but with different edge of node features. It would be good if there are some discussions on these cases.
Here's a minor comment on the representation of the paper. Although the concept of node embedding and structural representations get clearer as reading, the introduction, where I couldn't find any reference of these, seems unlikely to clarify the difference between these two concepts. The familiar representation in Section 3 might be the first part where the readers could get some intuition about their differences.


Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and structural (syntactic) task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting.

Summary:
=========
This paper aims to disentangle semantics and syntax in contextualized word representations. The main idea is to learn a transformation of the contexualized representations that will make two word representations to be more similar if they appear in the same syntactic context, but less similar if they appear in different syntactic contexts. The efficacy of this transformation is evaluated through cluster analysis, showing that words better organize in syntactic clusters after the transformation, and through low-resource dependency parsing.
The paper presents a simple approach to transform word representations and expose their syntactic information. The experiments are mostly convincing. I would like to see better motivation, more engagement with a wider range of related work, and more thorough quantitative evaluations. Another important question to address is also what kind of semantic/syntactic types of information are targeted, and how to handle the tradeoff between them, for instance for different purposes.
Main comments:
==============
1. Motivation: I found the motivation for the problem understudied a bit lacking. The main motivation seems to be to disentangle semantic and syntactic information. But why should we care about that? Beyond reference to disentangling in computer vision, some more motivation would be good. The few-shot parsing is a good such motivation, although the results are a bit disappointing (see more on this below). Another possible motivation is potential applications of disentanglement in language generation. There is a line of work on style transfer also in language generation, and it seems plausible that the methodology could be applied to such tasks.
2. The present work is well-differentiated from work on extracting syntactic information from word representations via supervised ways, as the current work does so in an unsupervised way. I don't quite get the terminological differentiation between "mapping" and "extracting" in the introduction, but the idea is clear.
3. Have you considered alternative representations of word pairs besides the different of their transformations f(x)-f(y)?
4. I found it interesting that the word representation from BERT is the concatenation of layer 16 with the mean of all the other layers. This is motivated by Hewitt and Manning's findings, and [5] found similar results. However, the different between layer 16 and others is not that large as to warrant emphasizing it so much. Perhaps a scalar mix with fine-tuning may work better, as in [5], or another method. Have you tried other word representations? I also wonder whether it makes sense to use different layers for different parts of the triplet loss, depending on whether to emphasize syntactic vs. semantic similarity.
5. The introduction lays out connections to some related work, but leaves several relevant pieces missing. See examples below.
6. The results in 3.3 are limited but useful. The comparison with a PCA-ed and reduced representation is well thought of, because of the risk with low-resource and high dimensionality. That said, I found the gap between the proposed syntax model and the ELMo-reduced disappointingly small. Even in the LAS, it seems like the difference is very small, ~0.5, although it's hard to tell from the figure. Providing the actual numbers and a measure of statistical significance would be helpful here.
7. Some care should be taken to define what kind of semantics is targeted here. In several cases this is "lexical semantics", but then we have "meaning" in parentheses sometimes (end of intro). Obviously, there's much more to semantics and meaning that the lexical semantics, so a short discussion of how the work views other, say compositional semantics, would be good.
Other comments:
===============
1. The introduction seeks a representation that will ignore the similarity between "syrup" in (2) and (4). I wonder if "ignoring" is too strong. One may not want to lose all lexical semantic information. Moreover, the proposed triplet loss does not guarantee that information is ignored (and justly so, in my opinion).
2. In the example, "maple" and "neural" are said to be syntactically similar, although "maple syrup" is a noun compound while "neural networks" is an adjective-noun. Shouldn't they be treated differently then? Unless the notion of syntax is more narrow and just looks at unlabeled dependency arcs.
3. Some experimental choices are left unexplained, such as k=6 (section 2.1) or mapping to 27 dims (section 2.3); these two seem potentially important.
4. Section 2.3: do you also back-prop back into the BERT/ELMo model weights?
5. The dataset statistics in section 3 do not match those in section 2.2. Please clarify.
6. The qualitative cluster analysis via t-SNE (3.1) is compelling. It could be made stronger by reporting quantitative clustering statistics such as cluster purity before and after transformation.
7. In the examples showin in 3.1, it would be good to give also the nearest neighbor before the transformation for comparison.
8. The quantitative results in 3.2 convey the point convincingly. It's good to see also the lexical match measure going down. The random baseline is also a good sanity check to have. It would be good to provide full results with BERT, at least in the appendix and at least for section 3.2, maybe also for 3.3.
9. More related work:
+ Work that injects syntactic information into word representations in a supervised way, such as [1,2]
+ Work that shows that word embeddings contain different kinds of information (syntactic/semantic), and propose simle linear transformations to uncover them.
+ Engaging with the literature on style transfer in language generation would be good, as mentioned above for motivation, but also to situate this work w.r.t to related style transfer work.
+ Another line of work that may be mentioned is the variety of papers trying to extract syntactic information from contextualized word representations, such as constructing trees from attention weights. There were a few such papers in BlackboxNLP 2018 and 2019.
Typos, phrasing, formatting, etc.:
============================
- Abstract: a various of semantic... task -> various semantic... tasks; use metric-learning approach -> use a metric-learning approach; in few-shot parsing setting -> in a few-shot parsing setting
- Wilcox et al. does not have a year
- Introduction: few-shots parsing -> few-shot parsing
- Method: extract vectors -> extracts vectors; Operativly -> Operatively
- Section 3: should encourages -> should encourage; a few-shots settings -> a few-shot setting
- 3.2: -- was not rendered properly
- 3.3: matrix that reduce -> reduces
References
==========
[1] Levy and Goldberg. 2014. Dependency-Based Word Embeddings
[2] Bansal et al. 2014. Tailoring Continuous Word Representations for Dependency Parsing
[3] Artetxe et al. 2018. Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
[4] Tenney et al. 2019. BERT Rediscovers the Classical NLP Pipeline
[5] Liu et al. 2019. Linguistic Knowledge and Transferability of Contextual Representations


In this paper, we aim at tackling a general issue in NLP tasks where some of the negative examples are highly similar to the positive examples, i.e., hard-negative examples). We propose the distant supervision as a regularizer (DSReg) approach to tackle this issue. We convert the original task to a multi-task learning problem, in which we first utilize the idea of distant supervision to retrieve hard-negative examples. The obtained hard-negative examples are then used as a regularizer, and we jointly optimize the original target objective of distinguishing positive examples from negative examples along with the auxiliary task objective of distinguishing soften positive examples (comprised of positive examples and hard-negative examples) from easy-negative examples. In the neural context, this can be done by feeding the final token representations to different output layers. Using this unbelievably simple strategy, we improve the performance of a range of different NLP tasks, including text classification, sequence labeling and reading comprehension.

This paper proposes to improve performance of NLP tasks by focusing on negative examples that are similar to positive examples (e.g. hard negatives). This is achieved by regularizing the model using extra output classifiers trained to classify examples into up to three classes: positive, negative-easy, and negative-hard. Since those labels are not provided in the original data, examples are classified using heuristics (e.g. negative examples that contain a lot of features predictive of a positive class will be considered as hard-negative examples), which are used to provide distant supervision. This general approach is evaluated on phrase classification tasks, one information extraction task, and one MRCQA task.
Although the proposed approach is interesting, this paper has several weaknesses (i) the method is not sufficiently justified or analyzed; (ii) there are missing links with previous work (notably on domain adversarial training); (iii) experimental setting is rather weak.
1) About the justification of the approach:
1.1) I feel like the proposed are not intuitively justified enough. They point out that "L_2 can be thought as an objective to capture the shared features in positive examples and hard-negative examples". Why would that be good from an intuitive perspective ?
1.2) L_3 is forcing the model to group all the hard-negative examples together. Do you have an intuition why would that be useful ?
1.3) What happens if the model overfits the hard negative examples in the training set ? This would mean that it has captured some features that can distinguish positive / negative label. Why would L_3 help in that case ?
2) About related approaches:
2.1) How does this method relate to domain adversarial training applied to positive and hard-negatives and adversarial examples in general ?
2.2) Would similar performance be obtained by virtual adversarial training for example ?
3) About the experimental setting:
3.1) The performance reported is well below the use of recent work on these datasets and recent models such as BERT. Would these improvements carry over to bigger architectures ?
3.2) In SST, the paper say they use BERT large, but the baseline performance (81.5) is well below BERT large performance in the original paper (94.9, https://arxiv.org/pdf/1810.04805.pdf). Why the mismatch ?
3.3) What's the proportion of hard-negative examples mined for training and test set ? While the heuristics used seem reasonable, without those numbers, it is impossible to know if the heuristics truly predict hard-negative examples.
3.4) Does the performance gain comes from better predicting hard-negative examples in the test set ? One could analyze the performance per error type (i.e. true positive, false negative, false positive (easy), false positive (hard)) with the baseline model and of the various proposed regularizing tasks (e.g. L_2 and L_3, both in training and test).
3.5) The heuristics are used to pick what should be adversarial examples, but there is no mentions of this concept in the text. Oversampling those adversarial examples could, potentially, improve the performance of the baseline model. It would be interesting to try this.
3.6) If possible, it would be good to add standard deviation to the results obtained running multiple runs.
3.7) The visualization sub-section is anecdotal and not especially illuminating, and its text seems to refer to a different example than the figures ("interesting" is not in the figures.)
Minor points:
- Section 3 (Models) may be made shorter, the models used are utterly simple. This could free up space for more experiments.
- In the tables, simply adding the name of the used model to the "L1" rows would be clearer.
- The description of the pipelined results in Section 4.1 does not match the results shown in the table.
- The citations are not well integrated with the text (\citep vs \cite), and the formatting of CRF changes in the last paragraph of Section 3.2.


In this work, we present WaveFlow, a small-footprint generative flow for raw audio, which is trained with maximum likelihood without density distillation and auxiliary losses as used in Parallel WaveNet. It provides a unified view of flowbased models for raw audio, including autoregressive flow (e.g., WaveNet) and bipartite flow (e.g., WaveGlow) as special cases. We systematically study these likelihood-based generative models for raw waveforms in terms of test likelihood and speech fidelity. We demonstrate that WaveFlow can synthesize high-fidelity speech and obtain comparable likelihood as WaveNet, while only requiring a few sequential steps to generate very long waveforms. In particular, our small-footprint WaveFlow has 5.91M parameters and can generate 22.05kHz high-fidelity speech 42.6× faster than real-time on a GPU without engineered inference kernels.

## Updated review
I have read the rebuttal. The new version of the paper is definitely clearer, especially the contribution section and the experimental results. The new version addresses all my concerns, hence I am upgrading my rating to Accept.
## Original review
This paper presents the WaveGlow model, a generative model for raw audio. The model is based on a 2D-matrix approach, which allows to generate the audio with a fixed amount of step. The model is shown to be a generalization of the two main approaches for raw audio generation, autoregressive flow and bipartite flow. The model is evaluated and compared with related work on an objective evaluation (Log-likelihood) and a subjective evaluation (MOS), and is shown to be a trade-off between memory footprint, generation speed and quality.
I think this paper should be accepted, for the following reasons:
- The theoretical framework presented is novel and significant, as it provides a unified view of the two main approaches for neural waveform generation.
- The experiments are reasonably convincing, although they could be improved.
Detailed comments:
- In the subjective evaluation section (5.2), Table 5 is hard to decipher, especially given that there are three measurements to take into account, so it's not easy to see the benefit of the approach. Maybe the results should be organised differently, for instance grouping them according to one measurement could help, typically showing what speed and MOS each of the three models can achieve for a given model size. Maybe plotting speed vs MOS for the same model size could also be interesting.
- In the same section, is the WaveNet model the original one, or the Parallel WaveNet ? if it's the original, why not include Parallel WaveNet in the table ?
- Typo at the end of Section 1: "We orgnize" -> "organize"


Deep reinforcement learning has successfully solved a number of challenging control tasks. However, learned policies typically have difficulty generalizing to novel environments. We propose an algorithm for learning programmatic state machine policies that can capture repeating behaviors. By doing so, they have the ability to generalize to instances requiring an arbitrary number of repetitions, a property we call inductive generalization. However, state machine policies are hard to learn since they consist of a combination of continuous and discrete structure. We propose a learning framework called adaptive teaching, which learns a state machine policy by imitating a teacher; in contrast to traditional imitation learning, our teacher adaptively updates itself based on the structure of the student. We show how our algorithm can be used to learn policies that inductively generalize to novel environments, whereas traditional neural network policies fail to do so.

# Summary
This paper proposes a technique for synthesis of state machine policies for a simple continuous agent, with a goal of
them being generalizable to out-of-distribution test conditions. The agent is modeled as a state machine with constant
or proportional actions applied in each node (regime), and switching triggers between the regimes represented as
length-2 boolean conditions on the observations. The technique is evaluated on 7 classic control environments, and found
to outperform pure-RL baselines under "test" conditions in most of them.
# Review
I am not an expert in RL-based control, although I'm familiar with the recent literature that applies formal methods to
these domains. I find the studied settings valuable albeit fairly limited, but the paper's method undeniably shows
noticeable improvement on these settings. Inductive generalization is an important problem, and the authors' approach of
limiting the agent structure to a particular class of state-machine policies is a reasonable solution strategy.
That said, the complexity of synthesizing a state machine policy clearly caused the authors to limit their supported
action and condition spaces considerably (Figure 6). That, I'm assuming, limits the set of applicable control
environments where optimization is still feasible. The authors don't provide any analysis of complexity or empirical
runtime of the optimization process. Breaking it down for each benchmark would allow me to appreciate the optimization
framework in Section 4 much more. As it stands, Section 4 describes a complex optimization process with many moving
parts, some of which are approximated (q* and p(τ|π,x₀)) or computed via EM iteration until convergence (π*). It is hard
to appreciate all this complexity without knowing where the challenges manifest on specific examples.
Section 4.2 needs an example, to link it to the introductory example in Figure 1. The "loop-free" policies of the
teacher are, in programmatic terms, _traces_ of the desired state machine execution (if I understand correctly), but
this is not obvious from just the formal definition.
The EM optimization for the student policy makes significant assumptions on the action/condition grammars. Namely, the
algorithm iterates over every possible discrete "sketch" of every program, and then numerically optimizes its continuous
parameters (Appendix A). When the action/condition grammars grow, the number of possible programs there increases
combinatorially. Is there a way to adapt the optimization process to handle more complex grammars, possibly with
decomposition of the problem following the program structure?
Section 5 needs a bit more details on the Direct-Opt baseline. It's unclear how the whole state machine policy (which
includes both discrete and continuous parts) is learned end-to-end via numerical optimization. Granted, the baseline
performs terribly, but would be great to understand how it models the learning problem in order to appreciate why it's
terrible.
Why were the "Acrobot" and "Mountain car" benchmarks removed from the main presentation of results?


Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computational limitations become even more cumbersome. We consider robust transfer learning, in which we transfer not only performance but also robustness from a source model to a target domain. We start by observing that robust networks contain robust feature extractors. By training classifiers on top of these feature extractors, we produce new models that inherit the robustness of their parent networks. We then consider the case of "fine tuning" a network by re-training end-to-end in the target domain. When using lifelong learning strategies, this process preserves the robustness of the source network while achieving high accuracy. By using such strategies, it is possible to produce accurate and robust models with little data, and without the cost of adversarial training. Additionally, we can improve the generalization of adversarially trained models, while maintaining their robustness.

Paper summary: This paper explores the problem of robustly transfer learning using only standard training (as opposed to adversarial training (AT)) on the target domain. The authors start by highlighting that intermediate representations learned by adversarially trained networks are themselves fairly robust. Then they propose two strategies for robust transfer from a robust model trained on the source domain: (1) naturally fine-tuning the final linear layer on the target domain and (2) naturally fine-tuning all the layers using lifelong learning strategies. They study transfer between CIFAR10 and CIFAR100, as well as, from ImageNet to CIFAR10/100.
High-level comments: Overall, I find the paper interesting and well-written. Prior work from Hendrycks et al. showed that AT on the source domain followed by *adversarial* fine-tuning on the target domain attains better performance as compared to just AT on the target domain. The main contribution of this paper is to show that using instead careful *natural* fine-tuning on the target domain is sufficient to recover a reasonable amount of this robustness.
Even though the clean/robust accuracy of the proposed approach is lower than just doing adversarial training/prior work from Hendrycks et al., I feel this paper could be useful to the community for two main reasons:
1. The authors perform a nice exploration of the thesis that robust models have robust representations, and how this connects to transfer learning. In particular, the experiments in Figure 1 (the effect of naturally re-training later layers of a robust network on its robustness) and Figure 5 (where the authors show that their naturally fine-tuned models have some of the unexpected benefits from Tsipras et al.) seem particularly interesting.
2. Despite its lower accuracy, this approach could be useful in settings where data is scarce or compute is expensive, and hence adversarial training on the target domain is not successful.
Specific comments/questions:
i. Could the authors clarify what they mean by point 3 (re: validation accuracy drop) below Table 3? As far as I can tell, the drop in clean validation accuracy between Table 1 and Table 2 are similar for both the naturally and adversarially pre-trained models.
ii. In Table 2, it would also be interesting to see the performance when the source domain is CIFAR10 and the target domain is CIFAR100.
iii. For Table 2 is the eps=8? This should be mentioned in the caption as it is important to highlight that Tables 2 and 3 are not directly comparable.
iv. Why isn’t experiment in Figure 3 should be repeated for CIFAR10 as well? The authors should add this result to the paper, even if in the appendix. It is important to verify that this trend is not specific to CIFAR100 and holds across datasets (even though CIFAR10/100 are not too different).
v. The comment at the end of page 6 re: natural model is confusing (“Note, this seems to...perfectly”)---as far as I can tell, Figure 4 does not include the results of fine-tuning a naturally trained model.
vi. General comment motivated by the comment ("Note...perfectly") mentioned 5 above: For all the adversarial evaluation in the paper, the authors should also try CW attacks/black-box attacks to get a more confident estimate of their model’s robustness.
vii. The authors reference prior work on the tradeoff between robustness and accuracy and motivate Section 6 as an avenue to alleviate this trade-off for their model. However, I don’t see the lower performance of their model as an instance of this trade-off---the model in the paper performs worse in terms of both clean and adversarial accuracy. The approach proposed in Section 6 seems interesting, but more as an approach to improve the *overall* performance of the model. The authors mention this in retrospect, but I think the narrative of this section should be modified to make this clearer.
viii. In Table 5, in the experiments corresponding to CIFAR100+ -> CIFAR100, is the dataset split into two halves (for the source and target domains) or is the fine-tuning performed on the same data. In general, I find it odd that natural fine-tuning on the *same data* can improve both the clean and adversarial accuracy of the model (compared to the CIFAR100+ robust baseline). Is the robust model trained long enough/with enough hyperparameter search?
Overall, the exploration in the paper seems novel and could be useful to the community. Thus, I recommend acceptance.


While neural network models achieve impressive classification accuracy across different tasks, they can suffer from poor calibration of their probabilistic predictions. A Bayesian perspective has recently suggested that dropout, a regularization strategy popularly used during training, can be employed to obtain better probabilistic predictions at test time (Gal & Ghahramani, 2016a). However, empirical results so far have not been encouraging, particularly with convolutional networks. In this paper, through the lens of ensemble learning, we associate this unsatisfactory performance with the correlation between the models sampled with dropout. Motivated by this, we explore the use of various structured dropout techniques to promote model diversity and improve the quality of probabilistic predictions. We also propose an omnibus dropout strategy that combines various structured dropout methods. Using the SVHN, CIFAR-10 and CIFAR-100 datasets, we empirically demonstrate the superior performance of omnibus dropout relative to several widely used strong baselines in addition to regular dropout. Lastly, we show the merit of omnibus dropout in a Bayesian active learning application.

PAPER SUMMARY: The paper argues that (ensembles of models with different types of dropout applied to each model) perform better than (ensembles in which the same type of dropout is applied to each model). They attribute this to increasing model diversity in the former case, and experimentally validate their claims.
MAJOR COMMENTS:
1. Motivation Unclear:
- Notational issues in (2): MSE(h_t | x) involves y, but y is not specified in the definition. As a result, later E_{x}[…] is evaluated disregarding the dependence on y, which is ambiguous.
- Derivation of the second equality in (2) is not obvious, and needs a detailed proof. Notational issues exist in switching between H and h.
- The above is used to argue that “the more diverse the models, the better performance achieved”. It is unclear how this follows from (2).
2. Unclear / Imprecise Writing
- Before Sec. 3.4:  “This is because higher Dropout rates lead to smaller effective network capacities” Needs reference.
- “Dropout uncertainty can be obtained sequentially”. Unclear what this means.
3. The entire proposed method is described in one line, “we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods”. It is very unclear as to what the authors mean by combination.
4. Experiments
- It seems that Omnibus dropout leads to a moderate “diversity”, and improves ensemble performance only for SVHN, out of the three datasets tested in Figure 4. For SVHN, the improvement is 0.1% accuracy, which is within the standard error (0.1). Hence, it seems that the proposed method provides no performance improvement.
- A Similar statement is true for the other metrics (NLL, Brier Score, ECE).
Score [Scale of 1-10]:
3: Reject
The paper needs either more convincing experiments demonstrating the claims or theoretical analysis explaining the behavior for small networks. The paper needs to be rewritten to improve presentation, and state motivation, problem statement, and contributions clearly.


MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years. New applications with stringent real-time requirements in highly constrained devices require further compression of MobileNets-like already computeefficient networks. Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4 − 6 bits albeit with a modest to significant drop in accuracy. While quantization to sub-byte values (i.e. precision ≤ 8 bits) has been valuable, even further quantization of MobileNets to binary or ternary values is necessary to realize significant energy savings and possibly runtime speedups on specialized hardware, such as ASICs and FPGAs. Under the key observation that convolutional filters at each layer of a deep neural network may respond differently to ternary quantization, we propose a novel quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets. The layer-wise hybrid filter banks essentially combine the strengths of full-precision and ternary weight filters to derive a compact, energy-efficient architecture for MobileNets. Using this proposed quantization method, we quantized a substantial portion of weight filters of MobileNets to ternary values resulting in 27.98% savings in energy, and a 51.07% reduction in the model size, while achieving comparable accuracy and no degradation in throughput on specialized hardware in comparison to the baseline full-precision MobileNets.

The paper presents a quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets.
Strength:
(1)	The paper proposes to only quantize easy-to-quantize weight filters of a network layer to ternary values while also preserving the representational ability of the overall network by relying on few full-precision difficult-to-quantize weight filters.
(2)	The proposed method maintains a good balance between overall computational costs and predictive performance of the overall network. Experimental results show that the proposed hybrid filter banks for MobileNets achieves savings in energy and reduction in model size while preserving comparable accuracy.
(3)	The description is clear in general.
Weakness:
(1)	Though the paper claims that recent works on binary/ternary quantization either do not demonstrate their potential to quantize MobileNets on ImageNet dataset or incur modest to significant drop in accuracy while quantizing MobileNets with 4-6-bit weights, it may worth comparing to the methods that achieved start-of-art results on other datasets to demonstrate the efficiency of the proposed method.
(2)	Figure 1 and Figure 2 is a little blurry.
(3).  How is about the performance compared to latest work? Is it possible to apply current framework to MobileNetV2 ? If can, what's performance?


We propose to study the problem of few-shot graph classification in graph neural networks (GNNs) to recognize unseen classes, given limited labeled graph examples. Despite several interesting GNN variants being proposed recently for node and graph classification tasks, when faced with scarce labeled examples in the few-shot setting, these GNNs exhibit significant loss in classification performance. Here, we present an approach where a probability measure is assigned to each graph based on the spectrum of the graph's normalized Laplacian. This enables us to accordingly cluster the graph base-labels associated with each graph into super-classes, where the L p Wasserstein distance serves as our underlying distance metric. Subsequently, a super-graph constructed based on the super-classes is then fed to our proposed GNN framework which exploits the latent inter-class relationships made explicit by the super-graph to achieve better class label separation among the graphs. We conduct exhaustive empirical evaluations of our proposed method and show that it outperforms both the adaptation of state-ofthe-art graph classification methods to few-shot scenario and our naive baseline GNNs. Additionally, we also extend and study the behavior of our method to semi-supervised and active learning scenarios.

This paper introduces few-shot learning for graph classification. The authors propose a pre-training->fine-tuning approach to handle graph classes unseen at training time (and in only a few shots at test time).
At a high level, and to my understanding, their method a priori generates the ingredients for a graph of graphs, a "super-graph", in two steps: first, it discovers a prototype graph for each graph class in the dataset, then it clusters the prototype graphs into a set of super-classes by k-NN. Both of these operations rely on the spectral distance between graphs, defined in this work using the spectrum of a graph's normalized Laplacian matrix and the pth Wasserstein distance between probability measures. The intuition is that the super-graph built from these ingredients helps model latent relations between graph classes; these relations can be used at test time to improve classification of unseen graph classes.
During (pre-)training, the model builds super-graphs on each batch of data. It uses super-graph information in two ways: an auxiliary classification head (an MLP called C^sup) is trained to map graph embeddings to their corresponding super-class labels, and the super-graph itself, whose nodes are embeddings for individual graphs in the batch, passes through a graph attention network (GAT) that outputs a base class for each graph -- this is the classification head C^GAT. The graph embeddings themselves come from a feature extractor F_θ implemented as a graph isomorphism network (GIN).
During the fine-tuning stage the model adapts to and classifies graphs from classes unseen during training. Here the parameters of the feature extractor GIN and the C^sup MLP are frozen. C^sup outputs a set of super-class labels that are used to construct a super-graph, which in turn feeds into C^GAT, which in turn yields labels for the test graphs. C^GAT (but not the GIN or C^sup) fine-tunes on a small number q of labelled examples of each novel test class. The full model is evaluated on unlabelled examples from the novel test classes. This process assumes that the novel test graph classes belong to the same set of super classes as the training graph classes, a point that is, unfortunately, not discussed.
There's a lot to digest in this paper, on both the technical and architectural sides. There are graphs of graphs (super-graphs) and different GNN variants operating on both, with the output of one graph network, the feature extracting GIN, feeding into another, the GAT classifier. Understanding all of these pieces and how they fit together is challenging for the reader: I got lost somewhat in the Classifier description in Section 4, while Section 3 defines many things and gives some math that might be extraneous. It is also not immediately obvious that fine-tuning takes place on the set G_N and testing on the set G_U. Overall, though, the paper became clear to me with time and I found the overall presentation to be good. Some additional figures that depict the super-graph construction and clustering would be useful.
The construction and use of the super-graph structure to model relations between graph classes is interesting and novel to me, though it relies on well-established techniques (Wasserstein barycenters, Lloyd's algorithm for k-NN). The architecture itself, which combines GINs and GATs, is also novel to me; a downside is that it is highly complex.
Experiments were undertaken on two datasets and seem fairly thorough, with variance established on a high number of seeds (high in the deep learning literature). They demonstrate that the proposed method makes significant improvements over baselines. The baselines are somewhat limited because, as the authors state, "there do not exist any standard state-of-the-art methods for few-shot graph classification". However, I do not think the authors should be penalized for trying something new. On the other hand, given the novelty of the task, it would be nice to see an investigation/discussion of how few-shot graph learning differs from few-shot image learning (where there has been much more work).
I found the ablation and sensitivity studies illuminating, and I was pleased to see that the authors do support their claim that the super-graphs improve class separation over the feature extractor embeddings -- the GIN-k-NN baseline results provide evidence of this.
One place where I lack confidence in the results: I am not very familiar with the datasets used (TRIANGLES and Letter-High) nor how standard they are in the graph-learning literature. The authors do not even describe in the paper what the graph classes in these datasets actually are or represent, which would be good to know.
Overall, I think the paper is worth seeing and discussing at the conference, although it could be improved in various ways.
Minor errors:
- there appears to be bracket imbalance in eq.6
- "Lloyd's" is misspelled a few times
Reviewer's note: I have significant experience in few-shot learning but not in graph neural networks.


We highlight a lack of understanding of the behaviour of Convolutional Graph Neural Networks (GNNs) in various topological contexts. We present 4 experimental studies which counter-intuitively demonstrate that the performance of GNNs is weakly dependent on the topology, sensitive to structural noise and the modality (attributes or edges) of information, and degraded by strong coupling between nodal attributes and structure. We draw on the empirical results to recommend reporting of topological context in GNN evaluation and propose a simple (attribute-structure) decoupling method to improve GNN performance.

The paper presents four experimental set-ups to get a better understanding how the performance of Graph Neural Networks (GNN) depend on topological and/or nodal information.
As the paper is not really in my research area, I would have liked the paper to be a bit more self-contained, but the writing of the paper is generally clear.
The contributions of the paper are mainly experimental and show different "surprising" aspects about GNNs. I found the experimental results to be interesting, especially those presented in Section 5 about decoupling attribute and topological information. However, I think the presentation may be improved and more information (e.g. about how the experiments were conducted or more plots) should be reported in the main paper or the appendix.
Regarding the experiments, the authors made choices about which subsets of datasets (Table 2) or which subsets of measures (Table 1) to use. It would be nice to explain how those choices are made and if they are well-justified. Otherwise, it feels that the presented experimental results have been hand-picked.
Regarding Section 3, I guess that Figure 1 presents averaged results over datasets or over models. I think it would be important to share the results per datasets or per models, as averaging may hide some key aspects.
On page 3, the third paragraph could be illustrated with some plots. Besides, the its first sentence seems to contradict the last sentence of the paragraph before Section 4. Moreover, the text in the fourth paragraph doesn't seem to fit well Fig. 2b for Amazon computers. Am I misreading this plot?
Regarding Section 4, I think it would have been interesting to report the metrics about the topological features for the perturbed graphs. Are they very different from those for the orignal graphs?
Regarding Section 6, the sentence "In the previous sections, we have demonstrated that GNNs .. are robust to perturbations to it" on page 7 seems to contradict the conclusion of Section 4.
I found the experiments in this section to be less convincing. The edge addition technique seems to be ad-hoc and it is not clear why it would improve the performance or not. For instance, how was the value 6°*density-G chosen?
Overall, although I found some of the experimental results very intriguing, I think the paper may not be ready for publication in its current state.


The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statistics over the path ensemble, which allows one to flexibly drive the learning process in a promotion focus or prevention focus. On top of this formulation, we systematically study the impacts of different regulatory focuses. Our findings reveal that regulatory focus, when chosen appropriately, can result in significant benefits. In particular, for the environments with sparse rewards, promotion focus would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, prevention focus is preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.

The paper studies the problem of advantage estimation for actor-critic RL algorithms. The key observation is that the advantage can be computed using 1-step returns, 2-step returns, etc. The paper suggests that, instead of choosing a fixed n, we should aggregate these advantageous together. If the maximum is taken, the resulting policy will be exploratory (i.e., have a "promotion focus"); if the minimum advantage is taken, the resulting policy will be risk sensitive (i.e., have a "regulatory focus").
The paper presents results on a number of tasks. A few toy examples show instances where the proposed method works better. Experiments on sparse reward environments show that taking the max advantage provides for exploration that outperforms a baseline (EX2). On a walking talk show that the min-approach can outperform state-of-the-art on-policy RL (PPO); the paper suggests this is because the min-approach is implicitly risk sensitive. Experiments on some of the Mujoco control tasks and some of the Atari games show that taking the advantage with the maximum absolute value performs well, as compared to PPO.
Overall, I think the main contribution of this paper is the finding that there exist smarter ways of computing the advantage. A second contribution is connecting the ideas of risk-sensitive and risk-seeking control with ideas from psychology (regulatory fit theory).
I am leaning towards rejecting this paper. On the one hand, the paper is well written, well motivated, and the experiments quite thorough. On the other hand, I don't think the paper is particularly useful, either from a theory or algorithmic perspective. It is not surprising that if we add an additional hyperparater to an existing RL algorithm and tune that hyperparameter, we'll do better than the existing RL algorithm. The ablation experiments in Fig 6 and Fig 7 suggest that how the advantage estimates are combined, and how they are "mixed" with a standard advantage estimate, matter a lot. Further, the fact that using a smaller number of advantage estimates worked better (point #2 on pg 5, Effect of Ensemble Size in Appendix A) suggests that the ensemble size is an important hyperparameter, and that risk-seeking / risk-aversion (i.e., regulatory vs promotion focus) cannot alone explain why the proposed method works. I think that, for this idea to be useful, it must be equipped with a fixed value for these hyperparameters that works well across a wide range of tasks (e.g., a learning rate of 3e-4 works well for Adam on most tasks), or an automated strategy for choosing this hyperparameter (e.g., the automatic entropy tuning in SAC). I would consider increasing my review if either of these were included.
A second concern is that, despite the close connections with risk-sensitive and risk-seeking control (discussed in Section 2), none of these prior works are compared against. Many of these prior works include a temperature parameter for trading off risk-seeking vs risk-aversion (e.g., \beta in Eq 11 of [Mihatsch 2002]), which is arguably a more transparent (to the user) and easier to analyze (for the RL researcher) than the order statistics used in this paper. Given the wealth of prior work on risk-sensitive and risk-seeking control, I think it's important to know whether these prior methods already solve the problem at hand (choosing between risk-seeking and risk-aversion). I would consider increasing my review if one of these methods were included as a baseline.
Minor comments
* "that humans own" -> "that humans' own"
* The max strategy seems quite closely related to UCB-based exploration. I'd be curious to learn about some discussion on the similarities/differences
* Fig 6 -- Can you add error bars to show the variance across random seeds?
* Appendix D -- How does this approach and the results shown in Fig 9 differ from standard GAE?
----------------------- UPDATE AFTER (NO) AUTHOR RESPONSE ----------------------------
The authors did not post a response, so I will maintain my vote to "weak reject" this paper. I would encourage the authors to incorporate the feedback in all reviews and submit the paper to a future conference.


In this paper, we show that a simple coloring scheme can improve, both theoretically and empirically, the expressive power of Message Passing Neural Networks (MPNNs). More specifically, we introduce a graph neural network called Colored Local Iterative Procedure (CLIP) that uses colors to disambiguate identical node attributes, and show that this representation is a universal approximator of continuous functions on graphs with node attributes. Our method relies on separability, a key topological characteristic that allows to extend well-chosen neural networks into universal representations. Finally, we show experimentally that CLIP is capable of capturing structural characteristics that traditional MPNNs fail to distinguish, while being state-of-the-art on benchmark graph classification datasets.

This paper proposes an extension of MPNN which leverages the random color augmentation to improve the representation power of MPNN. Authors also prove that two variants of the proposed method have universal representation power (one is exact and the other holds in expectation) from the separability perspective. Experiments on some small graph benchmark datasets and structural property tests are reported.
Overall, the paper seems to make a good contribution on advocating a new perspective of representation power of GNNs, i.e., separability, and proposes a variant to empirically improve representation power. However, I do have quite a few concerns listed as below which impedes my understanding and prevents me from giving a high score.
Pros:
1, The separability perspective of representation power seems novel.
2, The coloring based method is interesting and simple to implement.
3, The graph property test experiments are good testbeds to verify the representation power of various GNNs.
Cons & Questions:
1, The overall paper seems lack of focus in a sense that section 3 and 4 discuss too much on general universality whereas the main contribution, i.e., section 5 is not explained clearly.
2, If I understood correctly, the max operator in Eq. (5) only aggregates the “colored” representation within the group of nodes which shared the same attributes. How do you further get the representation of the whole graph? When k=1, the max operator in Eq. (5) becomes identity, wouldn’t 1-CLIP method be equivalent to augmenting random color as extra node features to GNNs?
3, The whole section 6 is just a very common GNN aggregation operator, I do not understand why authors claim it as “a novel universal neighborhood representation”. Also, the notation in Eq. (8) is not rigorous, what do you exactly mean by psi(x, y) as an MLP? Do you mean concatenating x and y as an input to MLP?
4, The experimental results on the benchmark datasets are less impressive as the mean performances are close to the WL-test results and the variances are considerably large. Moreover, why is the MPNN baseline missing, not mentioning other state-of-the-art GNNs? Same GNN baselines are missing in the structural property tests as well.
5, A closely relevant reference [1] is missing. The equivalence between universal approximation and graph isomorphism testing is studied in [1]. I think it is necessary to discuss the relationship. A comparison with [1] both theoretically and empirically would be make the paper more convincing.
6, Since k-CLIP with some k such that 1 < k < infinity achieves the best performance in the experiments, does k-CLIP still have universal representation theoretically?
7, Many notations are introduced without clear explanation. For example, what does lower-case c stand for? If it stands for the color per node, why does permutation appears in the definition of Eq. (4)? If I understood correctly, Eq. (4) is the set of all colorings which does not depend on permutation anyway. What does S refer to in Eq. (8)?
8, What are variants of CLIP reported in Table 1? Are they 1-CLIP? Also, the multiple bold numbers in Table 1 are quite confusing.
9, Wouldn't Eq. (6) indicate factorial growth rather than the claimed exponential one?
Typos: CDNN in table 1 should be DCNN
[1] Chen, Z., Villar, S., Chen, L. and Bruna, J., 2019. On the equivalence between graph isomorphism testing and function approximation with GNNs. arXiv preprint arXiv:1905.12560.